Although the concept of valence figures in many studies of voting behavior, very few have investigated its sources. In this article, we address this deficiency by assessing the extent to which individual valence assessments are affected by the left–right policy distance between parties and respondents as well as by their locations relative to the center of the left-right spectrum. Using data from the Comparative Study of Electoral Systems, we find very widespread support for both hypotheses. Correcting for differential item functioning (DIF) reveals that, although respondents tend to over-estimate the distance to ‘opposite-side’ parties, these misperceptions do not account for our findings. Indeed, with DIF corrected, most surveys reveal tendencies not only for opposite-side parties to be given lower valence scores in general, but also for policy distance to be counted more heavily against them. The article concludes with a discussion of possible sources of this bilateral structuring of valence assessments.