Second language pronunciation research and teaching relies on human listeners to assess second language speakers’ performance. Most applied linguists working in this area have been satisfied that listener ratings are reasonably reliable when well‐controlled research protocols are implemented. We argue, however, that listeners demonstrate a certain amount of reliability in their ratings of speakers stemming from shared expectations of a speaker's language and social groups, rather than from the speech itself. In this article, we discuss evidence from perceptual psychology, sociolinguistics, and phonetics demonstrating a sizable listener influence on speech perception. We conclude by suggesting ways for research and teaching to acknowledge and contend with the role of the listener.