This study investigates whether raters' knowledge of test takers' first language (L1) affects how the raters orient themselves to the task of rating oral speech. The authors qualitatively investigated the effects of accent familiarity on raters' score assignment processes. Twenty‐six trained raters with a second language of Mandarin Chinese, Korean, or Spanish rated English speech samples from 72 English language learners from three L1 backgrounds (Mandarin Chinese, Korean, and Spanish). Afterward, raters watched videos of themselves rating and discussed their rating processes. Analyses of data, assisted by the qualitative analysis software package QSR NVivo 8, revealed that some raters attended to and were influenced by test‐taker accent and L1. Raters who were heritage language learners discussed how their personal heritage status influenced how they thought about the speech samples from speakers whose L1 matched their heritage language. Although sensitivity to test‐taker accents seemed to occur naturally in the rating process, findings suggest that when raters have learned or know, to varying degrees, the test takers' L1, they tend to orient themselves to the speech in a biased way, compromising test reliability.