This study compared contextualized and non‐contextualized personality measures in operational hiring situations, unlike previous research which has largely relied on student or job incumbent samples. Comparisons were made with regard to (A) the frames‐of‐reference adopted by applicants when responding to the measures, (B) relations with subsequent employment interview scores, (C) applicant reactions, and (D) mean scores for the personality scales. The findings highlight potential concerns with using non‐contextualized personality measures for employee selection, as job applicants will likely adopt various frames‐of‐reference unrelated to the intended focus of measurement (i.e., work). Results also indicated that it may be premature to assume contextualized measures elicit improved applicant reactions despite their apparent job‐relatedness. The discussion concludes with suggested directions for future research.