A recent meta‐analysis showed that contextualized personality inventories have incremental predictive validity over generic personality inventories when predicting job performance. This study aimed to investigate the differences between two types of contextualization of items: Adding an ‘at work’ tag versus completely modifying items. One hundred thirty‐nine pharmacy assistants from 29 pharmacies filled out a generic, a tagged, and a completely modified personality inventory. The assistants also provided participant reactions for each of the personality inventories. Performance ratings were collected from the supervising pharmacists. We expected to find incremental criterion validity for both the tagged inventory and the completely modified inventory for predicting job performance. However, the results showed an unexpected decrease in predictive validity for the contextualized inventories. Contextualized inventories were liked less than the generic inventory, but were considered somewhat more face valid and predictive by the participants.