We systematically compared different measures of attentional bias (i.e., reaction times, the N2pc component in the EEG, and explicit stimulus ratings) on their ability to reveal attentional engagement to threatening versus neutral facial stimuli in a Dot Probe Task and tested their relation to trait measures of general and social anxiety. We found that the N2pc component reflects a bias toward angry faces with excellent internal consistency. Similar results were obtained for explicit ratings. Reaction time (RT) differences, however, were not indicative of attentional biases and showed zero odd‐even reliability. We further found that both higher (i.e., more negative) N2pc amplitudes and earlier peak latencies were associated with more severe symptoms of social anxiety even when controlling for general trait anxiety. The explicit rating biases were also specifically associated with social anxiety. Conversely, the RT bias was not related to social anxiety levels but to general trait anxiety. This highlights the importance of valid and reliable outcome measures for interventions such as attentional bias modification protocols. Mutual exclusivity of different bias operationalizations is discussed.