Autism spectrum disorder (ASD) is among other things characterized by specific impairments in emotion processing. It is not clear, however, to what extent the typical decline in affective functioning is related to the specific autistic traits. We employed The Autism Spectrum-Quotient (AQ) to quantify autistic traits in a group of 500 healthy individuals and investigate whether we could detect similar difficulties in the perception of emotional expressions in a broader autistic phenotype. The group with high AQ score was less accurate and needed higher emotional content to recognize emotions of anger, disgust, and sadness. Our findings demonstrate a selective impairment in identification of emotional facial expressions in healthy individuals that is primarily related to the extent of autistic traits.