Inattentiveness and social desirability might be particularly problematic for self‐report scales in creativity and arts research. Respondents who are inattentive or who present themselves favorably will score highly on scales that yield positively skewed distributions and that assess socially valued constructs, such as scales measuring creativity and arts knowledge. A total of 204 undergraduates completed an online survey with several self‐report measures (the Creative Achievement Questionnaire, the Biographical Inventory of Creative Behavior, and the Aesthetic Fluency Scale). Many metrics of inattentiveness were included, such as directed response items, self‐reported attentiveness, and scales to catch inconsistent and patterned responses. The Balanced Inventory of Desirable Responding measured facets of social desirability. We found high rates of inattentive responding. A latent class analysis indicated that around 25% of the sample was potentially inattentive, and inattentive respondents received higher scores on the creativity and arts scales. Excluding problematic participants increased the effect sizes of interest, consistent with the coarsening effect of careless participants. Social desirability, in contrast, had essentially no relationship with the creativity and arts scales. These findings suggest that social desirability is probably less of a problem than researchers fear, but that inattentiveness is an underappreciated threat. Some practical guidelines are suggested.