Objectives The Hospital Anxiety and Depression Scale (HADS) is a well‐validated, self‐report measure of both anxiety and depression. It is frequently used with people with dementia. However, its structural validity has never been examined in this population. The current study used confirmatory factor analysis (CFA) to assess this. Methods Baseline data from two intervention studies for people with mild to moderate dementia were combined (N = 268). CFA was used to test whether a one, two or three factor structure best fit the data. Indices of model misspecification were examined to test for poor quality items, and models re‐specified accordingly. Finally, measurement invariance across gender and different levels of cognitive impairment was assessed. Results A one‐factor structure did not fit the data. Two and three factor structures fitted the data equally well. Model fit was improved by removal of two items. Measurement invariance was adequate across gender, but poor across groups with differing levels of cognitive impairment. Conclusion The HADS is acceptable and feasible but difficult to interpret in a dementia population. We suggest that it should be interpreted as measuring two separate factors of anxiety and depression and not one ‘distress’ factor. However, two items may need to be removed, affecting cut‐off scores. Poor measurement invariance means the HADS may not be a good tool for measuring differences in anxiety and depression between those with mild and those with moderate cognitive impairment. Copyright © 2016 John Wiley & Sons, Ltd.