Objectives To identify the factors underlying prostate cancer (PCa) patients' depression–anxiety, sexual problems, urinary dysfunction and androgen deprivation therapy (ADT)‐linked breast changes and hot flushes, and test these as predictors of loss of masculinity (LoM) over 36 months following diagnosis. Methods One thousand seventy patients from the TROG 03.04 (RADAR) trial the EORTC QLQ C‐30 and PR 25 questionnaires, and the International Prostate Cancer Symptom Score of the American Urological Association at baseline, 3, 7, 12, 18, 24 and 36 months. Selected items from these scales were factor‐analysed to identify a four‐component solution for responses at 18 and 36 months, and these components were regressed against a single‐item measuring LoM. Results Depression–anxiety factor was the most powerful predictor of LoM at both time points, followed by sexual problems of ADT side effects (breast changes and hot flushes). Urinary dysfunction was not a consistent predictor of LoM. Depression–anxiety was also the most significant factor distinguishing between those men who reported LoM and those who did not. Conclusions Although LoM is often reported as arising from ADT, the relative power of depression–anxiety in predicting LoM, both at the selected time points and using a time‐lagged analysis, plus the finding that depression–anxiety was the most consistent difference between men who reported LoM and those who did not, argues for the presence of adverse mood states as being the key ingredient in deciding if PCa patients experience loss of their feelings of masculinity. Copyright © 2013 John Wiley & Sons, Ltd.