Objectives The adverse sexual effects of androgen deprivation therapy (ADT) on men with prostate cancer have been well described. Less well known is the relative degree of sexual dysfunction and bother associated with ADT compared to other primary treatment modalities such as radical prostatectomy. We sought to describe the trajectory and relative magnitude of changes in sexual function and bother in men on ADT and to examine demographic and clinical predictors of ADT's adverse sexual effects. Methods Prostate cancer patients treated with ADT (n = 60) completed assessments of sexual function and sexual bother 3 times during a 1‐year period after the initiation of ADT. Prostate cancer patients treated with radical prostatectomy only and not receiving ADT (n = 85) and men with no history of cancer (n = 86) matched on age and education completed assessments at similar intervals. Results Androgen deprivation therapy recipients reported worsening sexual function and increasing bother over time compared to controls. Effect sizes for the differences in sexual function were large to very large, and for bother were small to very large. Age younger than 83 years predicted relatively poorer sexual function, and age younger than 78 years predicted greater sexual bother at 12 months in men on ADT compared to men not on ADT. Conclusions Most men on ADT for prostate cancer will never return to baseline levels of sexual function. Interventions focused on sexual bother over function and designed to help couples build and maintain satisfying relationship intimacy are likely to more positively affect men's psychological well‐being while on ADT than medical or sexual aids targeting sexual dysfunction.