Objective Exposure to stressful events is associated with both occurrence of depression and also vascular disease. The objective of this study was to determine whether higher levels of stress exposure was related to measures of pathological brain aging, specifically white matter hyperintensity volumes, in older adults with and without depression. Methods The sample included 130 depressed and 110 never‐depressed older adults aged 60 years or older enrolled in a longitudinal study at an academic medical center. Participants completed clinical assessments, assessment of stressful event exposure and perceived stress, and magnetic resonance imaging at baseline and after 2 years. Analyses examined both cross‐sectional and longitudinal relationships between stress measures and white matter hyperintensity volumes. Results There were no statistically significant relationships observed between cross‐sectional baseline stress measures and either baseline hyperintensity volume or 2‐year change in hyperintensity volume. However, after controlling for demographic variables and baseline measures, change in stressor exposure was associated with change in hyperintensity volumes. In this analysis, increased stressor exposure was associated with greater increases in white matter hyperintensity volume, while reductions in stressor exposure were associated with less increase in hyperintensity volume. This relationship did not significantly differ based on the presence of either depression or medical comorbidities. Conclusions This work adds to a growing literature associating exposure to stressful events in later life with more rapid pathological brain aging. Work is needed to understand the physiological mechanisms by which stress exposure has this effect and examine whether stress reduction techniques may modify these observed outcomes. Copyright © 2016 John Wiley & Sons, Ltd.