Objective Clinical anxiety disorders are associated with white matter hyperintensities and diffusion abnormalities measured using diffusion tensor imaging. However, it is not known if this association extends into individuals with mild anxious symptoms without formal diagnosis, in those who are older, or in those who have atherosclerosis. The current study explores whether white matter integrity and/or organization significantly associates with anxious symptoms in older adults with and without atherosclerosis. Methods We recruited older adults (ages 55–90 years); 35 with clinically diagnosed atherosclerotic vascular disease (AVD) and 22 without AVD. Anxious symptoms were measured using the validated Symptom Checklist‐90‐Revised. Fractional anisotropy (FA), a proxy for white matter organization and health, was measured in the white matter globally, by lobe, and in several smaller regions of interest suggested by the literature. Partial correlations between anxious symptoms and FA were calculated, controlling for significant covariates. Results Participants with and without AVD did not differ in severity of anxious symptom endorsement. There was a unique inverse relationship between white matter health and anxious symptoms in the AVD participants, but not in healthy comparisons. Significant relationships were observed in the superior longitudinal fasciculus (r = −0.476, df = 32, p = 0.004), as well as the cingulum bundle, the frontal lobes, and the parietal lobes. Conclusions Anxiety symptoms uniquely correlated with low FA in older adults with atherosclerosis. These findings may have implications for future research on the topic of anxiety in aging and vascular disease and warrant replication. Copyright © 2013 John Wiley & Sons, Ltd.