Although the relationship between aging and cognitive decline is well established, there is substantial individual variability in the degree of cognitive decline in older adults. The present study investigates whether variability in cognitive performance in community‐dwelling older adults is related to the presence of whole brain or tract‐specific changes in white matter microstructure. Specifically, we examine whether age‐related decline in performance on the Montreal Cognitive Assessment (MoCA), a cognitive screening tool, is mediated by the white matter microstructural decline. We also examine if this relationship is driven by the presence of cardiovascular risk factors or variability in cerebral arterial pulsatility, an index of cardiovascular risk. Sixty‐nine participants (aged 43–87) completed behavioral and MRI testing including T1 structural, T2‐weighted FLAIR, and diffusion‐weighted imaging (DWI) sequences. Measures of white matter microstructure were calculated using diffusion tensor imaging analyses on the DWI sequence. Multiple linear regression revealed that MoCA scores were predicted by radial diffusivity (RaD) of white matter beyond age or other cerebral measures. While increasing age and arterial pulsatility were associated with increasing RaD, these factors did not mediate the relationship between total white matter RaD and MoCA. Further, the relationship between MoCA and RaD was specific to participants who reported at least one cardiovascular risk factor. These findings highlight the importance of cardiovascular risk factors in the presentation of cognitive decline in old age. Further work is needed to establish whether medical or lifestyle management of these risk factors can prevent or reverse cognitive decline in old age.