Blacks are especially hard hit by cognitive impairment at older ages compared to whites. Here, we take advantage of the Health and Retirement Study (1998–2010) to assess how this racial divide in cognitive impairment is associated with the racial stratification of life course exposures and resources over a 12-year period among 8,946 non-Hispanic whites and blacks ages 65 and older in 1998. We find that blacks suffer from a higher risk of moderate/severe cognitive impairment at baseline and during the follow-up. Blacks are also more likely to report childhood adversity and to have grown up in the segregated South, and these early-life adversities put blacks at a significantly higher risk of cognitive impairment. Adulthood socioeconomic status is strongly associated with the risk of cognitive impairment, net of childhood conditions. However, racial disparities in cognitive impairment, though substantially reduced, are not eliminated when controlling for these life course factors.