There is increasing interest in both the cumulative and long‐term impact of early life adversity on brain structure and function, especially as the brain is both highly vulnerable and highly adaptive during childhood. Relationships between SES and neural development have been shown in children older than age 2 years. Less is known regarding the impact of SES on neural development in children before age 2. This paper examines the effect of SES, indexed by income‐to‐needs (ITN) and maternal education, on cortical gray, deep gray, and white matter volumes in term, healthy, appropriate for gestational age, African‐American, female infants. At 5 weeks postnatal age, unsedated infants underwent MRI (3.0T Siemens Verio scanner, 32‐channel head coil). Images were segmented based on a locally constructed template. Utilizing hierarchical linear regression, SES effects on MRI volumes were examined. In this cohort of healthy African‐American female infants of varying SES, lower SES was associated with smaller cortical gray and deep gray matter volumes. These SES effects on neural outcome at such a young age build on similar studies of older children, suggesting that the biological embedding of adversity may occur very early in development. Relationships between SES and neural development have been shown in older children, however, less is known regarding the impact of SES on neural development in infants. We examined effects of SES on cortical gray, deep gray, and white matter volumes in term, healthy, appropriate for gestational age, African‐American, female infants at 5 weeks of age and found that lower SES was associated with smaller cortical gray and deep gray matter volumes. These SES effects on neural outcome at such a young age build on similar studies of older children, suggesting that the biological embedding of adversity may occur very early in development.