Infant mortality is a major problem in the United States. Understanding the relationship between geographic variation and birth outcomes is important to develop and implement targeted interventions. This study examined the relationship between contextual residential distance to public housing and infant mortality. All singleton births (N = 30,554) and infant deaths (N = 361) occurring between January 1, 1997 and December 31, 2007 in Richmond, VA were analyzed. Maternal race, education, smoking during pregnancy, method of payment, gestational age, and sex of the infant were significantly associated with infant mortality. African Americans were 1.6 times as likely to reside 1 to 2 miles or less closer to public housing (p<.0001) than European Americans. Residential contextual variables play a role in poor birth outcomes, and public health professionals should be aware of this association when developing interventions.