Research indicates that people from low socioeconomic and ethnic minority backgrounds disproportionately fail to meet the weekly national recommendations for physical activity. Perceptions of environmental factors may be significant barriers to walking in communities experiencing high rates of crime and poverty and may affect women and men differently. The present study investigated sex differences in the impact of perceptions of the environment and overall neighborhood quality on walking in an underserved population. Participants were 195 African American women and 95 African American men. Baseline data indicated that women reported greater concerns about safety, poorer perceptions of aesthetics, and lower neighborhood satisfaction than men. Regression analyses showed that perceptions of access to places for walking significantly predicted walking for both women and men and that sex moderated the effect of neighborhood satisfaction. These results suggest that interventions focused on building accessibility to places for walking while also addressing social environmental concerns are needed in high-crime communities.