Perceptions of neighborhood safety are positively associated with perceptions of neighborhood violence. However, research has yet to examine whether specific types of violence such as sexual violence moderate this relationship that are more salient for women. Using street‐intercept interviews with 343 adults in 9 neighborhoods of a U.S. city with high rates of poverty, unemployment, and crime, we examine the relationship of perceived neighborhood violence to perceived safety in the context of gender while controlling for neighborhood assets that moderate perceptions of neighborhood safety and violence. We hypothesized that gender would moderate the relationship between perceived neighborhood violence and safety, and that women's perceptions of neighborhood safety would be significantly influenced by neighborhood sexual violence, but not other types of violence. Although women and men in these high crime, urban neighborhoods did not differ in their perceptions of neighborhood safety or violence, perceived sexual violence did significantly moderate safety by gender; women's perceptions of neighborhood sexual violence predicted perceived safety in their neighborhood. Importantly, gender did not moderate perceived safety for other types of violence. These results illustrate the importance of taking gender and perceived sexual violence into account to understand neighborhood safety in adults, particularly women.