Will the consequences of residential segregation, that is, spatial concentration of marginalised populations on the one hand, and spatial concentration of affluent populations on the other hand, generate a situation where individual life trajectories are influenced by where individuals grow up? Our aim is to analyse how poverty risks and early income career at adult age are influenced by different neighbourhood contexts in early youth. We use Swedish longitudinal register data, and follow individuals born in 1980 until 2012. Residential context is measured in 1995 at age 15 by expanding a buffer around the residential locations of each individual and, by computing statistical aggregates of different socio-demographic variables for that population. The results show that poverty risks increase for individuals growing up in areas characterised by high numbers of social allowance recipients living nearby, whereas elite geographical context is favourable for both women’s and men’s future income.