As physical inactivity may track from adolescence to adulthood, it is important to identify social determinants of physical inactivity in early life. However, most studies have measured socioeconomic position as one dimension. We examine whether multiple dimensions of socioeconomic position, in addition to other dimensions of inequality (i.e., gender, immigrant background), associate with physical inactivity at two time points in youth. Longitudinal data were drawn from the Swedish Level of Living Survey (N = 765) and analysed by gender-stratified logistic regression. Among girls, low parental social class (odds ratio [OR] = 2.63, 95% confidence interval [CI; 1.28, 5.42]) and income (OR = 2.28, 95% [CI 1.12, 4.65]) were associated with physical inactivity, while immigrant background (OR = 2.33, 95% CI [1.03, 5.23]) and a low level of parental education (OR = 3.38, 95% CI [1.15, 9.95]) predicted physical inactivity among women. Among boys, low parental income (OR = 3.27, 95% CI [1.39, 7.69]) was associated with physical inactivity, whereas immigrant background (OR = 2.29, 95% CI [1.04, 5.03]) predicted physical inactivity among men. Our results suggest that physical inactivity is socially patterned, but different dimensions of social stratification should not be considered interchangeable as they may operate independently, through intersection with gender, and at different time points in youth in increasing the risk of physical inactivity.