Background: Understanding how the determinants of behaviour vary by context may support the design of interventions aiming to increase physical activity. Such factors include independent mobility, time outdoors and the availability of other children. At present little is known about who children spend their time with after school, how this relates to time spent indoors or outdoors and activity in these locations. This study aimed to quantify who children spend their time with when indoors or outdoors and associations with moderate to vigorous physical activity (MVPA). Methods: Participants were 427 children aged 10–11 from Bristol, UK. Physical activity was recorded using an accelerometer (Actigraph GT1M) and matched to Global Positioning System receiver (Garmin Foretrex 201) data to differentiate indoor and outdoor location. Children self-reported who they spent time with after school until bed-time using a diary. Each 10 second epoch was coded as indoors or outdoors and for ‘who with’ (alone, friend, brother/sister, mum/dad, other grown-up) creating 10 possible physical activity contexts. Time spent and MVPA were summarised for each context. Associations between time spent in the different contexts and MVPA were examined using multiple linear regression adjusting for daylight, age, deprivation and standardised body mass index. Results: During the after school period, children were most often with their mum/dad or alone, especially when indoors. When outdoors more time was spent with friends (girls: 32.1%; boys: 28.6%) than other people or alone. Regression analyses suggested hours outdoors with friends were positively associated with minutes of MVPA for girls (beta-coefficient [95% CI]: 17.4 [4.47, 30.24]) and boys (17.53 [2.76, 32.31]). Being outdoors with brother/sister was associated with MVPA for girls (21.2 [14.17, 28.25]) but not boys. Weaker associations were observed for time indoors with friends (girls: 4.61 [1.37, 7.85]; boys: (7.42 [2.99, 11.85]) and other adults (girls: 5.33 [2.95, 7.71]; boys: (4.44 [1.98, 6.90]). Time spent alone was not associated with MVPA regardless of gender or indoor/outdoor location. Conclusions: Time spent outdoors with other children is an important source of MVPA after school. Interventions to increase physical activity may benefit from fostering friendship groups and limiting the time children spend alone.