Insufficient physical activity (PA) and poor physical fitness are risks for several noncommunicable diseases among working-aged men. PA programs have been launched to increase activity levels in the population but working-aged men have been underrepresented in these programs. The aim of the present cross-sectional study was to evaluate validity of a short scale for psychosocial factors among Finnish working-aged men who participated in a PA campaign. The study examined also the associations between psychosocial factors and phase of PA change across fitness groups. Physical fitness was assessed with a body fitness index constructed on the basis of a handgrip test, the Polar OwnIndex Test, and body composition analysis (InBody 720). The men were classified into low (n = 162), moderate (n = 358), and high (n = 320) body fitness index groups. Psychosocial factors and self-reported phase of PA change were assessed with a questionnaire. Psychometric properties of the scale were assessed with confirmatory factor analysis and differences between phases of PA change were examined with one-way analysis of variance. The evaluated scale included factors for self-efficacy, goal setting, skills, and social support. Good physical fitness was related to better perceived self-efficacy and ability to manage one’s PA environment. Goal setting was critical for PA change at all fitness levels. Better understanding of the interactions between psychosocial factors and PA change could help in targeting PA programs to low-fit men. Further study should examine the validity of the improved psychosocial measure.