Aim This study aimed to compare relationships of leisure‐time, household, and work‐related physical activity (PA) with physical performance by age and gender in older Japanese adults. Methods This cross‐sectional study included 525 community‐dwelling older adults (73.3 ± 5.2 years) recruited in 2009–2011 in Kasama City, rural Japan. We used the Physical Activity Scale for the Elderly to assess PA variables. Physical performance was evaluated by 11 performance tests: grip strength, single‐leg balance, functional reach, sit‐and‐reach, standing time from long sitting position, sit‐to‐stand, timed up and go, habitual walk, hand working, and simple and choice reaction times. Results After adjusting for potential confounders, leisure‐time PA in young‐old (≤74 years) men correlated significantly with eight performance tests (absolute value of Spearman's partial rank correlation coefficient: r = 0.18–0.39), whereas in old‐old (≥75 years) men it correlated with three performance tests (r = 0.20–0.23). Although leisure‐time PA correlated with six performance tests (r = 0.19–0.22) in young‐old women, there were no significant correlations between leisure‐time PA and performance tests in old‐old women. Household PA of young‐old men (r = 0.20–0.23) and old‐old women (r = 0.26–0.34) correlated with four performance tests. In old‐old men and young‐old women, no significant correlation was found between household PA and performance tests. Work‐related PA did not relate significantly to any performance tests in any groups. Conclusion This study showed that leisure‐time PA is related to physical performance, especially in young‐old men and women, and household PA is especially related in young‐old men and old‐old women. Our findings suggest that supporting strategies for maintaining physical functions would differ by gender and age. Geriatr Gerontol Int 2013; ●●: ●●–●●.