Aim The aim of the present study was to investigate the association between various body composition parameters and the risk of mild cognitive impairment (MCI) in older Japanese adults, as well as potential sex‐related differences in the risk of MCI. Methods Participants underwent cognitive tests, and were divided into 840 participants with MCI (mean age 71.9 ± 5.5 years) and 1740 without MCI (mean age 71.3 ± 5.2 years). Body composition parameters were measured using a bioelectrical impedance analyzer. Multiple logistic regression analysis was then carried out to examine the associations between body composition parameters and risk of MCI. Results After adjusting for confounding factors, those in the lowest quartile for fat‐free mass had a higher risk of MCI than those in the highest quartile (men: odds ratio [OR] 1.96, 95% confidence interval CI 1.24–3.10; women: OR 1.49, 95% CI 1.01–2.19). Loss of muscle mass in the upper (OR 2.17, 95% CI 1.40–3.37) and lower (OR 1.99, 95% CI 1.25–3.15) limbs was associated with a higher MCI risk in men. However, only loss of muscle mass in the lower limbs was associated with a higher MCI risk (OR 1.61, 95% CI 1.06–2.44) in women. No associations were found between obesity measures and MCI. Conclusions We found that loss of fat‐free mass was associated with MCI in older adults, regardless of sex. We also found that appendicular muscle mass was more closely associated with MCI in men than in women. These results suggest that the association between appendicular muscle mass and MCI might have different underlying mechanisms based on sex. Geriatr Gerontol Int 2017; ••: ••–••.