Aim To explore the relationships between the built environment and older adults’ physical function. Methods The present cross‐sectional study carried out in 2010–2012 used data drawn from 509 community‐dwelling older adults aged 65–86 years living in Kasama City, a Japanese rural region. We evaluated physical function with the following performance tests: grip strength, sit‐to‐stand, timed up & go and walking speed. Using geographic information systems, we measured population density and the number of destinations related to daily life, community centers, medical facilities and recreational facilities within participants’ neighborhoods. Results After adjusting for potential confounders, we found lower population density was related to poor performance of sit‐to‐stand and walking speed in both sexes, and grip strength in women (trend P < 0.05). A lower number of daily life‐related destinations was related to poor performance of sit‐to‐stand and walking speed in men, and grip strength and sit‐to‐stand in women. Similarly, the number of community centers was related to walking speed in both sexes. The number of medical and recreational facilities was also related to some physical performance in both sexes. A lower land use mix score, calculated by principal component analysis, was related to lower performance of sit‐to‐stand and walking speed in men, and grip strength and sit‐to‐stand in women. Conclusion The present study suggests that, although there are some sex differences, low population density, land use mix, and fewer daily life‐related destinations, community centers, medical facilities and recreational facilities are negative determinants of physical function. Geriatr Gerontol Int 2017; 17: 382–390.