Aim Polypharmacy is a serious medical problem among older adults. Polypharmacy can cause adverse drug reactions and is associated with frailty. Several drugs, particularly psychotropic medications, can cause cognitive impairment. Recent research also suggests that polypharmacy can cause cognitive impairment. We investigated the prevalence of polypharmacy, and examined the association between polypharmacy and cognitive impairment in a large sample of community‐dwelling older adults in Japan. Methods A questionnaire covering sociodemographic variables was sent to all residents aged ≥65 years in an urban residential district of Tokyo, Japan (n = 7682). Next, 3000 individuals were randomly selected as study participants. A trained nurse and researcher interviewed participants in their own homes to obtain information about cognitive status, prescribed medications, and current medical history. Cognitive status was assessed using the Mini‐Mental State Examination. Polypharmacy was defined as treatment with six or more prescribed medications. Results Of the 1270 respondents who completed an interview, 1152 were included in the analysis of medications. The prevalence of polypharmacy was 28.0% (n = 323). When the older adults in the medications analysis were stratified by Mini‐Mental State Examination scores, polypharmacy was present in 48.3% of those scoring <24, and 25.7% of those scoring ≥24 (χ2 = 26.76, P < 0.001). After controlling for potential confounding factors (including psychotropic medications), a multivariate logistic regression analysis showed an association between polypharmacy and cognitive impairment (odds ratio 1.83, 95% confidence interval 1.10–3.02; P = 0.019). Conclusions Polypharmacy was associated with cognitive impairment among urban community‐dwelling older adults. Geriatr Gerontol Int 2017; 17: 1286–1293.