Objective Few studies examined the association between perceived stress and cognitive function in older adults. This study will examine which aspects of perceived stress especially impact cognitive function. Methods Cross‐sectional data of 1099 older adults between 64 and 100 years from the Longitudinal Aging Study Amsterdam were used. Perceived stress and its subscales perceived helplessness and perceived self‐efficacy were measured with the Perceived Stress Scale. Cognitive function was assessed regarding memory, processing speed and executive function. Univariate and multivariate linear regression analyses were performed between the stress measures and the domains of cognitive function. Results Perceived stress was associated with worse processing speed, direct and delayed recall, semantic fluency and digit span backwards (range β = −0.10; −0.11; p < 0.01). The subscale perceived helplessness showed negative associations only with processing speed (β = −0.06, p < 0.05) and delayed recall (β = −0.06, p < 0.05), which became nonsignificant after the adjustment for depressive symptoms or sense of mastery. The subscale perceived self‐efficacy was significantly associated with better cognitive function, also after adjustment for depressive symptoms or sense of mastery (range β = 0.10; 0.18; p < 0.01). Conclusions In older adults, especially perceived self‐efficacy showed independent associations with a broad range of cognitive functions. Perceived self‐efficacy might be an important factor in reducing stress and the prevention of cognitive decline. Copyright © 2016 John Wiley & Sons, Ltd.