Previous investigations into whether the APOE‐ε4 allele exerts cognitive effects at midlife have been inconclusive. We have advanced a “cognitive phenotype” hypothesis arguing that the ε4 allele of the apolipoprotein E gene (APOE) is associated with lower efficiency of neuronal plasticity thereby resulting in poorer cognitive performance independently of the pathology of Alzheimer's disease (Greenwood et al., ). This hypothesis is best tested at midlife, prior to the neuron loss associated with AD diagnosis. This hypothesis predicts that the ε4 allele would alter cognition regardless of age through plasticity mechanisms, but would not induce longitudinal decline in midlife. The alternative “prodrome” hypothesis predicts that the APOE‐ε4 allele would be associated with longitudinal cognitive decline as early as midlife due to prodromal effects of AD. We tested these hypotheses with a working memory task in a large cross‐sectional sample of cognitively screened APOE‐ε4 carriers and non‐carriers and also in a small longitudinal sample over 3 years. The sample was divided into middle‐aged (mean age 50, range 40–59) and older (mean age 69, range 60–84) individuals. Cross‐sectionally, we observed that older, but not middle‐aged, APOE‐ε4 carriers had lower accuracy than ε4 non‐carriers, mainly under the hardest discrimination condition. Longitudinally, we observed increases in accuracy in middle‐aged APOE‐ε4 carriers, suggesting a cognitive phenotype that includes ability to benefit from experience. We observed a longitudinal decrease in older APOE‐ε4 carriers, suggesting an AD prodrome.