Purpose Many breast cancer survivors report cancer and cancer treatment‐associated cognitive change. However, very little is known about the relationship between physical activity and subjective memory impairment (SMI) in this population. The purpose of this study is to examine the relationship between physical activity and SMI and longitudinally test a model examining the role of self‐efficacy, fatigue and distress as potential mediators. Methods Post‐treatment breast cancer survivors (N = 1477) completed measures of physical activity, self‐efficacy, distress (depression, concerns about recurrence, perceived stress, anxiety), fatigue and SMI at baseline and 6‐month follow‐up. A subsample (n = 362) was randomly selected to wear an accelerometer. It was hypothesized that physical activity indirectly influences SMI via exercise self‐efficacy, distress and fatigue. Relationships were examined using panel analysis within a covariance modeling framework. Results The hypothesized model provided a good fit in the full sample (χ2 = 1462.5, df = 469, p = <0.001; CFI = 0.96; SRMR = 0.04) and the accelerometer subsample (χ2 = 961.8, df = 535, p = <0.001, CFI = 0.94, SRMR = 0.05) indicating increased physical activity is indirectly associated with reduction in SMI across time, via increased exercise self‐efficacy and reduced distress and fatigue. Conclusions Higher levels of physical activity, lower levels of fatigue and distress and higher exercise self‐efficacy may play an important role in understanding SMI in breast cancer survivors across time. Future research is warranted to replicate and explore these relationships further. Copyright © 2016 John Wiley & Sons, Ltd.