Objective Patient‐reported outcome measures are increasingly required as metrics for determining the efficacy of depression interventions. However, the scores obtained from self‐reports over the course of treatment may be biased because respondents restructured the factors of the depression symptoms (reconceptualization), changed their values of the symptoms in reflecting depression (reprioritization), or adjusted their standards of assessing the levels of symptoms (recalibration). The aim of present study is to assess response shifts on the depression measure. Method This study uses the procedures for assessing longitudinal measurement invariance to evaluate response shift effects on the Beck Depression Inventory II (BDI‐II) with 320 early adolescents suffering from depression (43.8% boys and 56.2% girls) over the course of their treatment in the school settings. Results The findings reveal clear signs of response shifts (recalibration) in the BDI‐II for depressed students. Compared with the factor scores before treatment, the factor scores after treatment were underestimated, especially on the negative attitude and performance difficulty factors. Conclusions Estimates of treatment efficacy based on observed test scores appear to be confounded by response shifts.