Objective Supervisor assessments of trainee competence are integral to ensuring that clinical psychology trainees reach competency benchmarks. The commonly used Clinical Psychology Practicum Competencies Rating Scale (CΨPRS) has been shown to elicit inflated ratings of competency. Hence, the aim of this study is to examine whether brief supervisor training reduces ratings by providing objective criteria with which supervisors can assess trainee competency. Method The ratings included were of 124 psychology trainees from nine Australian university clinical programmes. Of 170 supervisors, 32 completed the online training immediately prior to commencing the CΨPRS. Training required supervisors to rate the competency level described in five standardised vignettes (Beginner through to Competent). Vignette ratings, as determined by a panel of expert supervisors, were provided as feedback. A sixth calibration vignette was also rated (no feedback provided). Firstly, CΨPRS ratings from the trained and untrained supervisors were compared. Secondly, the difference between supervisor and expert ratings of the calibration vignettes were compared across trained and untrained groups. Results Trained supervisors provided lower CΨPRS ratings than untrained supervisors. In addition, trained supervisors (vs untrained supervisors) provided ratings of the calibration vignette that more accurately matched the ratings provided by the expert panel. Conclusions Brief online training using standardised vignettes was associated with lower CΨPRS ratings. The standardised vignettes helped calibrate supervisors’ ratings and likely attuned supervisors to the skills and competency levels that are expected at particular developmental stages. As a consequence, training appeared to reduce ratings, arguably resulting in more accurate assessments of trainee performance.