Despite the popularity of frame‐of‐reference training (FORT), it is not clear how different structural elements of FORT work in concert to improve rating accuracy. Furthermore, past rater training studies have lacked rigorous control groups leading to low thresholds for showing improvements in rating accuracy due to FORT. The current study allowed for the isolation of components of rater training that increase rating accuracy when compared to a rigorously designed control group. Results indicated that repeated rendering of practice ratings improve rating accuracy and this practice effect was amplified by practice rating feedback. Although accuracy‐based training content improved interrater agreement, it did not contribute to improvements rating accuracy over and above the control group. We discuss the implications of the findings in relation to best practices for designing rater training programs.