Group Anchoring in Rater‐Mediated Assessments with Sparse Designs and Rater Effects
Journal of Educational Measurement
Published online on July 05, 2026
Abstract
["Journal of Educational Measurement, Volume 63, Issue 3, Fall 2026. ", "\nAbstract\nSparse rating designs, where it is not possible for all raters to score all students, are often unavoidable in rater‐mediated assessment settings. Unfortunately, sparse ratings can lead to imprecise and inaccurate estimates of model parameters, including student ability, and reduce the detection of rater effects. In the context of latent trait models, researchers have proposed group anchoring as a potential post‐data‐collection solution to sparseness. We used a simulation study and real‐data analysis of ratings from 13 raters of 75 high‐school student musical performances to explore the impact of group anchoring on parameter estimates when compared with various disconnected and sparse rating designs. We manipulated the conditions of student sample size, rater effect presence, and rating design. Our results showed that student parameter correlations between fully connected estimates and group anchoring estimates were generally comparable to those from partially connected rating designs, especially with larger sample sizes. However, the results were less stable for estimating rater parameters. Although group anchoring may serve as a robust alternative to fully connected designs for student achievement, it may not always be appropriate when evaluating rater severity.\n"]