MetaTOC stay on top of your field, easily

When Cut Scores Impact Human Ratings: An Extended Many‐Facet Rasch Model for Category‐Specific Rater Severity Shifts

,

Journal of Educational Measurement

Published online on

Abstract

["Journal of Educational Measurement, Volume 63, Issue 3, Fall 2026. ", "\nAbstract\nThe many‐facet Rasch model (MFRM), widely used to analyze and evaluate rater‐mediated assessments, focuses on between‐rater differences in overall severity or leniency across facets and rating scale categories. Studying within‐rater severity differences, particularly abrupt shifts in a rater's tendency to assign harsh or lenient ratings around specific scale categories, requires an extended modeling approach. Specifically, to focus on local, category‐dependent severity effects, we propose the “many‐facet Rasch model for category‐specific severity shifts” (MFRM‐CSS). Building on Bayesian parameter estimation, we demonstrate the model's suitability for examining local severity changes at scale categories with special significance as cut scores. In a simulation study and real‐data analysis, we found that (a) differences in category‐dependent severity levels affected observed score distributions and passing rates, (b) the MFRM‐CSS reliably recovered true overall and local rater severity parameters, (c) ignoring local severity shifts biased overall severity and category threshold estimates, and (d) the MFRM‐CSS outperformed the baseline MFRM, which does not account for category‐specific severity shifts, in data‐model fit when applied to essay rating data. In the real dataset, the greatest impact of local severity was observed at the second threshold, where the pass‐fail cut score was set. The discussion highlights the practical implications of applying the MFRM‐CSS and suggests directions for future research.\n"]