Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis
Published online on November 11, 2016
Abstract
This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM–ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom they considered to be typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Out of a total of 3,310 examinees taking different trial versions of the placement test, 470 learner prototypes were identified. For this set of prototypes, Rasch model estimates of EFL proficiency served as input to a series of ROC analyses, one for each pair of adjacent proficiency levels. Cut scores were derived using the Youden index that maximizes the overall rate of correct classification and minimizes the overall rate of misclassification. Findings confirmed that this method allows for the setting of cut scores that show a high level of classification accuracy in terms of the correspondence with expert categorizations of examinee prototypes. In addition, the ROC-based cut scores were associated with higher classification accuracy than cut scores derived from a logistic regression analysis of the same data. Potential further uses and implications of the PGM–ROC approach in the context of language testing and assessment are discussed.