Finding Efficiency in the Design of Large Multisite Evaluations: Estimating Variances for Science Achievement Studies
American Journal of Evaluation
Published online on February 23, 2016
Abstract
Little is known empirically about intraclass correlations (ICCs) for multisite cluster randomized trial (MSCRT) designs, particularly in science education. In this study, ICCs suitable for science achievement studies using a three-level (students in schools in districts) MSCRT design that block on district are estimated and examined. Estimates of ICCs, by district, are computed for unconditional two-level (students in schools) hierarchical linear models using student-level science achievement raw scores as measured by the Texas Assessment of Knowledge and Skills in 2010–2011 for Grades 5, 8, 10, and 11. The average within-district ICCs for MSCRTs of science achievement range from 0.0781 to 0.0982 across grades. For Grade 5, a significant difference is found in the ICC estimate suitable for MSCRTs with many districts versus only a few districts. Consequently, evaluators of science education interventions should exhibit care when selecting ICC estimates for three-level MSCRT designs that block on district.