To narrow the achievement gap between English language learners (ELLs) and their native‐speaking peers in K–12 settings in the United States, effective instructional models must be identified. However, identifying valid observation protocols that can measure the effectiveness of specially designed instructional practices is not an easy task. This study examines the factorial validity of the widely used sheltered instruction observation protocol (SIOP), which proposes a systematic framework for planning, delivering, assessing, and evaluating instructional practices that can help ELLs attain English proficiency and achieve academically in content areas (Echevarría, Vogt, & Short, 2013). In a large city in the eastern United States, 102 SIOP‐trained in‐service teachers used SIOP as the performance evaluation instrument to rate the effectiveness of a video‐recorded SIOP lesson taught by a science teacher. Results of four exploratory factor analyses suggest that SIOP seems to measure four distinguishably stable performance evaluation factors in determining teaching effectiveness that is specifically characterized as sheltered instruction. Discrepancies between the current structure of the protocol and the latent structures identified here point to a need for more fine‐grained theoretical and operational foundations. Some suggestions are made about the restructuring of the model.