The purpose of this study was to compare the ability of different instruments, independently developed and traditionally used for measuring science teachers’ beliefs in short‐term interventions, to longitudinally measure teachers’ changing beliefs. We compared the ability of three self‐report instruments (Science Teaching Efficacy Belief Instrument Form A [STEBI], Teaching of Science as Inquiry instrument [TSI], Inquiry Teaching Beliefs instrument [ITB]) and one observational instrument (Reformed Teaching Observation Protocol [RTOP]) to appropriately measure high school chemistry teachers’ beliefs as they engaged in a two and a half year professional development program. Collectively our findings from these four instruments, across three separate cohort of teachers (N = 16), indicated conflicting changes in teacher beliefs. For example, the STEBI indicated teachers’ self‐efficacy remained unchanged or increased while the TSI indicated a concurrent decrease in self‐efficacy throughout the PD program. Additionally, the ITB seemed to indicate a decrease in teachers’ knowledge of inquiry while their interview data and RTOP scores indicated a concurrent increase in their knowledge of and ability to enact inquiry‐based practices. We reconcile these conflicting results and discuss the implications these findings have for validly and reliably measuring science teacher belief changes within longer duration PD. © 2016 Wiley Periodicals, Inc. J Res Sci Teach 53:1055–1081, 2016