This article draws upon the experiences of four recent efforts to synthesize the findings of quantitative studies in science education research. After establishing the need for research syntheses in advancing generalizable knowledge and causal effects research in our field, we identify a set themes that emerged in the process of conducting these syntheses. These themes include that many impact study designs were not conducive to quantitative synthesis, reporting practices of studies were often insufficient for inclusion in research synthesis, and very few replications were performed. These findings have the potential to limit the advancement of knowledge and discovery in science education research. We recommend that the science education research community pay closer attention to existing reporting standards and guidance, as well as consider the merits of a stronger commitment to study registration, replication, and data sharing. © 2016 Wiley Periodicals, Inc. J Res Sci Teach 53: 1216–1231, 2016