We propose that the causes of the effectiveness (or ineffectiveness) of police interventions can be better understood with an increased focus on the measurement of treatment implementation and outputs, as opposed to the more common "black box" conceptualizations of police interventions and outcome-only evaluations used in most experimental studies. We present findings from a randomized, experimental evaluation of broken windows policing at hotspots in three California cities. Our analysis suggests that variation in the treatment delivered to target street segments within and between cities limited the ability of the study to detect potential treatment impacts and was due in part to the failure of the police agencies to take ownership of the science of the intervention.