Objective To identify the most robust methods for evaluating alternative payment models (APMs) in the emerging health care delivery system environment. Study Design (Approach) We assess the impact of widespread testing of alternative payment models on the ability to find credible comparison groups. We consider the applicability of factorial research designs for assessing the effects of these models. Principal Findings The widespread adoption of alternative payment models could effectively eliminate the possibility of comparing APM results with a “pure” control or comparison group unaffected by other interventions. In this new environment, factorial experiments have distinct advantages over the single‐model experimental or quasi‐experimental designs that have been the mainstay of recent tests of Medicare payment and delivery models. Conclusions The best prospects for producing definitive evidence of the effects of payment incentives for APMs include fractional factorial experiments that systematically vary requirements and payment provisions within a payment model.