This article provides a thorough empirical analysis of random variation in shared savings arrangements. It uses claims data from seven provider coalitions that applied for certification to become Medicaid accountable care organizations (ACOs) in New Jersey to conduct Monte Carlo simulations under varying assumptions about true ACO savings. Among all the ACOs examined, the observed savings rate can be several percentage points higher or lower than the assumed true savings rate, leading to large probabilities of Type I and Type II error in determining the existence of savings. Although the effects of random variation are smaller for larger ACOs, the ACO-level coefficient of variation in health care spending also stands out as a highly relevant parameter. The risks of overpayment and underpayment can be minimized through modified specification of the savings measurement methodology. These findings have implications for the terms of shared savings arrangements negotiated between payers and providers.