Research summary: A recent study by Fitza argued that the prior estimates of the Chief Executive Officer (CEO) effect are conflated with events outside the CEO's control, are largely the result of random chance, and that the true CEO effect is smaller than has been previously estimated. We suggest that the empirical methodology employed by Fitza to support these claims substantially overstates the “random chance” element of the CEO effect. We replicate Fitza's findings, highlight methodological issues, offer alternative conclusions, and using multilevel modeling (MLM), suggest that his analyses mischaracterize the CEO effect. Managerial summary: Scholars and practitioners have debated for decades about the relative importance of Chief Executive Officers (CEOs) and the magnitude of the impact they have on firm outcomes. Clearly, decisions related to compensation, retention or termination, and leadership development all hinge on the assumptions made about the relative impact corporate leaders have on firm performance. Responding to earlier work that claimed the “CEO effect” was greatly exaggerated in past work and likely quite small, this study seeks to reaffirm the significant impact CEOs have on firm outcomes. Copyright © 2016 John Wiley & Sons, Ltd.