Research summary How much of the variance in firm performance can be attributed to CEOs? This question has been the focus of a long debate in management research. In a recent study I showed that a large portion of the performance differences that are often attributed to CEOs might in fact be due to chance. In a recent paper Quigley and Graffin argue that my conclusions can be avoided if more advanced methodological approaches are applied. Here I show that this is not the case, in fact if more realistic assumptions of how chance can affect firm performance are made the effect of CEO leadership is almost indistinguishable from the effect of chance, independent of the estimation methodology. Managerial summary Does it matter who the CEO of a company is? To what degree do CEOs influence firm performance? The answer to such questions has important consequences for firms, effecting for example how CEOs are chosen or how they are paid. Most studies that tried to quantify the degree to which CEOs influence firm performance have found relatively large CEO effects. I argue here that firm performance is influenced by a complex set of factors including the effect of chance (or luck) and that empirically past studies wrongly attribute the performance effect of such chance to the CEO. I show that if chance is properly taken into account the true influence of CEOs is in fact relatively small.