Research summary: Raters of firms play an important role in assessing domains ranging from sustainability to corporate governance to best places to work. Managers, investors, and scholars increasingly rely on these ratings to make strategic decisions, invest trillions of dollars in capital, and study corporate social responsibility (CSR), guided by the implicit assumption that the ratings are valid. We document the surprising lack of agreement across social ratings from six well‐established raters. These differences remain even when we adjust for explicit differences in the definition of CSR held by different raters, implying the ratings have low validity. Our results suggest that users of social ratings should exercise caution in interpreting their connection to actual CSR and that raters should conduct regular evaluations of their ratings. Managerial summary: Ratings of corporate social responsibility (CSR) guide trillions of dollars of investment, but managers, investors, and researchers know little about whether these ratings accurately measure CSR. In practice, there are examples of highly rated firms becoming embroiled in scandals and the same firm receiving sharply different ratings from different rating agencies. We evaluate six of the leading raters and find little overlap in their assessments of CSR. This lack of consensus suggests that social responsibility is challenging to measure reliably and that users of these ratings should be cautious in drawing conclusions about firms based on this data. We encourage the rating agencies to regularly validate their data in an effort to improve the measurement of CSR. Copyright © 2015 John Wiley & Sons, Ltd.