The global financial crisis exposed financial institutions to severe unexpected losses in relation to mortgage securitizations and derivatives. This article finds that risk models such as ratings are exposed to a large degree of systematic risk and parameter uncertainty. An out‐of‐sample forecasting exercise of the financial crisis shows that a simple approach addressing both issues is able to produce ranges for risk measures consistent with realized losses. This explains how financial markets were taken by surprise in relation to realized losses.