The cost of ruling effect on electoral support is well established. That is, governing parties tend to lose vote share the longer they are in power. Although we know this to be true, we do not know why it happens. This research examines whether the cost of ruling results at least in part from the tendency for governing parties to shift policy further away from the average voter. It first considers differences in political institutions and how they might influence cost of ruling owing to policy drift, and then tests the hypothesis focusing on U.S. presidential elections, which is an unfavorable case to find such an effect. Results confirm a clear cost of ruling effect in these elections and demonstrate that policy misrepresentation is an important mechanism. That is, the policy liberalism of presidents from different parties diverges over time as their tenure in the White House increases, and the degree to which it does matters for the presidential vote. Policy is not the only thing that matters, and other factors, in particular the economy, are more powerful. From the point of view of electoral accountability, however, the results do provide good news, as they indicate that substantive representation is important to voters. Elections are not simply games of musical chairs.