We know that most House seats remain within the same party over the course of a redistricting decade. For example, over 75% did so in the last decade. This gives rise to the question: "Why do some seats change hands and others not?" We seek to go beneath the standard answers (such as extent of electoral vulnerability as indicated by the previous victory margin, challenger qualifications, relative spending of challenger and incumbent, midterm loss affecting districts newly won by the president’s party, realignment effects that made Democrats in the South vulnerable) to examine the conditions of ideological competition that affect each of these factors and the concomitant probability of electoral defeat. We offer a general model of unidimensional party competition across multiple constituencies, where a party "chases the outliers" of the other party that are closest to its own ideological mean, thus eliminating "anomalous" districts which should be vulnerable to change in party control, and we test that model with data from the U.S. House of Representatives 1980–2006. Over time, Democrats capture liberal and moderately liberal districts held by Republicans, while Republicans capture conservative and moderately conservative districts held by Democrats. In a neo-Downsian world where candidates do not locate at the preferences of the median voter in the district but, rather, are shifted in the direction of their own party mean, we show that this outlier-chasing dynamic can be expected, in the long run, to "empty" out the center. This results in an equilibrium of ideologically distinct parties and a high level of polarization, involving a self-reinforcing dynamic in which the seats that become vulnerable change as the parties become more distinct. Indeed, rather than puzzling about why so much polarization exists, our work suggests that the real puzzle is why it has taken so long to get to the level of polarization we presently enjoy. We suggest that the combination of incumbency advantage, and multidimensionality of political competition might be the answer, with the Civil War role of race as an independent dimension slowly wearing off.