In this study, we evaluate how voter polarization and the level of partisanship influence electoral outcomes. We show that when the level of partisanship is low, the polarization of voter preferences translates into popular support for extreme parties. In contrast, longstanding attachments to mainstream (moderate) parties dampen the relationship between voter polarization and support for extreme parties. The implication of these findings is that the lack of voter attachment to parties contributes to extreme party competition, while strong attachment can help reduce party extremism even if electorates are polarized.