Objectives Although male height is positively associated with many aspects of mate quality, average height men attain higher reproductive success in US populations. We hypothesize that this is because the advantages associated with taller stature accrue mainly from not being short, rather than from being taller than average. Lower fertility by short men may be a consequence of their and their partner's lower scores on aspects of mate quality. Taller men, although they score higher on mate quality compared to average height men, may have lower fertility because they are more likely to be paired with taller women, who are potentially less fertile. Methods We analyzed data from The Integrated Health Interview Series (IHIS) of the United States (N = 165,606). Segmented regression was used to examine patterns across the height continuum. Results On all aspects of own and partner quality, shorter men scored lower than both average height and taller men. Height more strongly predicted these aspects when moving from short to average height, than when moving from average to taller heights. Women of a given height who scored lower on mate quality also had shorter partners. Conclusions Shorter men faced a double disadvantage with respect to both their own mate quality and that of their spouses. Scores of taller men were only marginally higher than those of average height men, suggesting that being tall is less important than not being short. Although effect sizes were small, our results may partly explain why shorter and taller men have lower fertility than those of average stature. Am. J. Hum. Biol., 2014. © 2014 Wiley Periodicals, Inc.