Risk ladders have the potential to improve numeric judgments of low‐likelihood events by providing information about the likelihoods of comparison risks, thereby letting respondents make risk estimates “in context.” However, to date this tool has been studied systematically only in communication of risk, not in elicitation of perceived likelihoods. In three studies, we evaluated the benefits of risk ladders on the consistency, validity, and mean‐level accuracy of elicited likelihood judgments. When estimates for low‐likelihood hazards were elicited using different numeric response scales (e.g., “1 in x” and “x in 100,000”), scale type had a strong effect on the magnitudes of the elicited estimates, and viewing a risk ladder (Experiment 1) or comparison risks (Experiments 2 and 3) did not attenuate this effect of scale type. Similarly, we found no evidence that risk ladders or comparison risks improved the convergent validity of numeric estimates, as measured using correlations with risk ratings made on alternative scale types. Finally, viewing comparison risks tended to reduce gross overestimation of rare events, with relatively less change in estimates for mid‐likelihood and high‐likelihood hazards. This suggests that comparison risks can spread responses to cover a wider range of values but do not ameliorate scale effects. In the elicitation of numeric risk estimates, how you ask matters, even if you let people make estimates “in context” through the use of comparative risk information. Copyright © 2015 John Wiley & Sons, Ltd.