The decision whether to explore new alternatives or to choose from familiar ones is implicit in many of our daily activities. How is this decision made? When will deviation from optimal exploration be observed? The current paper examines exploration decisions in the context of a multi‐alternative “decisions from experience” task. In each trial, participants could choose a familiar option (the status quo) or a new alternative (risky exploration). The observed exploration rates were more sensitive to the frequent outcome from choosing new alternatives than to the average outcome. That is, the implicit decision whether to explore a new alternative reflects underweighting of rare events: Over‐exploration was documented in “Rare Disasters” environments, and insufficient exploration was evident in “Rare Treasures” environments. In addition, the results reveal a decrease in exploration of new alternatives over time even when it is always optimal and some exploration even when it is never reinforcing. These results can be captured with models that share a distinction between “data collection” and “outcome‐driven” decision modes. Under the data collection mode, the decision maker collects information about the environment, to be used in future choices. Under the outcome‐driven mode, the decision maker relies on small samples of previous experiences with familiar versus unfamiliar alternatives, before the selection of a specific alternative. The predictive value of a two‐parameter “explorative sampler” quantification of these assumptions is demonstrated. Copyright © 2013 John Wiley & Sons, Ltd.