Previous studies have shown that language contributes to humans' ability to orient using landmarks and shapes their use of frames of reference (FoRs) for memory. However, the role of environmental experience in shaping spatial cognition has not been investigated. This study addresses such a possibility by examining the use of FoRs in a nonverbal spatial memory task among residents of an Andean community in Peru. Participants consisted of 97 individuals from Ancash Quechua‐speaking households (8–77 years of age) who spoke Quechua and/or Spanish and varied considerably with respect to the extent of their experience in the surrounding landscape. The results demonstrated that environmental experience was the only factor significantly related to the preference for allocentric FoRs. The study thus shows that environmental experience can play a role alongside language in shaping habits of spatial representation, and it suggests a new direction of inquiry into the relationships among language, thought, and experience.