Gestures, hand movements that accompany speech, affect children's learning, memory, and thinking (e.g., Goldin‐Meadow, 2003). However, it remains unknown how children distinguish gestures from other kinds of actions. In this study, 4‐ to 9‐year‐olds (n = 339) and adults (n = 50) described one of three scenes: (a) an actor moving objects, (b) an actor moving her hands in the presence of objects (but not touching them), or (c) an actor moving her hands in the absence of objects. Participants across all ages were equally able to identify actions on objects as goal directed, but the ability to identify empty‐handed movements as representational actions (i.e., as gestures) increased with age and was influenced by the presence of objects, especially in older children.