We compared loss and gain in communication from 1 to 2 years in children later diagnosed with autism spectrum disorder (n = 41), language impairment (n = 110) and in children with typical language development at 7 years (n = 831). Participants were selected from a prospective population cohort study of child language (the Early Language in Victoria Study). Parent-completed communication tools were used. As a group, children with autism spectrum disorder demonstrated slower median skill gain, with an increasing gap between trajectories compared to children with typical development and language impairment. A proportion from all groups lost skills in at least one domain (autism spectrum disorder (41%), language impairment (30%), typical development (26%)), with more children with autism spectrum disorder losing skills in more than one domain (autism spectrum disorder (47%), language impairment (15%, p = 0.0003), typical development (16%, p < 0.001)). Loss was most common for all groups in the domain of ‘emotion and eye gaze’ but with a higher proportion for children with autism spectrum disorder (27%; language impairment (12%, p = 0.03), typical development (14%, p = 0.03)). A higher proportion of children with autism spectrum disorder also lost skills in gesture (p = 0.01), sounds (p = 0.009) and understanding (p = 0.004) compared to children with typical development but not with language impairment. These findings add to our understanding of early communication development and highlight that loss is not unique to autism spectrum disorder.