Numerous studies have attempted to identify early physiological abnormalities in infants and toddlers who later develop autism spectrum disorder (ASD). One potential measure of early neurophysiology is the auditory brainstem response (ABR), which has been reported to exhibit prolonged latencies in children with ASD. We examined whether prolonged ABR latencies appear in infancy, before the onset of ASD symptoms, and irrespective of hearing thresholds. To determine how early in development these differences appear, we retrospectively examined clinical ABR recordings of infants who were later diagnosed with ASD. Of the 118 children in the participant pool, 48 were excluded due to elevated ABR thresholds, genetic aberrations, or old testing age, leaving a sample of 70 children: 30 of which were tested at 0–3 months, and 40 were tested at toddlerhood (1.5–3.5 years). In the infant group, the ABR wave‐V was significantly prolonged in those who later developed ASD as compared with case‐matched controls (n = 30). Classification of infants who later developed ASD and case‐matched controls using this measure enabled accurate identification of ASD infants with 80% specificity and 70% sensitivity. In the group of toddlers with ASD, absolute and interpeak latencies were prolonged compared to clinical norms. Findings indicate that ABR latencies are significantly prolonged in infants who are later diagnosed with ASD irrespective of their hearing thresholds; suggesting that abnormal responses might be detected soon after birth. Further research is needed to determine if ABR might be a valid marker for ASD risk. Autism Res 2016, 9: 689–695. © 2015 The Authors Autism Research published by Wiley Periodicals, Inc. on behalf of International Society for Autism Research