It has been proposed that language impairments in children with Autism Spectrum Disorders (ASD) stem from atypical neural processing of speech and/or nonspeech sounds. However, the strength of this proposal is compromised by the unreliable outcomes of previous studies of speech and nonspeech processing in ASD. The aim of this study was to determine whether there was an association between poor spoken language and atypical event‐related field (ERF) responses to speech and nonspeech sounds in children with ASD (n = 14) and controls (n = 18). Data from this developmental population (ages 6–14) were analysed using a novel combination of methods to maximize the reliability of our findings while taking into consideration the heterogeneity of the ASD population. The results showed that poor spoken language scores were associated with atypical left hemisphere brain responses (200 to 400 ms) to both speech and nonspeech in the ASD group. These data support the idea that some children with ASD may have an immature auditory cortex that affects their ability to process both speech and nonspeech sounds. Their poor speech processing may impair their ability to process the speech of other people, and hence reduce their ability to learn the phonology, syntax, and semantics of their native language. As a group, children with ASD show atypical neural auditory processing of both speech and nonspeech sounds, which is linked to overall spoken language ability. Children with ASD also show immature auditory brain responses at the group level. Within the ASD group, poor language processing is strongly associated with dissimilar speech and nonspeech responses in the left hemisphere (within the 200–400ms higher cognitive processing time window). An impacted or immature auditory processing system may contribute to less stable representations of sounds, and interfere with the development of spoken language perception and production in ASD.