The P1m component of the auditory evoked magnetic field is the earliest cortical response associated with language acquisition. However, the growth curve of the P1m component is unknown in both typically developing (TD) and atypically developing children. The aim of this study is to clarify the developmental pattern of this component when evoked by binaural human voice stimulation using child‐customized magnetoencephalography. A total of 35 young TD children (32–121 months of age) and 35 children with autism spectrum disorder (ASD) (38–111 months of age) participated in this study. This is the first report to demonstrate an inverted U‐shaped growth curve for the P1m dipole intensity in the left hemisphere in TD children. In addition, our results revealed a more diversified age‐related distribution of auditory brain responses in 3‐ to 9‐year‐old children with ASD. These results demonstrate the diversified growth curve of the P1m component in ASD during young childhood, which is a crucial period for first language acquisition. Autism Res 2016, 9: 1216–1226. © 2016 International Society for Autism Research, Wiley Periodicals, Inc.