Children with autism vary widely in their language abilities, yet the neural correlates of this language variability remain unclear, especially early in development. Diffusion tensor imaging (DTI) was used to examine diffusivity measures along the length of 18 major fiber tracts in 104 preschool‐aged boys with autism spectrum disorder (ASD). The boys were assigned to subgroups according to their level of language development (Low: no/low language, Middle: small vocabulary, High: large vocabulary and grammar), based on their raw scores on the expressive language (EL) and receptive language (RL) sections of the Mullen Scales of Early Learning (MSEL). Results indicate that the subgroups differed in fractional anisotropy (FA), mean diffusivity (MD), and radial diffusivity (RD) along the inferior longitudinal fasciculus (ILF) in both hemispheres. Moreover, FA correlated significantly with Mullen EL and RL raw scores, but not ADOS severity score, along the left and right ILF. Subgroups also differed in MD (but not FA) along the left superior longitudinal fasiculus and left corticospinal tract, but these differences were not correlated with language scores. These findings suggest that white matter microstructure in the left and right ILF varies in relation to lexical development in young males with ASD. The findings also support the use of raw scores on language‐relevant standardized tests for assessing early language‐brain relationships. Autism Res 2017. © 2017 International Society for Autism Research, Wiley Periodicals, Inc.