Dyslexia is heritable and associated with phonological processing deficits that can be reflected in the event‐related potentials (ERPs). Here, we recorded ERPs from 2‐month‐old infants at risk of dyslexia and from a control group to investigate whether their auditory system processes /bAk/ and /dAk/ changes differently. The speech sounds were presented in an oddball paradigm. The children were followed longitudinally and performed a word reading fluency test in second grade. The infant ERPs were subsequently analyzed according to high or low reading fluency in order to find a neurophysiological precursor of poor reading fluency. The results show that the fluent reading children (from both the at‐risk and the control group) processed the speech sound changes differentially in infancy as indicated by a mismatch response (MMR). In the control group the MMR was frontally positive and in the fluent at‐risk group the MMR was parietally positive. The non‐fluent at‐risk group did not show an MMR. We conclude that at‐risk children who became fluent readers were better at speech processing in infancy than those who became non‐fluent readers. This indicates a very early speech processing deficit in the group of later non‐fluent readers. We recorded ERPs from 2‐month‐old infants at‐risk of dyslexia and from a control group to investigate whether their auditory system processes speech sound changes differently depending on their reading fluency in second grade. In both the fluent reading at‐risk and the control group we found an MMR to the speech sound changes whereas the non‐fluent at‐risk group did not show an MMR. We conclude that already at a very early age there is a speech processing deficit in children who later on become non‐fluent readers.