This study aimed to investigate the development of visual and auditory temporal processing among children with and without dyslexia and to examine the roles of temporal processing in reading and reading-related abilities. A total of 362 Chinese children in Grades 1–6 were recruited from Taiwan. Half of the children had dyslexia, and the other half were typically developing children who matched the dyslexic group on age, intelligence, and gender. Our results indicate that for typically developing children, the visual and auditory modalities follow the same developmental trend: The children in first and second grades performed significantly worse than the older children. Among the children with dyslexia, however, significant improvements in the visual modality were observed with increasing age. Furthermore, although both modalities were important for all reading-related abilities and for Chinese character reading in first and second grades, the visual modality significantly predicted only orthographic knowledge and Chinese character reading in third and fourth grades. In contrast, the auditory modality affected only phonological awareness. In fifth and sixth grades, only visual temporal processing slightly contributed to the orthographic knowledge and Chinese character reading of the dyslexic group. Also, the relationship between temporal processing and Chinese character reading is strongly influenced by age.