It is unclear how music training leads to superior memory of language. In the present study, we investigated whether musically trained adults (musicians) have superior segmental and tonal loops using non-musical Mandarin stimuli. Forty-three musicians and thirty-nine demographically matched non-musically trained adults (non-musicians) participated in this study, all native Chinese. Memory spans, typical indicators of capacity of the phonological loop, were measured in both visual and auditory modalities in the following three conditions: (1) a segmental condition, defined as different syllables with the same high level tone; (2) a tonal condition (suprasegmental condition), defined as the same syllable with different tones; and (3) a mixed condition, defined as different syllables with different tones. The results revealed a main effect of condition. Memory spans of the tonal conditions were significantly smaller than those of segmental and mixed conditions, regardless of group and modality. Moreover, a significant condition by group interaction was found. Musicians outperformed non-musicians in the tonal conditions, but not in the segmental or mixed conditions in both modalities. These findings suggest that there are tonal and segmental loops for Mandarin, and musicians, compared to controls, have larger memory spans for Mandarin tones but not segments.