It is unclear what roles native language (L1) and second language (L2) play in the perception of lexical tones in a third language (L3). In tone perception, listeners with different language backgrounds use different fundamental frequency (F0). While English listeners use F0 height, Mandarin listeners rely more on F0 direction. The present study addresses whether knowledge of Mandarin, particularly as an L2, results in speakers’ reliance on F0 direction in their perception of L3 (Cantonese) tones. Fifteen English-speaking L2 learners of Mandarin constituted the target group, and 15 English monolinguals and 15 native Mandarin speakers, with no background in other tonal languages, were included as control groups. All groups had to discriminate Cantonese tones either by distinguishing a contour tone from a level tone (F0 direction pair) or a level tone from another level tone (F0 height pair). The results showed that L2 learners patterned differently from both control groups by using F0 direction as well as F0 height under the influence of L1 and L2 experience. The acoustics of the tones also affected all listeners’ discrimination. When L2 and L3 are similar in terms of the presence of lexical tone, L2 experience modulates the perception of L3 tones.