Across cultures and age groups music has a powerful impact on human affective states. We examined the effect of these affective responses on children’s and adults’ ability to label musical excerpts as major or minor. Adults, 10-year-olds, and 5-year-olds rated affective quality of excerpts that differed by mode, tempo, pitch, and excerpt type and then categorized the excerpts by mode. Trial-by-trial assessment of category judgments indicated performance was accurately predicted by subjects’ association of affective valence with musical properties. Specifically, strength of this association prior to training predicted greater category knowledge, types of errors, and age differences in learning.