Recent research has shown that both language and number processing are clear examples of distributed and connected processing in the human brain, emphasizing the importance of white matter connections between the associated cortex sites. Against this background we hypothesized joint cognitive processes and functions in a cross‐domain manner to be reflected by the involvement of specific white matter tracts. Therefore, we evaluated white matter connectivity for the specific cognitive process of semantic classification, which is an integral part of tasks commonly employed to investigate the neural correlates of language and number processing. In line with our expectations, fiber tracking results clearly indicated a common ventral network for semantic classification for the domains of language and number processing. Thereby, the present data are hard to reconcile with a localizationalist view on processing characteristics of the human brain, but strongly suggest that white matter connectivity should be considered when investigating the neural underpinnings of human cognition.