By comparing faces with Chinese characters, the category‐sensitive and orientation‐sensitive N170 adaptation effects were examined when the stimuli were adapted by category (within vs. between) and orientation (same vs. different). The category‐sensitive N170 adaptation was present for both faces and characters, supporting the assertion that the perception of faces and characters recruits domain‐specific processing rather than general processing of visual expertise. In addition, the orientation‐sensitive N170 adaptation was present only for faces but not for characters, suggesting that only faces recruit orientation‐specific processing. More importantly, the orientation‐sensitive N170 adaptation was not only present for inverted faces but also for upright faces, indicating that there are distinct neuron populations respectively sensitive to upright and inverted faces.