An auditory selective attention set allows one to enhance the processing of goal‐relevant sound events, which is reflected by the enhancement of the N1 event‐related potential (ERP). The present study investigated whether the sensory consequences of distraction (i.e., involuntary attention changes triggered by infrequent sensory events) can be revealed as the removal of this attentional ERP enhancement. Continuous tones featuring occasional gaps were presented, and participants performed a gap‐detection task. Independently from gaps, abrupt pitch changes (glides) were introduced, either rarely or frequently, in separate conditions. Whereas rare glides preceding gaps by 150 ms strongly impacted gap‐detection performance and gap‐related N1 amplitudes, their impact on gaps following rare glides by 650 ms was significantly smaller in both measures. This result demonstrates the utility of N1 in probing the sensory impact of auditory distraction.