Complex nonlinear and nonstationary signals can be adaptively analyzed by the Hilbert–Huang transform through empirical mode decomposition and the Hilbert transform to generate the instantaneous energy. The instantaneous energy was able to display the local characteristics of the signals and had good time–frequency analysis capability, it is therefore widely applied to the analysis of vibration signals in the field of gear fault diagnosis. However, only a few extracted intrinsic mode functions through empirical mode decomposition can reflect fault feature or closely related to the faults but others are irrelevant. Therefore, the fault feature of the instantaneous energy for all intrinsic mode functions was not obvious and the accuracy of diagnosis was low. Aimed at solving this problem, a fault leading rate evaluation algorithm was proposed that can select those intrinsic mode functions, which reflect fault features (it was called the dominant intrinsic mode function) from all intrinsic mode functions. In the paper, this algorithm was applied to gear fault feature extraction. By calculating the instantaneous energy of the dominant intrinsic mode function the method could accurately extract gear fault feature and improve the accuracy of diagnosis. Both simulated signals and experimental signals of a Klingelnberg bevel gear were analyzed to verify the effectiveness and correctness of the algorithm.