During the operation process of a gearbox, the vibration signals can reflect the dynamic states of the gearbox. The feature extraction of the vibration signal will directly influence the accuracy and effectiveness of fault diagnosis. One major challenge associated with the extraction process is the mode mixing, especially under such circumstance of intensive frequency. A novel fault diagnosis method based on frequency-modulated empirical mode decomposition is proposed in this paper. Firstly, several stationary intrinsic mode functions can be obtained after the initial vibration signal is processed using frequency-modulated empirical mode decomposition method. Using the method, the vibration signal feature can be extracted in unworkable region of the empirical mode decomposition. The method has the ability to separate such close frequency components, which overcomes the major drawback of the conventional methods. Numerical simulation results showed the validity of the developed signal processing method. Secondly, energy entropy was calculated to reflect the changes in vibration signals in relation to faults. At last, the energy distribution could serve as eigenvector of support vector machine to recognize the dynamic state and fault type of the gearbox. The analysis results from the gearbox signals demonstrate the effectiveness and veracity of the diagnosis approach.