The fault diagnosis of rotating machinery is quite important for the security and reliability of the overall mechanical equipment. As the main components in rotating machinery, the gear and the bearing are the most vulnerable to faults. In actual working conditions, there are two common types of faults in rotating machinery: the single fault and the compound fault. However, both of them are difficult to detect in the incipient stage because the weak fault characteristic signals are usually submerged by strong background noise, thus increasing the difficulty of the weak fault feature extraction. In this paper, a novel decomposition method, optimal resonance-based signal spares decomposition, is applied for the detection of those two types of faults in the rotating machinery. This method is based on the resonance-based signal spares decomposition, which can nonlinearly decompose vibration signals of rotating machinery into the high and the low resonance components. To extract the weak fault characteristic signals in the presence of strong noise effectively, the genetic algorithm is used to obtain the optimal decomposition parameters. Then, the optimal high and low resonance components, which include the fault characteristic signals of rotating machinery, can be obtained by using the resonance-based signal spares decomposition method with the optimal decomposition parameters. Finally, the high and the low resonance components are subject to the Hilbert transform demodulation analysis; the faults of rotating machinery can be diagnosed based on the obtained envelop spectra. The optimal resonance-based signal spares decomposition method is successfully applied to the analysis of the simulation and experiment vibration signals. The analysis results demonstrate that the proposed method can successfully extract the fault features in rotating machinery.