Focusing on tufting machine type DHUN801D-400, the complex dynamic model of coupling shaft system is built by using Riccati whole transfer matrix method, and the natural frequencies and mode shapes are analyzed. First, the components of coupling shafts system in tufting machine are introduced. Second, the structures of coupling shafts system are discretized and simplified. Third, the transfer matrix is constructed, the model is solved by using Riccati whole transfer matrix method, and then natural frequencies and mode shapes are obtained. Finally, the experimental results are quoted to demonstrate the applicability of the model. The results indicate that the Riccati whole transfer matrix method is well applicable for modeling the dynamics of complex multi-rotor systems.