Torsional stiffness of a rotary table plays an important role in the static and dynamic characteristics of a rotary feed drive system. This paper proposes an effective approach to estimate the torsional stiffness of a worm geared transmission usually employed in rotary table. First, the stiffness models of each component used in the transmission are extracted. Then, an expression for the coefficient relating angular displacements is derived and a general torsional stiffness model for the rotary feed drive is developed, taking the time-varying mesh stiffness into account. Finally, a stiffness test scheme is presented and conducted to verify the proposed stiffness model. Furthermore, the influences of the gear’s parameters on the resultant torsional stiffness are investigated based on the developed model. Results indicate the necessity to incorporate a time-varying mesh stiffness when the torsional stiffness of the rotary table is under estimation. Also, a high-frequency variation in profile of torsional stiffness is induced by the mesh stiffness of gear pairs at the high-speed stage. Parametric studies show that tooth width and number of teeth of the driven gear at low-speed stage are more sensitive to stiffness than those at high-speed stage. However, the number of teeth of the driving gears introduces different effects onto the torsional stiffness. The presented model can be used to estimate the torsional stiffness of a rotary feed system efficiently, especially during the preliminary design stage.