The flexibility of the guiding rope and the tension difference between two guiding ropes cause the lateral and torsional vibrations of the hoisting conveyance in the rope-guided hoisting system, respectively, which are theoretically investigated with two different cases in this paper. The assumed modes method is used to discretize the hoisting rope and two guiding ropes, and Lagrange equations of the first kind is adopted to derive the equations of motion, while the geometric matching conditions at the interfaces of the ropes are accounted for by the Lagrangian multiplier. Considering all the geometric matching conditions are approximately linear, the differential algebraic equations are transformed to a system of ordinary differential equations. The current method can obtain not only the accurate lateral displacements of two guiding ropes, but also the constraint forces between the hoisting conveyance and two guiding ropes. Further, the current method is verified by the ADAMS simulation. Finally, the effects of various parameters on the lateral and torsional vibrations of the hoisting conveyance are analyzed and results indicate that the appropriate tension difference and distance difference could decrease the maximum lateral displacement, which is useful to design super deep rope-guided hoisting system for the decrease of the vibration.