Heavy machine tool work under such high-load conditions that chatter vibrations are prone to occur, which significantly diminishes the machining efficiency and quality. Stability lobe diagrams are commonly used to select appropriate spindle speed and axial depth of cut to get rid of chatter and maximize the material removal rate. However, this needs precise identification of the dynamics of the entire machine tool structure, especially in the low-frequency range. Operational modal analysis has been the proven technique for estimating dynamic characteristics of machine tool structures in operation conditions. In this article, a complete methodology was presented for employing operational modal analysis for heavy machine tool in machining conditions. A random cutting exciting method originally presented by Minis is modified which generates pseudorandom impulse force to excite a heavy vertical lathe structure. And the excitation signal of random cutting force was modeled to analyze the effect of cutting parameters on energy and frequency band of the excitation. One operational modal analysis method, the pLSCF (referred to as PolyMAX) method, was employed to estimate modal parameters during machining. It was also observed in chatter tests that the operational modal analysis results are more accurate than the traditional impact test results in characterizing the dynamics of machine tool structure in machining.