Elastic connection of spindle–tool holder is one of the weak points of spindle systems. It directly affects dynamic characteristics of machining system. Stiffness modeling of spindle–tool holder is the key problem in dynamic analysis of spindle system. Considering the influence of cutting force, an analytical method combining classic elasticity theory with fractal theory is proposed to estimate the contact stiffness at the spindle–tool holder interface. The effectiveness of the proposed method is validated by self-designed experiment. An approximate model is proposed to replace finite element model in the processing of identification of contact stiffness of spindle–tool holder in order to save calculation time. On that basis, effects of cutting force on contact stiffness of spindle–tool holder are investigated in this article. The conclusion can be drawn that cutting force has great effect on stiffness of spindle–tool holder. Contact stiffness of spindle–tool holder can be obtained by the method proposed in this research on the premise of not making prototypes or utilizing hammer-hitting pulse-inspirit method. The calculating model and method in this article can be implemented to solve the contact stiffness at the spindle–tool holder during the design stage.