The nonlinear vibrations of the tapping-mode atomic force microscopy probe are investigated due to both nonlinearity in tip–sample contact force and curvature of the microcantilever probe. The nonlinear equations of motion for vibrations of the probe are obtained using Hamilton’s principle. In this work, the contact force is considered to be more dominant while previous works only consider Van der Waals force. The nonlinear contact force is expanded using a Taylor series to provide a polynomial with coefficients that are functions of the tip–sample distance. The outcome of this work allows the proper distance to be chosen before scanning to avoid instability in the response. Instability regions must be avoided for accurate imaging. The results show that initial tip–sample distance has a major effect on the stability of the frequency response and force response curves. For analytical investigation, the mode shapes of the atomic force microscopy probe are derived based on the presence of the nonlinear contact force as a boundary condition at the free end of the probe. The frequency response curve is obtained using the method of multiple scales. The results show that the effects of the nonlinearities due to probe geometry and contact force can be minimized. Minimizing the effects of nonlinearities allows for less cumbersome and calculation intensive software packages for atomic force microscopies. This research shows that one possible method of decreasing the nonlinearity effect is increasing the excitation force; however, this can increase the contact region and is not the best solution for canceling the nonlinearity effect. The superior method which is the major contribution of this paper is to find the optimal initial tip–sample distance and excitation force that minimize the nonlinearity effect. It is shown that at a certain tip–sample distance the quadratic and cubic nonlinearities cancel each other and the system responds linearly.