Error compensation technique is a recognized and cost-effective method to improve machining accuracy of machine tools. In this article, a new compensation method for geometric error is proposed based on Floyd algorithm and product of exponential screw theory. Based on topological structure and measured data, volumetric geometric error modeling is established by product of exponential screw theory. Then, the improved Floyd minimum-distance method was used to establish an error compensation model by adjusting weight unceasingly. In order to verify the effectiveness and generality of the method proposed in this article, two experiments were designed. A total of 5 five-axis machining centers of the same type with different use time were selected to carry out the simulation experiments. Results show that the Floyd method can provide higher compensation precision, that is, Floyd algorithm compensation method can keep positioning errors within the range [–8 µm, 9 µm]. In addition, roundness error, coaxial error, and surface roughness were reduced in the actual machining experiments of two machined conical tables. Therefore, it can be seen that the proposed compensation method is effective to improve machining accuracy of machine tools.