In this paper, a method of measuring the relative torsional angles of the flexible coupling under working condition was developed. A double-encoder measurement system was proposed to get the torsional angles. The torsional angles were divided into "constant torsional angle" and "alternating torsional angle". The measured results were analyzed further, which were applied to a series of flexible couplings. Firstly, the torsional stiffness of D-type flexible coupling was obtained according to the relationship between torques and torsional angles. The proposed method overcomes the difficulty that the angular displacements of flexible coupling cannot be measured online under operating condition. The feasibility and accuracy of the method developed in this study was verified by comparing stiffness curve with unloading curve obtained from static torsional experiment and values from factory inspection report. Furthermore, in view of the tear fault of A-type flexible coupling in a diesel generator set, the maximum alternating torsional angles of A-type, B-type, and C-type flexible couplings were measured under working condition, then safety margins were obtained for the diesel generator set. According to the safety margins of the three kinds of couplings, C-type flexible coupling is chosen to replace A-type, which provided the basis for type selection of flexible coupling. Finally, the method developed in this study would also be applied to the operation monitoring and fault diagnosis of flexible coupling online.