The edge chipping of holes, which is induced by mechanical machining, restricts the applications of brittle materials. Rotary ultrasonic machining is considered a suitable approach to machine holes in brittle materials with a smaller edge-chipping size. However, obvious edge chipping at the hole exit in rotary ultrasonic machining remains observable. In this study, conical diamond core drills with various characteristic angles () were designed to further reduce the edge-chipping size for rotary ultrasonic machining. Machining tests on quartz glass were conducted to evaluate the effectiveness of this new type of drill. Experimental results show that the conical drill can obviously reduce the edge-chipping size only when certain conditions are satisfied. The mechanism of edge-chipping reduction using a conical drill was revealed by the theoretical analysis and detailed observation of the thrust force and obtained cylinder. To guarantee the feasibility of the conical drill, its characteristic angle should exceed a critical value at a certain feed rate. A higher feed rate requires a higher critical characteristic angle. The other advantage of the conical drill is its ability to suppress the bad effects of increasing the feed rate on the stability of ultrasonic vibration.