Thanks to its superior mechanical and physical properties, silicon carbide is a promising mold material in glass molding of micro-structured optical elements. However, the high micro-structured surface quality can be hardly generated by means of conventional abrasive polishing because of the high hardness of silicon carbide. In this article, the ultrasonic vibration was introduced to assist abrasive polishing with an aim to improve the cylindrical groove arrays’ surface quality and also to increase the polishing efficiency. First, the comparison experiments between abrasive polishing and ultrasonic vibration–assisted polishing were conducted, and then, the factors affecting ultrasonic vibration–assisted polishing performance were investigated. The experimental results indicate that through conventional abrasive polishing, the surface roughness of silicon carbide cylindrical groove arrays decreases to 25.5 nm from the precision ground surface roughness (Ra) of 115.6 nm, while through ultrasonic vibration–assisted polishing, the surface roughness decreases to 8.6 nm with a factor of three times improvement. Furthermore, a 35-kHz vibration frequency corresponds to a smaller surface roughness than a 25-kHz vibration frequency under identical processing parameters. In the vibration amplitude range from 1.0 to 2.5 µm, the surface roughness accordingly decreases linearly, while in the range of 2.5–4.0 µm, ultrasonic vibration presents no assistance in improving the surface quality.