Physical activity has been shown to reduce colorectal cancer risk. However, the association of physical activity levels separately with colon and rectum cancer mortality is less clear. We included 226 089 Korean in a large prospective cohort study. We assessed the associations of physical activity levels (length, frequency, and intensity) with colon and rectum cancer mortality using Cox proportional hazards model. Men, but not women, who had regular physical activity showed a lower risk of death from colon cancer. The association for high intensity of physical activity was borderline significant in men (hazard ratio 0.67; 95% CI = 0.45-1.00 for >17.5 MET-hour/week). The associations were largely consistent among those aged <55 years, body mass index <25 kg/m², low high-density lipoprotein cholesterol, current smoker, and without hypertension or history of cardiovascular disease. In conclusion, higher length or intensity of physical activity decreased colon, but not rectum cancer mortality in men. Regular physical activity may have beneficial effects on colon cancer survival.