Background: Higher physical activity (PA) levels are known to be associated with lower risk of death. Less attention, however, has been paid to directly evaluate the effect of PA on the time by which a certain fraction of the population has died. Methods: A population-based cohort of 29,362 men 45 to 79 years of age was followed from January 1998 to December 2010. A total of 4,570 men died. PA was assessed through a self-administrated questionnaire. Adjusted differences in the number of months by which 10% (10th percentile) of the cohort has died, according to levels of total PA (TPA) and different domains of PA were estimated using Laplace regression. Results: Overall, the 10th survival percentile was 9.6 years, that is, 90% of the cohort lived longer than 9.6 years. We found a strong evidence of non-linearity between TPA and the 10th survival percentile (P-value < 0.001). Compared to men with the lowest TPA (29 metabolic equivalents (MET)-hrs/day), men with a median TPA (41 MET-hrs/day) had 30 months longer survival (95% CI: 25–35). Below the median TPA, every increment of 4 MET-hrs/day, approximately a 30 minutes brisk pace daily walk, was associated with a longer survival of 11 months (95% CI: 8–15). Above the median TPA additional activity was not significantly associated with better survival. Conclusions: We found that a physically active lifestyle is associated with a substantial improvement in survival time, up to 2.5 years over 13 years of follow-up.