Coumestrol is the one of the major phytoestrogens which is abundant in soybeans, legumes, brussel sprouts, and spinach. The beneficial effects of coumestrol are well known in various biological processes including; neuroprotective effects on the nervous system, function of the female reproductive system, anti‐bacterial properties, and anti‐cancer effects. Although the anti‐tumor activity of coumestrol has been demonstrated for ovarian, breast, lung, and cervical cancers, little is known of its effects on prostate cancer. Therefore, in the present study, we investigated the chemotherapeutic effects of coumestrol on two prostate cancer cell lines, PC3 and LNCaP. Our results showed that coumestrol decreased proliferation and migration and induced apoptosis in both PC3 and LNCaP cells. Moreover, effects of coumestrol on cell signaling pathways were investigated and it increased phosphorylation of ERK1/2, JNK, P90RSK, and P53 proteins in a dose‐ and time‐dependent manner whereas phosphorylation of AKT was reduced by coumestrol under the same conditions for culture of PC3 and LNCaP cells. In addition, mitochondrial dysfunction was induced by coumestrol as evidenced by a significant loss of mitochondrial membrane potential. Furthermore, cleavage of caspase‐3 and caspase‐9, the apoptotic proteins associated with mitochondria, also changed in response to coumestrol. Coumestrol also caused mitochondrial dysfunction resulting in an increase in ROS production in PC3 and LNCaP cells. These results suggest that coumestrol can inhibit progression of prostate cancer and may be a novel chemotherapeutic agent for treatment of prostate cancer via effects mediated via the PI3K/AKT and ERK1/2 and JNK MAPK cell signaling pathways. J. Cell. Physiol. 232: 862–871, 2017. © 2016 Wiley Periodicals, Inc. Our results indicated that coumestrol can inhibit progression of prostate cancer cells by affecting their viability and mitochondrial functions, and may be a novel chemotherapeutic agent for treatment of prostate cancer via effects mediated via the PI3K/AKT and ERK1/2 and JNK MAPK cell signal transduction cascades.