Objectives The Endangered collared brown lemur (Eulemur collaris) is the largest primate living in the littoral forest of southeastern Madagascar, a top priority habitat for biodiversity conservation on the island. Because this lemur is a key seed‐disperser, an evaluation of the structure and connectivity of the populations surviving in the forest fragments is urgently needed to guide conservation plans. Materials and Methods Genetic variability at autosomal microsatellites and mitochondrial DNA was investigated in a total of 49 collared brown lemurs sampled by non‐invasive methods in three littoral forest fragments and in the nearby lowland humid forest. Results The overall genetic diversity of E. collaris in the southeastern coastal region of Madagascar was lower than in other populations, as well as in other lemur species. The population appears highly structured, with less variable and more inbred groups inhabiting the littoral forest fragments compared to the inland area. Major barriers to gene flow were identified isolating littoral forest fragments from each other and from the inland lowland humid forest. Discussion Medium to long‐term drift and scarce gene flow is the scenario that best explains the current genetic distribution. Habitat discontinuities such as rivers and grassland between forest fragments played a major role in structuring the population. A common history of size contraction is pointed out by several genetic estimators, indicating a possible ecological crisis triggered around 1,300 years ago. The adoption of strategies aimed at facilitating gene flow and population growth appears crucial to delay further loss of genetic diversity.