This article aims to present an adaptive and robust cooperative visual localization solution based on stereo vision systems. With the proposed solution, a group of unmanned vehicles, either aerial or ground will be able to construct a large reliable map and localize themselves precisely in this map without any user intervention. For this cooperative localization and mapping problem, a robust nonlinear H filter is adapted to ensure robust pose estimation. In addition, a robust approach for feature extraction and matching based on an adaptive scale invariant feature transform stereo constrained algorithm is implemented to build a large consistent map. Finally, a validation of the solution proposed is presented and discussed using simulation and experimental data.