In this paper, a perturbation observer-based adaptive passive control scheme is developed to provide great robustness of nonlinear systems against the unpredictable uncertainties and disturbances therein. The proposed scheme includes a high-gain perturbation observer and a robust passive controller. The high-gain perturbation observer is designed to estimate online the perturbation aggregated from the combinatorial effect of system nonlinearity, parameter uncertainty, unmodelled dynamics and fast time-varying external disturbances. Then the robust passive controller, using the estimated perturbation, can produce the minimal control effort needed to compensate for the magnitude of the actual current perturbation. Furthermore, the convergence of estimation error of the high-gain perturbation observer and the closed-loop system stability are analyzed theoretically. Finally, two practical examples are given to show the effectiveness and advantages of the proposed approach over the accurate model-based passive control scheme and the linearly parametric estimation-based adaptive passive control scheme.