The purpose of this paper is to present a novel combined design of the voltage source converter–high voltage direct current (VSC-HVDC) and the power system stabilizer (PSS) controllers to obtain a better dynamical response. The proposed technique is applied to enhance the damping of the power system low-frequency oscillations (LFOs) and results are compared with traditional design. A fuzzy logic controller is designed for PSS (FPSS). Then, a chaotic optimization algorithm, which has a strong ability for finding the most optimistic results, is employed, in presence of FPSS, to search for optimal VSC-HVDC output feedback controller parameters. Moreover, a singular value decomposition method is utilized to select the most effective damping control signal of the VSC-HVDC output feedback controllers. The novel proposed controllers are evaluated on an AC/DC power system. The simulation results demonstrate that the combined controllers have an excellent capability for damping power system LFOs and greatly enhance the dynamic stability of the power system. Also, the system performance analysis under different operating conditions and some performance indices show the effectiveness of the proposed controllers. The benefit of the suggested procedure is greatly improving the dynamic response of the system. In addition, the overshoots, undershoots and the settling times are dramatically reduced by applying the proposed method.