In this paper, we used a Qball-X4 quad-rotor unmanned aerial vehicle (UAV) which was developed by the Quanser Company as the experimental platform. First, a fundamental mathematical model of the Qball-X4 quad-rotor UAV was built and a simulation model was set up based on the proposed mathematical model; then, a double closed-loop optimal proportional–integral–derivative (PID) controller based on integral of time multiplied by absolute error (ITAE) indices was designed according to the model structure. In consideration of the possible system error and data delay, we designed a corresponding Kalman filter, which can estimate the target trajectory and be put before the proposed PID controller to ensure their validity. Finally, simulation results of the system with presented PID controller and Kalman filter were shown to verify their effectiveness.