This paper investigates a leader-following formation control problem for second-order multi-agent systems with nonuniform time-varying communication delays under directed topologies. We first propose a consensus protocol and give a sufficient condition for second-order consensus of the system. Then, under a framework of multiple leaders, the protocol is applied to the formation control, including time-invariant formation and time-varying formation as well as time-varying formation for trajectory tracking. It is shown that the agents will attain the desired formation under the protocol. Finally, several simulations are conducted to illustrate the effectiveness of our theoretical results.