The ultrasonic impact treatment process is widely used to improve the fatigue life of the weldments by inducing compressive residual stresses at the sub-surface. The purpose of the article is to conduct the dynamic elastic–plastic finite element analysis of multiple impacts on 5A06 aluminum alloy with different controlled parameters. The numerical model was validated by pin drop test. The changes in penetration depth, maximum compressive residual stress, and surface residual stress were obtained by analyzing the residual stress field and equivalent plastic strain. The effect of impact times, impact velocities, pin shapes, and impact angles on the residual stress was investigated so that the ultrasonic impact treatment parameters could be controlled to obtain expected residual stress distributions.