In the context of the ongoing “migration crisis” the externalization of EU migration policy has continued. EU policy documents argue that cooperation with non‐EU countries is essential in order to manage migration flows. But how successful is this policy? The public policy literature teaches us that “policy success” can be defined in varying ways: as goal achievement; as political success; in terms of norms; in terms of the costs associated with the policy; temporally; and in the light of external factors. An application of this analytical framework to the EU Mobility Partnerships uncovers conceptual and methodological challenges, and above all highlights the need for evaluation of EU external migration policy to be taken more seriously.