Drawing on interview data from three countries (Australia, United States and England), this article examines setbacks and recovery in desistance from crime. We show that giving up crime is a fragile project and that the implications of fragility in desistance are rarely integrated into pre- and post-release support options. To shine a light on the ‘phenomenological foreground’ of this fragility, we use Matza’s concepts of desperation and infraction and analyse how and why would-be desisters come unstuck. We find that derailment in the desistance process (frequently articulated by interviewees as ‘fuck it’ moments) rarely signifies the desire to reoffend and more often equates to the loss of the practical and emotional capacity to desist from crime.