An accumulation of research evidence indicates that hate crimes are more serious than similar but otherwise motivated crimes in respect of the greater post-victimization distress for victims. Such evidence has been used by advocates of hate crime laws to justify greater penalties for hate crime offenders. However, in focusing on the commonalities of the post-victimization impacts inflicted by hate crimes, the research evidence to date has obscured the diversity of reactions between victims. Consequently, this article expands the evidence by illuminating the variation in reported victim impacts. The analysis presented uses data from the Crime Survey for England and Wales on racially motivated crime and reveals that not all victims report being affected by hate crime, not all victims are affected the same way, and some victims of racially motivated crime report less of an emotional impact than some victims of equivalent but otherwise motivated crimes. It is reasoned that in any individual case of hate crime the motivating sentiments of the offender provide an unreliable indicator of the harms inflicted on the victim. Therefore, a blanket uplift in penalty in every case which rests on the offender’s motivations cannot be justified if the justification for sentence uplift is to give offenders their just deserts for the harms they inflict. Instead, the justification must rest on the culpability of the offender for the harms they may or may not actually inflict. Just as there is variation in victim impacts, there will be variation in offender culpability: Discretion and flexibility in sentencing is therefore necessary to ensure justice for offenders.