Forensic researchers often assume that the widely used area under the curve (AUC) predictive validity statistic can be readily compared across risk assessment instruments and between studies. From risk distributions for 224,771 convicted English and Welsh offenders, I quantify the extent to which the AUCs of two continuously scored actuarial instruments are dependent upon sample heterogeneity and risk categorization. Sample composition can cause predictive validity to vary by 10% to 20% over chance between subpopulations, with higher AUCs for female subgroups whose scores have high variability, and lower AUCs when sampling is restricted by index offense and/or sentence type. Risk categorization has a potentially large effect on AUCs, which fall when few categories are used, especially when those categories contain unequal numbers of offenders. An improved understanding of how these factors can affect AUCs will inform professionals using risk assessment instruments and researchers comparing the validity of multiple instruments.