Research Summary The relationship between reliability and validity is an important but often overlooked topic of research on risk assessment tools in the criminal justice system. By using data from the Minnesota Screening Tool Assessing Recidivism Risk (MnSTARR), a risk assessment instrument the Minnesota Department of Corrections (MnDOC) developed and began using in 2013, we evaluated the impact of inter‐rater reliability (IRR) on predictive performance (validity) among offenders released in 2014. After comparing the reliability of a manual scoring process with an automated one, we found the MnSTARR was scored with a high degree of consistency by MnDOC staff as intraclass correlation (ICC) values ranged from 0.81 to 0.94. But despite this level of IRR, we still observed a degradation in predictive validity given that automated assessments significantly outperformed those that had been scored manually. Additional analyses revealed that the more inter‐rater disagreement increased, the more predictive performance decreased. The results from our cost–benefit analyses, which examined the anticipated impact of the MnDOC's efforts to automate the MnSTARR, showed that for every dollar to be spent on automation, the estimated return will be at least $4.35 within the first year and as much as $21.74 after the fifth year. Policy Implications Although it is unclear the degree to which our findings, which are somewhat preliminary, are generalizable to other offender populations and correctional systems, we believe the results are sufficiently promising to warrant greater interest in automating the assessment of risk and need. We anticipate many, if not most, correctional systems may need to invest in upgrading their IT infrastructure to support the use of automated instruments. But we also anticipate this investment would deliver a favorable return for our results suggest that automation reduces inter‐rater disagreement, which in turn improves predictive performance. Even if automation did not improve performance, the increased efficiency it produces would create reinvestment opportunities within correctional systems.