Background Formulation is a core competency of mental health professionals, drawing on a variety of sources of information. In England and Wales, the current strategy for offenders with personality disorder places formulation‐led management, generally by probation staff, at its core, but reliability and validity of the process remain unclear. Aims The first aim was to evaluate a checklist previously designed to establish quality of formulation, and the second to measure the impact of training and consultation on the ability of probation officers to formulate cases. Methods The inter‐rater reliability, test–re‐test reliability and internal consistency of the McMurran formulation checklist were calculated from the scores derived from randomised formulations completed by probation officers from fictitious case vignettes. The impact of training was measured by comparing pre‐ and post‐training formulations of these vignettes. Practice cases formulated by probation officers at psychologist‐facilitated consultation meetings over a 6‐month period were used to measure the impact of consultation. All formulations were scored blind by independent experts. Results Inter‐rater reliability, test–re‐test reliability and internal consistency of the scale were all acceptable. Training and practice did not significantly improve the probation officers' formulations. Conclusions The purpose and utility of formulation may vary according to the context in which it is applied. Progress in developing formulation skills may depend on the nature and length of the previous experience of this skill. Future research should take account of such variance, with this scale as a potentially useful aid in monitoring progress. Implications The capacity for teaching formulation to probation officers could be investigated further by comparing the process with formulation development by mental health experts with previously extensive formulation experience. Formulation will probably need to be adapted to meet the needs of the context in which it is developed. The relationship between formulation and management outcome was not investigated here and would be a further important step. Copyright © 2013 John Wiley & Sons, Ltd.