Objective Although competency‐based training of supervisors is now compulsory in many countries, there has been limited evidenced‐based literature to guide the profession as to the most effective ways of training supervisors. The aim of the current pilot study was to examine the effects of a supervisor training program on supervisor, supervisee, and evaluator perceptions of supervisory competence. Method Ten female and two male supervisors employed by a state‐wide counselling service participated in the study. Data, including taped supervision sessions relating to supervisor, supervisee, and evaluator perceptions of supervisory competence, skill, and process were collected at three time points; two prior to a supervisor training program and one following the program. Results The evaluator observed significant improvements in supervisory competence from pre‐ to post‐training. Although supervisors reported no significant change in competence as a result of training, they did report areas of positive changes in supervisory practice. Across time supervisors also reported a significant decrease in their perception of the supervisor–supervisee alliance on one measure, but this was not noted by the evaluator or supervisee, or on other measures of supervisory alliance. Conclusions Evaluator ratings of supervisor competency and qualitative supervisor feedback provide preliminary support for the effectiveness of supervisor training. However, the pilot study was underpowered and some of the measures require further psychometric testing, which will need to be addressed in future research.