Teaching practices within educational institutions have evolved through the increased adoption of technology to deliver the curriculum and the use of computers for assessment purposes. For educational technologists, there is a vast array of commercial computer applications available for the delivery of objective tests, and in some instances, organisations have opted to develop bespoke systems to meet their individual pedagogical requirements. However, there is very little research published on the usability of these systems and on the possible usability problems that could ultimately affect users' learning or users' marks in summative tests. This paper focuses on the effectiveness of the heuristic evaluation method for the evaluation of computer‐assisted assessment (CAA) systems and proposes a set of CAA heuristics for evaluating assessment tools. The results of these tests show that, with little training, novice evaluators can effectively perform an evaluation and could, using this heuristic set, identify genuine usability problems within the assessment tool. Therefore, educational technologists or software developers could use the new CAA heuristic set to aid their procurement or inform their design decisions.