In large class settings, individualized student–teacher interaction is difficult. However, teaching interactions (e.g., formative feedback) are central to encouraging deep approaches to learning. While there has been progress in automatic short‐answer grading, analysing student responses to support formative feedback at scale is arguably some way from being widely applied in practice. However, analysing student written responses can provide insights into student conceptions, thus directly informing teacher actions. Indeed, we argue that analysing student responses to provide feedback directly to teachers is as worthy a goal as providing individualized feedback to students and is achievable given the current state‐of‐the‐art in natural language processing. In this paper, we analyse student written responses to short‐answer questions posed in the context of a large first year health sciences course. Each question was designed to elicit deep responses. Our qualitative analysis illustrates the variability in student responses and reveals multiple relationships between these responses, course materials and the questions posed. Such information can be invaluable for teacher praxis. We conclude with a conceptual ‘dashboard’ that categorizes student responses and reveals relationships between responses, course resources and the questions. Such a dashboard could provide timely, actionable insights for teachers and help foster deep learning approaches for students.