In order to understand potential applications of collaborative problem‐solving (CPS) assessment tasks, it is necessary to examine empirically the multifaceted student performance that may be distributed across collaboration methods and purposes of the assessment. Ideally, each student should be matched with various types of group members and must apply the skills in varied contexts and tasks. One solution to these assessment demands is to use computer‐based (virtual) agents to serve as the collaborators in the interactions with students. This article proposes a human‐to‐agent (H‐A) approach for formative CPS assessment and describes an international pilot study aimed to provide preliminary empirical findings on the use of H‐A CPS assessment to inform collaborative learning. Overall, the findings showed promise in terms of using a H‐A CPS assessment task as a formative tool for structuring effective groups in the context of CPS online learning.