Background We investigated student responses to multiple‐choice concept questions during active learning activities where students write justifications for their answer choices. Purpose We selected two questions that asked students to apply the same concept in the same way but that have different surface features. We characterized students' responses to reveal how question attributes influence student thinking. Design/Method In Study 1, we compared responses from students who wrote justifications with those who did not. In Study 2, we modified one of the questions so the cues of the two questions better align. Data included the students' answers and their written justifications. We applied open coding to the written justifications to develop hierarchical categories common to both questions. Results For the Expanding Piston question, students who wrote justifications chose the correct answer more often; in the Balloon Rising question, less often. Analyses of justifications indicate that in the latter question, students activated more sophisticated reasoning processes, but did not choose the correct answer. We attribute half the difference between percentage correct in Study 1 to question cues and the remaining difference to an unfamiliar context. Conclusions We encourage instructors to solicit written justifications to multiple‐choice concept questions and to look for cases where students choose the correct answer using incorrect reasoning, ask students to construct graphics as they respond to text‐only concept questions, and seek ways to help students recognize how core concepts apply to a broad range of contexts.