Despite an extensive history of use in teaching Political Science subjects, long‐term scholarly studies of online role plays are uncommon. This paper redresses that balance by presenting five years of data on the Middle East Politics Simulation. This online role play has been run since the 1990s and underwent significant technical upgrade in 2013–14. The data presented here covers student feedback to this upgrade process and the factors that can influence their response. Key indications are that students tend to recognise when something is fit (or not) for its purpose and will forgo attractive and well‐appointed online environments if the underlying learning exercise is valued. However, there are limits to this minimalism and whilst designers do not need to replicate every Internet trend, attention needs to be paid to broader changes in technology, such as access platform and changing avenues of political communication. The study demonstrates that long‐term monitoring of online role play exercises is important to allow informed changes to be implemented and their impacts properly assessed.