This paper presents the results of a systematic review of the literature, including a meta‐analysis, about the effectiveness of the use of Alice software in programming learning when compared to the use of a conventional programming language. Our research included studies published between the years 2000 and 2014 in the main databases. We gathered 232 papers. Taking into account the selection criteria to make the meta‐analysis, we retained six papers with a quasi‐experimental design, with 464 participants in total. To combine the results we used the random effect model. It resulted in an effect size of 0.54 (Cohen's d) with a confidence interval between 0.34 and 0.74. We concluded that until now there have been few experimental results on the effectiveness of Alice programming language to introduce students in learning how to program. The results we found were the expression of different experimental treatments and distinguished teaching methods which made the comparison of the results obtained more subtle. However, the existing experimental results that were submitted to the meta‐analysis allowed us to assume with a certain margin of safety that a teaching strategy that uses Alice should obtain more effective results than the use of a conventional programming language.