Objectives This study identifies the most significant readability factors and examines ways of improving and evaluating Japanese health information text in terms of ease of reading and understanding. Methods Six different Japanese texts were prepared based on an original short text written by a medical doctor for a hospital web site intended for laypersons regarding chronic suppurative otitis media. Four were revised for single readability factor (syntax, vocabulary, or text structure) and two were modified in all three factors. Using a web‐based survey, 270 high school students read one of the seven texts, including the original, completed two kinds of comprehension tests, and answered questions on their impressions of the text's readability. Results Significantly higher comprehension test scores were shown in the true or false test for a mixed text that presented important information first for better text structure. They were also found in the cloze test for a text using common vocabulary and a cohesive mixed text. Conclusions Vocabulary could be a critical single readability factor when presumably combined with better text structure. Using multiple evaluation methods can help assess comprehensive readability. The findings on improvement and evaluation methods of readability can be applied to support effective health communication.