Objective To study how differences in quality score adjustments across Medicare Advantage contracts change comparisons for individuals and contracts. Data Sources Responses to the Medicare Advantage implementation of the Consumer Assessments of Healthcare Providers and Systems (CAHPS) survey from 2010 to 2014. Study Design We fit national‐and state‐level hierarchical models to predict CAHPS scores for individuals and contracts, adjusted for self‐reported education, general health, and mental health. We allow the effects of these variables on quality measures to vary across contracts with a hierarchical model. Data Collection/Extraction Methods We perform secondary data analysis. Principal Findings For average consumers, standard adjustment is sufficient to represent variation in contract quality standardized to a common population. For people with characteristics far from average, personalized reporting using their characteristics and contract‐specific coefficients can substantially change the expected quality measures across contracts. This effect is stronger when comparing among contracts within a state than across all contracts. Conclusions Customized reporting may help consumers select the best Medicare Advantage plan, but policies should protect against unintended consequences.