Background Evaluation of patients’ health care experiences is central to measuring patient‐centred care. However, different instruments tend to be used at the hospital or departmental level but rarely both, leading to a lack of standardization of patient experience measures. Objective To validate the Consumer Quality Index (CQI) Inpatient Hospital Care for use on both department and hospital levels. Design Using cross‐sectional observational data, we investigated the internal validity of the questionnaire using confirmatory factor analyses (CFA), and the generalizability of the questionnaire for use at the department and hospital levels using generalizability theory. Setting and participants 22924 adults hospitalized for ≥24 hours between 1 January 2013 and 31 December 2014 in 23 Dutch hospitals (515 department evaluations). Main variable CQI Inpatient Hospital Care questionnaire. Results CFA results showed a good fit on individual level (CFI=0.96, TLI=0.95, RMSEA=0.04), which was comparable between specialties. When scores were aggregated to the department level, the fit was less desirable (CFI=0.83, TLI=0.81, RMSEA=0.06), and there was a significant overlap between communication with doctors and explanation of treatment subscales. Departments and hospitals explained ≤5% of total variance in subscale scores. In total, 4‐8 departments and 50 respondents per department are needed to reliably evaluate subscales rated on a 4‐point scale, and 10 departments with 100‐150 respondents per department for binary subscales. Discussion and conclusions The CQI Inpatient Hospital Care is a valid and reliable questionnaire to evaluate inpatient experiences in Dutch hospitals provided sufficient sampling is done. Results can facilitate meaningful comparisons and guide quality improvement activities in individual departments and hospitals.