Objective Psychosocial distress is under‐recognized in children with cancer and other serious medical illnesses because of a focus on pressing medical concerns. Aims This study assessed the validity, inter‐rater reliability, sensitivity/specificity, acceptability, and feasibility of administration of a pediatric distress thermometer (DT) designed to screen for the presence of psychosocial distress in youth with serious medical illnesses. Materials & Methods Two hundred eighty‐one patient–caregiver–provider triads were enrolled from two hospital outpatient clinics. Patients diagnosed with cancer and other life‐threatening diseases, caregivers, and providers completed the DT and a DT acceptability rating. Patients and caregivers completed standardized measures of anxiety, depression, pain, and fatigue. Providers completed a measure of disease severity. Data collectors completed a feasibility rating. Results The DT was significantly correlated with both caregiver and patient reports of depression, anxiety, pain, and fatigue, exhibiting concurrent validity. Parent, child, and caregiver report demonstrated significant, moderate inter‐rater reliability, with lower concordance between raters in the youngest age group. The DT is a sensitive instrument for screening of psychosocial distress when compared with the selected gold standard (Brief Symptom Inventory 18 depression subscale and the Children's Depression Inventory). The DT is not highly specific but quickly identifies those in need of further psychosocial assessment. Discussion Screening, using an adapted pediatric DT, is valid, feasible, and acceptable to patients, caregivers, and medical providers across chronic medical illnesses. Conclusion As patient and caregiver reports are not always concordant, both patient and caregiver report of distress are important for the provider to obtain clinically meaningful information to guide interventions. Published 2015. This article is a U.S. Government work and is in the public domain in the USA