Caregiver report is the most common measure of change in pediatric psychiatry. Yet, placebo response rates pose significant challenges to reliably detect a treatment response. The present study simulated an eight‐week clinical trial protocol for Autism Spectrum Disorder (ASD) for the purpose of testing the feasibility and validity of several outcome measures. Twenty caregivers answered questions about their child's behavior on their smartphone each week and completed a battery of paper questionnaires during weeks one and eight. No treatment was administered. Caregivers reported a significant decrease in problem behaviors on the Aberrant Behavior Checklist (ABC) (29% decrease) and general ASD behaviors on the Social Responsiveness Scale (SRS) (7% decrease). There was also a trend of behavior improvement from smartphone questions but no significant changes in clinical ratings of core diagnostic features of ASD. Participation in a comprehensive protocol in the absence of a particular treatment significantly influenced how caregivers perceived the severity of their children's problem behaviors. These placebo‐like effects represent substantial challenges for randomized controlled trials (RCTs) that use treatment as usual and have implications for future behavioral and pharmacological treatment trial designs. Autism Res 2017, 10: 1567–1572. © 2017 International Society for Autism Research, Wiley Periodicals, Inc.