1 - (3 MIN 4) Convergent Validity of Clinician-observer Reported Treatment Fidelity in Community Youth Mental Health

Clinician fidelity to active components of evidence-based interventions (EBIs) in real-world contexts is hypothesized as a critical reason for the “implementation cliff”—a reduction in benefits of treatment from efficacy trials to community settings (Weisz et al., 2014; Garland et al., 2013). Therefore, improving clinician fidelity to EBIs for youth mental health is critical for the successful implementation of treatment in real-world community settings (McLeod et al., 2013; Weisz et al., 2014). Developing effective, efficient, and pragmatic fidelity measures is an essential precursor for enhancing clinicians’ adherence to EBIs (Becker-Haimes et al., 2021; Brookman-Frazee et al., 2021; Glasgow & Riley, 2013; Hogue et al., 2022; Wang et al., 2023). An often-used approach to measure fidelity is observational coding, considered the “gold standard” in research (McLeod et al., 2009). Although widely used in research, the observational coding approach is costly, time-intensive, and not feasible in contexts with limited resources (Perepletchikova, 2011; Schoenwald et al., 2011). An alternative to observational coding is self-report measures that are quick, non-intrusive, and can be implemented daily or more frequently throughout the intervention (Brookman-Frazee et al., 2021; Hogue et al., 2022). Across the few studies in the literature that examined the concordance between clinician self-reports and observationally coded adherence to EBIs (Hogue et al., 2015), mixed results were yielded (weak concordance: Becker-Haimes et al., 2022; Herschell et al., 2020; moderate concordance: Hogue et al., 2015, 2022; strong concordance: Cho et al., 2025; Wang et al., 2023).
Our study aimed to evaluate the convergent validity of a clinician self-report measure—Clinician Self-Report of Session Fidelity (CSSF)—and an observer-rated measure of fidelity—Therapist Integrity in Evidence-Based Interventions (TIEBI). Using secondary data from the Youth FIRST trial, we analyzed 499 sessions delivered by 22 clinicians to 55 youth and their caregivers. The FIRST transdiagnostic intervention has EBI strategies encompassed by five broad principles of change, namely, Feeling Calm (physiological relaxation), Increasing Motivation (use of environmental contingencies), Repairing Thoughts (cognitive restructuring), Solving Problems (sequential problem-solving), and Trying the Opposite (exposure, behavioral activation, and prosocial behavioral rehearsal) (Weisz & Bearman, 2020). We found statistically significant, moderate to strong correlations (r = .50 to .74, p < .001) between clinician self-reports and observational coding at the level of broad principles. Clinicians reported broad principles more reliably than the specific strategies, even though they were noted to generally underreport most principles. Ultimately, this high convergence of clinician- and observer-reported fidelity provided evidence in support of clinician self-report measures as a feasible, scalable, and cost-effective approach, especially in resource-limited settings. Consequently, ensuring that more youth and families receive high-quality treatment in typical service settings.