4 - (SYM 3) Development of an Automated, Scalable Method for Assessing Fidelity to Cognitive Processing Therapy

Therapist fidelity to evidence-based treatments (the extent to which the therapist delivers the treatment as designed) predicts treatment outcomes. Yet measuring therapist fidelity is costly and burdensome. Scalable, quantitative methods for assessing therapist fidelity to evidence-based treatments (EBTs) are needed. This presentation will demonstrate the process of development of a large language model (LLM) based method for scalable, automated detection of therapist fidelity to cognitive processing therapy (CPT), an EBT for PTSD. This was developed using therapy transcripts from a trial of text-message CPT (“CPT Text”) versus treatment as usual, both delivered via the Talkspace platform (N = 378). We adapted a validated CPT fidelity measure, the Therapist Adherence and Competence Protocol (Wiltsey Stirman et al., 2021) for the novel format (CPT Text), which occurs asynchronously over multiple brief interactions rather than via traditional therapy sessions. The adapted measure assesses fidelity at the message level; each individual therapist message receives a label indicating which, if any, CPT skill was performed in the message (e.g., identifying a stuck point, Socratic questioning, assigning a challenging beliefs worksheet). After establishing good interrater reliability with a set of labeled messages (from n ~ 30 transcripts) containing 10-20 positive examples of each skill (preliminary kappa = .79), models will be trained and tested against a larger set of labeled messages (from n ~ 60 transcripts). Model accuracy will be established by identifying F1 scores (harmonic means between precision and recall) of .40 and above on the held-out, “test” set of labeled messages. This method represents a novel, scalable, LLM-based method for detecting fidelity in CPT Text, and extensions of this work could be applied to traditional delivery of CPT. Automated, scalable, and fine-grained labeling of therapy skills across the course of therapy may someday facilitate real-time feedback to therapists on their delivery of EBTs and time- and order-based analyses of the active ingredients of EBTs, lending insight into which treatments work best for whom.