Comparative Effectiveness of Three Digital Interventions for Adults Seeking Psychiatric Services

This randomized clinical trial assesses the comparative effectiveness of digital mindfulness-based, cognitive behavioral therapy–based, and personalized feedback tools for patients awaiting formal mental health treatment.


Introduction
2][3] While there are several moderately effective treatment options for common mental health conditions, including medications and psychotherapy, there is a wide gap between the demand and capacity for mental health services, prolonging wait times and increasing the prevalence of symptomatic adults with unmet mental health needs. 4,5Digital mental health interventions (DMHIs) are promising tools for bridging this gap between mental health care need and capacity. 6,7 estimated 85% of US adults own a smartphone, making DMHIs accessible to most of the population, including socioeconomically disadvantaged groups. 80][11] Meta-analyses across various DMHIs, particularly those incorporating skills from evidence-based treatments, such as cognitive behavioral therapy (CBT) and mindfulness, have reported improvement in depressive and anxiety symptoms with small to moderate effect sizes compared with control conditions. 12,13rthermore, studies have demonstrated the benefits of personalized feedback-notification interventions that generate tips and strategies meant to support better health behaviors. 14,15Thus, although robust evidence has shown DMHIs to be effective compared with usual care or waiting list conditions, relatively little is known about the comparative effectiveness of different DMHIs.
To understand the potential role of DMHIs in bridging gaps in mental health care, in the present pragmatic randomized clinical trial (RCT), we evaluated the comparative effectiveness of different DMHIs for patients scheduled for in-clinic mental health visits.Given the interest in transdiagnostic approaches and the practicality of a universal approach to providing DMHIs to patients scheduled for outpatient services, intervention groups were evaluated for changes in depression, anxiety, substance use, and suicidality after 6 weeks of intervention.We hypothesized that (1)

Methods
This RCT was conducted from May 13, 2020, to December 12, 2022.Written informed consent was obtained from all participants.The University of Michigan Institutional Review Board approved all procedures in the study.The trial protocol is provided in Supplement 1.We followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. 16

Setting and Participants
Participants were adults seeking outpatient psychiatric services from several mental and behavioral health clinics within the University of Michigan Health System.Seventy-seven percent of patients

Study Design and Procedures
Patients meeting initial inclusion criteria were contacted via telephone call, text, or email.
Participants completed the baseline survey of demographics and clinical assessments using a health Recommendations for daily use and specific modules or tracks (eg, depression or anxiety) were automatically provided for the Silvercloud and Headspace arms based on baseline symptoms.
Silvercloud has a global user base of over 500 000 people and includes psychoeducation and self-guided modules introducing CBT skills (eg, mood monitoring, activity scheduling, and cognitive restructuring) via text, videos, and journaling exercises.This mobile application has demonstrated considerable benefits of reducing depression and anxiety symptoms (effect size range, 0.50-0.63)8][19] Headspace includes a library of guided meditations and strategies for improving sleep, decreasing anxiety, and coping with stress.It has a user base of over 2 000 000 people and has also demonstrated efficacy in reducing depressive and anxiety symptoms (effect size range, 0.24-0.26)[22] In the EPF arms, feedback was displayed on the MyDataHelps dashboard and delivered through pop-up notifications.A subset of notifications included personalized content based on their specific data, with content categories of physical activity, sleep, and mood.Example messages were "Giving to others can have the added bonus of making us feel good as well.Who can you do a small kindness for today?Notice the impact on your mood," and "Your daily step average since the start of the study is [average daily step count].Sometimes it's tough to get moving, and yet you've had some really great active days.Keep it up!" 14 Notification messages were delivered twice a day at 10 AM and 2 PM to all participants in the EPF arm.Some of the messages for participants randomized to Silvercloud plus EPF or Headspace plus EPF were tailored to provide feedback and encouragement on using the application, with message categories such as therapy minutes for the Silvercloud plus EPF arm and mindful minutes for the Headspace plus EPF arm.The DMHIs were completely independent of the clinicians or health care system from which participants were awaiting services.

Outcomes
The primary outcome of the study was change in depression score as measured by the 9-item Patient Health Questionnaire-9 (PHQ-9; score range: 0-27, with higher scores indicating greater depression symptoms). 23Secondary outcomes were changes in anxiety, substance use, and suicidality from baseline to 6-week follow-up.Anxiety was measured by the 7-item Generalized Anxiety Disorder-7 (GAD-7; score range: 0-21, with higher scores indicating greater anxiety symptoms). 24Substance use was measured using an adapted version of the Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST) 25 that rates frequency of use in the past 3 months for 9 different classes of substances on a Likert scale ranging from 0 (indicating never) to 4 (indicating daily or almost daily).
Presence of any substance use was detected by the ASSIST outcome.Suicidality was assessed using the Positive and Negative Suicidal Ideation (PANSI), a 14-item measure with a negative subscale (PANSI-; score range: 8-40, with higher scores indicating greater risk for suicidal behavior) and a positive subscale (PANSI+; score range: 6-30, with lower scores indicating greater risk for suicidal behavior). 26

Randomization and Blinding
Participants were randomized using a 2:1:1:1:1 allocation, with the EPF-only arm being twice the size of other groups.Computer randomization was done in blocks of 12 and 18 with no stratification within MyDataHelps once the informed consent record was complete and patients were identified as eligible.The order was concealed from study staff who were enrolling patients; the study dashboard revealed a patient's assignment when the patient completed all enrollment procedures and the baseline survey.Follow-up surveys were delivered via MyDataHelps, and outcomes were self-reported; therefore, no blinding procedures regarding outcomes assessment were necessary for study staff.

Statistical Analysis
With primary analyses using the analysis of covariance (ANCOVA) test for intervention effects between the 5 groups, 95% power was achieved with a small effect size of 0.25 for a minimum sample size per group of 65, assuming 2 variables, a numerator df of 5 and α = .05,according to G*Power. 27Patient-specific change from baseline scores for the PHQ-9, GAD-7, and PANSI were calculated for all participants who completed the 6-week follow-up survey.All analyses followed the intent-to-treat principle.The effect of intervention arms on change from baseline for each of these outcomes was assessed through ANCOVA, adjusting for baseline value.The effect of study group on ASSIST at 6 weeks was assessed using logistic regression for the outcome of any substance use. ANCOVA

Study Participants
A total of 2079 participants completed the baseline survey (Figure

Primary Outcome
The mean (SD) baseline PHQ-9 depression score was 12.7 (6.4).Overall, depression scores decreased by 2.5 points from baseline to 6-week follow-up (n = 1885) for all 5 arms (marginal mean differences

Secondary Outcomes
Similar to PHQ-9 depression scores, GAD-7 anxiety scores significantly improved from baseline to the 6-week follow-up in all groups, and the magnitude of improvement was not significantly different across the 5 intervention arms in the 3 pooled comparisons.With respect to suicidality, on the PANSI+ subscale, the Headspace-only arm had significantly greater improvement than the Silvercloud plus EPF arm (MD in mean change = 0.94; 95% CI, 0.09-1.78;P = .02).There were no significant differences when arms were pooled based on EPF only vs the other 4 arms.However, pooled Headspace only and Headspace plus EPF arms had significantly greater improvement than pooled Silvercloud only and Silvercloud plus EPF arms (MD in mean change = 0.63; 95% CI, 0.20-1.06;P = .004).Scores on the PANSI-subscale decreased for all 5 intervention arms from baseline to follow-up, and the magnitude of improvement was not significantly different when examined individually or when pooled (Table 3).In sensitivity analyses, we examined outcomes after applying attrition weighting to address missing data at follow-up.
Results did not reveal any significant differences in patterns of change for primary and secondary outcomes.
In logistic regression analyses, abstinence from substances did not significantly change during the study period overall and was not significantly different among any of the 5 intervention arms (abstinence ranged from 14.0% to 19.7% at baseline to 16.2% to 20.6% at follow-up; no individual arm had a more than 4.2% improvement in abstinence rates).Findings for change in substance use Abbreviations: EPF, enhanced personalized feedback; PHQ-9, Patient Health Questionnaire-9 (score range: 0-27, with higher scores indicating greater depression symptoms).e Symptom-level change estimates were adjusted for baseline PHQ-9 score via analysis of covariance.Mean changes in PHQ-9 scores were significantly different from 0 at P < .001for all intervention arms.There were no significant differences in mean changes in PHQ-9 scores among the 5 intervention arms when adjusted for multiple comparison using a Tukey-Kramer adjustment (F 4,1879 = 1.19;P = .31).  e Mean change in value was adjusted for baseline scores via analysis of covariance (ANCOVA).Mean change in GAD-7 and PANSI-scores were significantly different from 0 at P < .001for all intervention arms.For GAD-7 and PANSI-scores, there were no significant differences in mean change in scores among the 5 intervention arms.Mean change in PANSI+ scores were significantly different from 0 at P < .01 for all interventions except Silvercloud + EPF.Among the pairwise comparisons for PANSI+ scores, the only significant difference was Silvercloud + EPF vs Headspace only at 0.9 (95% CI, 0.1-1.8;P = .02).All pairwise comparisons were adjusted for multiple comparisons using the Tukey-Kramer method.
across groups were also not significant when alcohol and marijuana use were examined as individual, continuous dependent variables.

Discussion
Contrary to our hypotheses, randomization to various DMHIs did not result in differing outcomes with respect to depression, anxiety, suicidality, and substance use among adults scheduled for outpatient psychiatry services.Specifically, there were no differences between participants randomized to receive passive EPF only compared with those randomized to an active commercial mobile application (Headspace or Silvercloud) with or without EPF.There were also no significant differences between patients who were randomized to Headspace only or Silvercloud only and those randomized to Headspace plus EPF or Silvercloud plus EPF.However, patients randomized to Headspace had significantly greater improvements on the PANSI+ subscale compared with those randomized to Silvercloud, although these improvements were negligible.
Participants across the DMHI groups had modest improvements in depression, anxiety, and suicide risk.The magnitude of change in depressive symptoms was similar to that in prior studies of mindfulness applications (SD change, 0.4), 21 although it was slightly lower than prior CBT-based Silvercloud studies (PHQ-9 change range, 3.6-5.2). 17,18 However, prior Silvercloud studies included clinician feedback and 15-minute reviews of application module use on a weekly basis, which was not included in the present study, and had more than 3 hours of median use time.
Improvements in the current study may have been achieved with a human component and/or greater engagement.Most patients were already on a waiting list for more than 3 months before being contacted to schedule an initial appointment (and becoming eligible for the study), and symptom reduction was unrelated to the number of formal mental health visits, suggesting that the symptom reductions were not fully explained by alternatives, such as regression to the mean or parallel interventions.While this RCT was not designed to evaluate causality or efficacy, the improvements across intervention groups were consistent with prior literature demonstrating the effectiveness of these DMHIs 18,22 and suggest that DMHIs were generally beneficial, without any intervention or pairing of interventions being notably superior to the others evaluated in this study.

Strengths and Limitations
This study has several strengths, including a large transdiagnostic clinical population of treatmentseeking adults and a pragmatic design embedded in existing service delivery health care systems.
However, the findings must be considered within the context of the existing limitations of the study.
The benefits of the pragmatic design come at the expense of greater control over facets of enrollment, including wait time for services and other potential confounders.Most participants attended formal psychiatric care visits within the 6-week study period.Although a sensitivity analysis did not find significant differences when adjusting for number of clinical visits, there may have been aspects of formal care that contributed to the improvement in symptoms across groups and diluted the ability to identify differences between applications in effects on symptom reduction.8][19][20][21][22] The study sample was diverse with respect to including various presenting clinical concerns at various clinics but had similar demographic characteristics to those of the treatment-seeking population (ie, mostly White and female patients), and it is unclear whether findings may be generalizable to more diverse samples.While differences were not observed in intervention effects on depression, anxiety, suicidality, or substance use outcomes at the 6-week follow-up, there may have been other unmeasured, clinically relevant factors or mechanisms that responded differently as a function of randomization to the DMHIs.
patients randomized to the passive enhanced personalized feedback (EPF)-only intervention would report less improvement in mental health symptoms compared with patients randomized to the more active CBT-based or mindfulness-based DMHIs and that (2) patients receiving both CBT-based or mindfulness-based DMHIs and EPF would show greater improvements than patients who received a CBT-based or mindfulness-based intervention alone without EPF.A secondary analysis compared the performance of the CBT-based and mindfulness-based interventions (regardless of EPF) across these symptom domains, but there were no a priori hypotheses regarding comparative effectiveness.

Figure . Flow
Figure.Flow Diagram of Study Participants

a N = 640 for 6 -
week and change in values.b N = 311 for 6-week and change in values.c N = 313 for 6-week and change in values.d N = 310 for 6-week and change in values.
Three Digital Interventions for Adults Seeking Psychiatric Services Included participants were English speakers aged 18 years or older with either a scheduled future mental health appointment or an initial appointment completed within the past 60 days and daily access to a smartphone for use in the study.Participants were excluded if they had cognitive or guardianship restrictions that prevented them from providing informed consent or if they had an active eating disorder that, based on patient or staff judgment, made the tracking device and activity monitoring contraindicated.Race and ethnicity data were collected merely to characterize the sample.

Table 3
provides within-group estimates and 95% CIs, and eFigure 2 in Supplement 2 provides boxplots.

Table 2 .
Symptom-Level Change for Depression Among the 5 Intervention Arms