Contrasting a Mobile App With a Conversational Chatbot for Reducing Alcohol Consumption: Randomized Controlled Pilot Trial

Background Mobile apps have shown considerable promise for reducing alcohol consumption among problem drinkers, but like many mobile health apps, they frequently report low utilization, which is an important limitation, as research suggests that effectiveness is related to higher utilization. Interactive chatbots have the ability to provide a conversational interface with users and may be more engaging and result in higher utilization and effectiveness, but there is limited research into this possibility. Objective This study aimed to develop a chatbot alcohol intervention based on an empirically supported app (Step Away) for reducing drinking and to conduct a pilot trial of the 2 interventions. Included participants met the criteria for hazardous drinking and were interested in reducing alcohol consumption. The study assessed utilization patterns and alcohol outcomes across the 2 technology conditions, and a waitlist control group. Methods Participants were recruited using Facebook advertisements. Those who met the criteria for hazardous consumption and expressed an interest in changing their drinking habits were randomly assigned to three conditions: the Step Away app, Step Away chatbot, and waitlist control condition. Participants were assessed on the web using the Alcohol Use Disorders Identification Test, Adapted for Use in the United States, Readiness to Change Questionnaire, Short Inventory of Problems-Revised, and Timeline Followback at baseline and at 12 weeks follow-up. Results A total of 150 participants who completed the baseline and follow-up assessments were included in the final analysis. ANOVA results indicated that participants in the 3 conditions changed their drinking from baseline to follow-up, with large effect sizes noted (ie, η2=0.34 for change in drinks per day across conditions). However, the differences between groups were not significant across the alcohol outcome variables. The only significant difference between conditions was in the readiness to change variable, with the bot group showing the greatest improvement in readiness (F2,147=5.6; P=.004; η2=0.07). The results suggested that the app group used the app for a longer duration (mean 50.71, SD 49.02 days) than the bot group (mean 27.16, SD 30.54 days; P=.02). Use of the interventions was shown to predict reduced drinking in a multiple regression analysis (β=.25, 95% CI 0.00-0.01; P=.04). Conclusions Results indicated that all groups in this study reduced their drinking considerably from baseline to the 12-week follow-up, but no differences were found in the alcohol outcome variables between the groups, possibly because of a combination of small sample size and methodological issues. The app group reported greater use and slightly higher usability scores than the bot group, but the bot group demonstrated improved readiness to change scores over the app group. The strengths and limitations of the app and bot interventions as well as directions for future research are discussed. Trial Registration ClinicalTrials.gov NCT04447794; https://clinicaltrials.gov/ct2/show/NCT04447794

The CONSORT-EHEALTH checklist is intended for authors of randomized trials evaluating web-based and Internet-based applications/interventions, including mobile interventions, electronic games (incl multiplayer games), social media, certain telehealth applications, and other interactive and/or networked electronic applications. Some of the items (e.g. all subitems under item 5 -description of the intervention) may also be applicable for other study designs.
The goal of the CONSORT EHEALTH checklist and guideline is to be a) a guide for reporting for authors of RCTs, b) to form a basis for appraisal of an ehealth trial (in terms of validity) CONSORT-EHEALTH items/subitems are MANDATORY reporting items for studies published in the Journal of Medical Internet Research and other journals / scientific societies endorsing the checklist.
As the CONSORT-EHEALTH checklist is still considered in a formative stage, we would ask that you also RATE ON A SCALE OF 1-5 how important/useful you feel each item is FOR THE PURPOSE OF THE CHECKLIST and reporting guideline (optional).
Mandatory reporting items are marked with a red *.
In the textboxes, either copy & paste the relevant sections from your manuscript into this form -please include any quotes from your manuscript in QUOTATION MARKS, or answer directly by providing additional information not in the manuscript, or elaborating on why the item was not relevant for this study. yes: all primary outcomes were significantly better in intervention group vs control partly: SOME primary outcomes were significantly better in intervention group vs control no statistically significant difference between control and intervention potentially harmful: control was significantly better than intervention in one or more outcomes inconclusive: more research is needed Other: not submitted yet -in early draft status not submitted yet -in late draft status, just before submission submitted to a journal but not reviewed yet submitted to a journal and after receiving initial reviewer comments submitted to a journal and accepted, but not published yet Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if offline products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "This study aimed to develop a chatbot alcohol intervention based on an empirically supported app (Step Away) for reducing drinking and to conduct a pilot trial of the 2 interventions." Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Those who met the criteria for hazardous consumption and expressed an interest in changing their drinking habits were randomly assigned to three conditions", "participants were assessed on the web" subitem not at all important 1 2 3 4 5 essential subitem not at all important a  s  e  d  (  s  e  l  f  -a  s  s  e  s  s  m  e  n  t  )  v  s  .  f  a  c  e  -t  o  -f  a  c  e  a  s  s  e  s  s  m  e  n  t  s  i  n  t  h  e  M  E  T  H  O  D  S  s  e  c  t  i  o  n  o  f  t  h  e  A  B  S  T  R  A  C  T Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Participants were recruited using Facebook advertisements.", "Those who met the criteria for hazardous consumption and expressed an interest in changing their drinking habits were randomly assigned to three conditions: the Step Away app, Step Away chatbot, and waitlist control condition. Participants were assessed on the web." Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it)  I  N  T  R  O  D  U  C  T  I  O  N   2  a  )  I  n  I  N  T  R  O  D  U  C  T  I  O  N  :  S  c  i  e  n  t  i  f  i  c  b  a  c  k  g  r  o  u  n  d  a  n  d  e  x  p  l  a  n  a  t  i  o  n  o  f  r  a  t  i  o  n  a  l  e   D  o  e  s  y  o  u  r  p  a  p  e  r  a  d  d  r  e  s  s  s  u  b  i  t  e  m  1  b  -i  v  ? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "A total of 150 participants who completed the baseline and follow-up assessments were included in the final analysis.", Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Results indicated that all groups in this study reduced their drinking considerably from baseline to the 12-week follow-up, but no differences were found in the alcohol outcome variables between the groups, possibly because of a combination of small sample size and methodological issues. The app group reported greater use and slightly higher usability scores than the bot group, but the bot group demonstrated improved readiness to change scores over the app group." subitem not at all important Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Access to evidence-based treatment and support for addressing excessive alcohol use is a public health priority given that alcohol continues to be the third leading preventable cause of death in the United States, and its excessive use is responsible for >95,000 deaths each year", "Technology-based interventions, including mobile apps, have great potential to meaningfully expand access to treatment and have been shown to be acceptable among alcohol and other substance users", "In an attempt to increase intervention engagement, we developed a chatbot version of Step Away that incorporates the app modules into a chatdelivered intervention, which we postulated could result in higher use and engagement over the Step Away app.", "This paper presents results from a 3-month pilot study that compared effectiveness and participant engagement differences among individuals randomly assigned to one of three groups: (1) Step Away app, (2) Step Away bot, and (3) assessment-only delayed condition (control). Our study builds on the existing literature by indicating whether a chatbot-delivered version of a smartphone-delivered intervention has superior engagement and alcohol outcomes compared with an app with similar intervention features over a 12-week duration, and whether these interventions produce superior outcomes over a waitlist control condition." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "This paper presents results from a 3-month pilot study that compared effectiveness and participant engagement differences among individuals randomly assigned to one of three groups: (1) Step Away app, (2) Step Away bot, and (3) assessment-only delayed condition (control). Our study builds on the existing literature by indicating whether a chatbot-delivered version of a smartphone-delivered intervention has superior engagement and alcohol outcomes compared with an app with similar intervention features over a 12-week duration, and whether these interventions produce superior outcomes over a waitlist control condition." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Participants were enrolled and randomly assigned to one of three groups: Step Away app (for iPhone and Android smartphones), Step Away chatbot, and assessment-only delay (control). Participants were assessed for their alcohol consumption and related behaviors when they enrolled in the study (baseline) and again 12 weeks later (follow-up). Age and gender stratification were used to ensure a relatively even distribution across each intervention or control group.", "we analyzed data from 150 participants, 55 (36.7%) app users, 50 (33.3%) bot users, and 45 (30%) participants in the delay group." subitem not at all important Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "After examining participant responses regarding study eligibility, additional participants were removed because of having numerous prescreen submissions under the same IP address which represented phishing, or the automatic scoring through the prescreen allowing ineligible participants into the study (eg, they indicated being currently in alcohol treatment which was an exclusion criteria). Subsequently, the baseline surveys were sent to 197 participants. A few participants (n=6) were further found to be ineligible after examining their prescreening surveys, leaving 191 eligible baseline surveys. At follow-up, 163 participants completed the survey. After removing 13 participants owing to failing our validation checks" Bug fixes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug fixes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have influenced study design such as staff changes, system failures/downtimes, etc.
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Advertisements were targeted to all Facebook users who may meet the following criteria: age ≥18 years; have either an iPhone or an Android phone; reside in the United States; not in another form of alcohol treatment or using another mobile health alcohol intervention, be an active drinker, and have proficiency in English language.
The prescreening survey asked potential study participants about these criteria as well as all 10 questions from the Alcohol Use Disorders Identification Test, Adapted for Use in the United States (USAUDIT). Those who met these criteria, and who had a USAUDIT score between 8 and 24 inclusive for males aged ≤65 years and a score between 7 and 24 inclusive for females as well as males aged ≥65 years, were invited to complete the consent form and the baseline survey" Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clarified. Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic, and clarify if this was a purely webbased trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these.
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Participant recruitment was conducted through Facebook advertisements, which provided a link to the study website and the web-based prescreening survey," "Baseline survey links were emailed to participants once they were manually reviewed by the study team to confirm their eligibility.", "Follow-up surveys were matched to the baseline surveys for each participant. An email with a link to the follow-up survey was manually sent to each participant as they became eligible (ie, after they had used the app or chatbot for 12 weeks, or 12 weeks after their study enrollment for the control group)." Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Screening and administering consent forms and baseline and follow-up surveys were all done using the web-based survey platform Qualtrics" subitem not at all important Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial.
Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of specific group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Participants were enrolled and randomly assigned to one of three groups: Step Away app (for iPhone and Android smartphones), Step Away chatbot, and assessment-only delay (control)." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study " Step Away is a smartphone app designed to deliver empirically based alcohol assessment and intervention for individuals who drink at hazardous levels that may present health risks.
Step Away is the next generation of an earlier app that we tested (the Location-Based Monitoring and Intervention System-Alcohol) with individuals with an alcohol use disorder, which demonstrated significant 6-week reductions in alcohol consumption, along with ratings by participants as being very helpful in changing their drinking habits [39]. This study also indicated that the amount of use of Location-Based Monitoring and Intervention System-Alcohol features was related to changes in alcohol consumption. The app offers eight modules in addition to daily alcohol consumption and craving tracking: (1) assessment and feedback on alcohol consumption relative to age-specific norms, drinking-related problems, and monetary costs of drinking, including daily prompting to complete a brief questionnaire on drinking behavior and cravings during the prior 24 hours, and weekly feedback highlighting progress toward goals; (2) goal setting, which asks participants to select abstinence or moderation as a goal; (3) rewards, which prompts them to set up a reward for meeting their goal and reminds them to reward themselves when their goal is met (eg, 30 days of no drinking); (4) cravings, which offers 6 in-the-moment interventions for coping with cravings; (5) moderation or abstinence strategies, which consists of simple behavioral strategies tailored to the participant's goal; (6) supportive persons, which provides tools for connecting with participant-identified friends or family when additional support is needed; (7) reminders, which encourages the creation of visual reminders of their reasons for changing their drinking habits, including the ability to upload inspirational photos to make a change; and (8) new activities, which recommends healthy behaviors and the ability to schedule selected activities within the smartphone calendar.
Step Away also provides real-time intervention options; that is, when a participant clicks on the "Get Help" icon, they are provided with strategies for managing cravings or negative emotions and contacting a national treatment finder service to receive help finding in-person treatment in their area." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21 -generalizability). Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator". Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Your answer  T  E  :  P  r  e  f  e  r  a  b  l  y  ,  t  h  i  s  i  s  s  h  o  w  n  i  n  a  C  O  N  S  O  R  T  f  l  o  w  d  i  a  g  r  a  m  ) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study " Figure 1 shows the flow of this study." Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other figures or tables demonstrating usage/dose/engagement. Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Recruitment began in early June 2020 and was completed in early September 2020 when the target sample size was reached." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study Report multiple "denominators" and provide definitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at specific pre-defined time points of interest (in absolute and relative numbers per group). Always clearly define "use" of the intervention. In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational definitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is defined (e.g., timeout after idle time) [1] (report under item 6a). Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "This study sought to develop a chatbot version of the empirically supported app, Step Away, and conduct a pilot trial to determine if a chatbot version could provide enhanced use and outcome effectiveness over the app version. We also sought to pilot trial a methodology that included randomly assigning participants who were hazardously consuming alcohol and interested in making a change to their drinking to one of three conditions: the Step Away app, the Step Away bot, and a waitlist control condition.
Results from this pilot study indicated that self-reported alcohol consumption from baseline to the 12-week follow-up decreased substantially in all groups. Effect sizes suggested that changes in alcohol consumption and drinking-related problems were within the large effect range. However, the results also suggested that there were no statistically significant differences in alcohol consumption variables between the 3 groups, suggesting that the waitlist control condition changed their drinking at a similar level to the intervention groups.", "Another focus of this study, which is perhaps equally important for alcohol outcome analysis, is the utilization assessment of the 2 interventions. In contrast to our expectations, our results suggested that the app was used more frequently and for a longer time than the bot." Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "This study had numerous limitations, many because of its pilot nature, which are related to recommendations for future research. First, the sample was insufficient for detecting small to medium effect sizes, and our findings reflected this limitation.
We estimated that a sample size of 195 would provide enough power to detect between-group effect sizes, as shown in this study. Second, the methodology for assessing the waitlist control condition likely resulted in reactivity to assessment phenomena, which made detecting differences between the interventions and the control condition challenging. Future studies in this area would be wise, if using a waitlist control condition, to delay assessment, a method that has been used successfully in other studies [14]. We also had limited time to undertake this study and our 12-week follow-up period may have been insufficient for differences between interventions to emerge. Previous research with Step Away showed that participants continued to reduce their alcohol intake at 6 months [43] and that 45% of participants were still actively engaged at the 6-month follow-up. A 12-month follow-up would provide a more detailed picture of how users remain engaged with the interventions over time and how this engagement is related to improvement. Finally, although this study set out to contrast the 2 interventions and determine which had higher use and effectiveness, this contrast is perhaps not ideal. A more beneficial strategy may involve combining these technologies Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or briefly explain why the item is not applicable/relevant for your study "Clinical Trials NCT04447794"