User Experience of CBT Apps for Depression : An Analysis of App Functionality and User Reviews

Background: Hundreds of mental health apps are available to the general public. With increasing pressures on healthcare systems. they offer a potential way for people to support their mental health and wellbeing. However, while many are highly rated by users, few are evidence-based. Equally, our understanding of what makes apps engaging and valuable to users is limited. Objective: The aim of this paper was to analyse functionality and user opinions of mobile applications (apps) purporting to support Cognitive Behavioural Therapy (CBT) for depression and to explore key factors that have impact on user experience and support engagement. Methods: We systematically identified apps described as being based on CBT for depression. We then conducted two studies. In the first, we analysed the therapeutic functionality of apps. This corroborated existing work on apps’ fidelity to CBT theory, but we also extended prior work by examining features designed to support user engagement. Engagement features found in CBT apps for depression were compared with those found in a larger group of apps that support mental wellbeing in a more general sense. Our second study involved a more detailed examination of user experience, through a thematic analysis of publicly available user reviews of CBT apps for depression. Results: We identified 31 apps that purport to be based on CBT for depression. Functionality analysis (Study 1) showed that they offered an eclectic mix of features, including many not based on CBT practice. CBT apps used less varied engagement features compared to 253 other mental wellbeing apps. The analysis of 1,287 user reviews of CBT apps for depression (Study 2) showed that apps are used in a wide range of contexts, both replacing and augmenting therapy, and allowing users to take active role in supporting their mental health and wellbeing. Users, including health professionals, valued and used apps that incorporated both core CBT and non-CBT elements, but concerns were also expressed regarding the unsupervised use of apps. Positivity was seen as important to engagement, e.g. in the context of automatic thoughts, users express a preference to capture not just negative, but also positive ones. Privacy, security and trust were crucial to the user experience. Conclusions: CBT apps for depression need to do a better job of incorporating evidencebased CBT elements. Equally, a positive user experience is dependent on other design factors, including consideration of varying contexts of use. App designers should be able to clearly identify the therapeutic basis of their apps, but they should also draw on evidencebased strategies to support a positive and engaging user experience. The most effective apps are likely to strike a balance between evidence-based CBT strategies and evidencebased design strategies, including the possibility of eclectic therapeutic techniques.


Introduction
Mental health difficulties are a leading cause of disability worldwide [1,2], with depression alone affecting 98.7 million people [2]. Responding to the urgent need to provide more people with access to effective treatments, substantial research has been undertaken on the use of technology to increase access to mental health treatment [3][4][5][6][7][8]. Much of this work has focused on the development and evaluation of computerised Cognitive Behavioural Therapy (CBT) [5,[8][9][10][11][12][13]. CBT incorporates both behavioural and cognitive aspects and provides a structured approach for recognising and addressing negative thinking patterns and underlying beliefs [14]. Due to this structured approach, it lends itself well to being adapted to computerised platforms, both as self-directed [9] and therapistguided [10] online treatment.
More recently mobile applications (apps) have provided an alternative to computerised CBT interventions. With 76% of UK and 81% of US adults owning a smartphone [15,16], there is a strong argument for the potential of apps to help in providing flexible access to mental health support [17,18]. Studies have been conducted focusing on the development of mobile apps to support mental health [19][20][21][22][23][24][25][26][27][28] and more detailed reviews and analyses of existing research are also available [4,[29][30][31]. This work indicates the potential of appropriately designed apps and could drive future innovation in mental health apps to ultimately deliver large-scale impact on public health. However, given the openness of app stores to developers [32,33], challenges with regulating health apps [34], and the time it typically takes for evidence-based research to make its way into healthcare practice [35], it is unsurprising that current research is not always reflected in the apps available in app stores.
Recently, several papers have reviewed apps with the aim of assessing the extent to which they are grounded in theory, especially CBT [36,37], or to evaluate the extent of expert involvement [38,39]. They suggest that current apps tend to lack an evidence-base [31,38] and often combine evidence-based features with other approaches not supported by research [37]. Furthermore, there does not seem to be any correlation between apps' ratings and popularity and the presence of evidence-based features [37].
We agree that the lack of an evidence base in publicly available apps is a significant cause for concern. However, high ratings of apps defined as "inconsistent with evidence" [37] suggest that they might be important to users. The existence of these apps provides an important opportunity to investigate and understand factors that facilitate user engagement with mental health apps. Through app reviews submitted to app stores, people using these apps have provided a large body of data regarding their user experiences, context of use and features they value. Previous work within the Human-Computer Interaction (HCI) community has demonstrated the benefits of using public reviews to investigate user attitudes towards and experiences of existing apps [40][41][42][43][44]. Researchers have analysed user reviews of mood tracking apps [44], looked at general use of health apps, e.g. [45] or the types of health apps people with depression use [46]. To date, user attitudes, specifically towards mental health and wellbeing apps, and context of use have not been investigated in detail. Alongside efficacy, user experience and engagement are critical factors to the overall effectiveness of mental health technologies [47], therefore it is important to investigate what it is that the users themselves value.
In recent years a number of important approaches have emerged for examining mobile health apps. For example, MARS (the Mobile App Rating Scale) developed by Stoyanov et al. provides a tool for assessing the quality of mobile health apps [49]. It includes a dedicated section on engagement, that allows an assessor to score individual apps based on key engagement features. Chen et al. provide a framework specifically for evaluating mobile mental health apps [48], that allows patients and mental health service providers to evaluate app by their usefulness, usability, and integration and infrastructure. We view the approach applied in this paper as complementary, but distinct from these prior approaches. Instead of providing detailed analysis of individual apps, we focused on the thematic analysis and synthesis of user perspectives on engagement and therapeutic features across a range of apps. Analysing app reviews can provide insight on what end-users find engaging and how the apps are used and identify mismatches between what researchers believe to be important and what users actually find engaging. It can help us to better understand why current apps are highly rated and leverage this understanding in the design of compelling, evidence-based apps going forward. As an approach it can also offer distinct insights to recent app reviews that have incorporated traditional usability evaluations. For example, Huguet et al. [36] applied Nielsen's expert led, heuristic evaluation approach to assess the usability of CBT apps for depression. Whereas usability assessed the degree to which users can easily-or with minimal training-use and understand the app, our approach offers complementary insights on how and why people use apps and the particular features to which are engaging or unengaging. This paper has two key aims. Firstly, to systematically analyse the therapeutic elements and engagement approaches used in apps described as being based on CBT for depression. Secondly, to analyse publicly available user reviews of these apps in order to provide a more detailed understanding of the user experience and of what makes apps engaging and valuable to users.

Data collection
Our initial analysis of apps aimed at identifying two key groups: 1. CBT apps for depression: apps that self-identify as implementing CBT to target depression.
2. Mental wellbeing apps: a broader group of apps that includes apps that aim to address mental health problems such as anxiety and depression, but also issues such as stress, worry, mood or emotional wellbeing.
To limit the scope of our study we deliberately excluded apps targeting more severe disorders such as bi-polar disorder or less common disorders such as obsessive-compulsive disorder.
A detailed analysis of CBT apps for depression is the core focus of this paper. We consider both engagement features and therapeutic features (Study 1), and user opinions (Study 2). The set of apps addressing mental wellbeing more generally is not subjected to the same level of scrutiny as it is too large and too diverse in terms of therapeutic approaches applied. For these apps we analyse engagement features only to allow a comparison with engagement approaches found in CBT apps for depression. Figure 1 illustrates the overall systematic process used to identify relevant apps.

Phase 1: Initial keyword search and data clean up
We first defined the following groups of keywords: 1. General keywords related to mental health and wellbeing: "mental health", "mental wellbeing", "emotional wellbeing", depression, anxiety, stress, mood, emotions.
We used scripts [50,51]   Phase 2: Potentially relevant apps identified Next, we undertook the first manual screening. Following the approach used in [38], we manually reviewed the 1,680 apps by examining each app's title and short description. This allowed us to identify "not relevant" apps that would be excluded and "potentially relevant" apps that would be included in the next phase. We excluded the following types of apps: • Apps specifically addressing less common or more severe mental health disorders, e.g. substance misuse, OCD; or other health conditions, e.g. diabetes, chronic pain; • General health tracking apps and single-purpose wellbeing apps, e.g. for mindfulness meditation only. • Apps to support mental health professionals and students, and apps that require an access code (e.g. that are part of a study, insurance plan, employer wellness scheme).
• Apps not available in English.
1,297 apps were marked as not relevant and further 16 turned out to be duplicates. This resulted in a set of 367 "potentially relevant" apps.

Phase 3: Apps with clearly identified functionality
Next, we manually reviewed the full descriptions of all 367 "potentially relevant" apps. During this process, we identified 43 more apps that met the exclusion criteria described above; two apps that provided no information about their functionality; one app that was available for smartwatches only; and 37 duplicates. At the end of this phase we had identified 284 apps which we classify as "mental wellbeing apps".

Phase 4: Apps using CBT and targeting depression
The aim of this stage was to identify the final subset of apps that were self-identified as (1) focusing on depression AND (2) based on CBT. This resulted in 31 apps which we classify as "CBT apps for depression".

Study 1: Functionality analysis
The aim of this study was primarily to examine the functionality of the 31 CBT apps for depression. For each app we recorded both engagement features and therapeutic features. We defined therapeutic features as functionality that aims to help users manage their mental health and wellbeing, and engagement features as functionality that encourages regular use, makes app content more appealing and in general helps users to stay engaged with therapy or the app itself.
For each app, we recorded all features listed on its description page (e.g. mood-tracking, discussion forums, reminders etc.) or visible on screenshots; this approach has also been used in other app reviews [43]. We also noted mentions of expert involvement in app creation (health professionals, researchers, etc.). Features were recorded by the first author and regularly reviewed and discussed with others.
Next, to assess whether the features of 31 CBT apps for depression reflect CBT practice, we asked two researchers (a Clinical Psychologist who is also an accredited CBT therapist, and an HCI researcher experienced in designing technologies to support CBT) to independently match them against the elements in a recognised CBT Competence Framework [14]. For each feature they indicated (yes/no) whether it represented one of CBT competencies in relation to treatment for depression. Inter-rater agreement was 73% with raters disagreeing regarding eight items, including seven items where the disagreement was between a definitive answer (yes/no) and a "maybe". Disagreements were resolved through a discussion and the final categorisation is available in Table 1.
Finally, we also undertook a brief analysis of the engagement features available in mental wellbeing apps in general. A detailed review of the therapeutic features and approaches applied across this larger group was beyond the scope of this paper.
We used scripts [50,51] to automatically download all reviews for the 31 CBT apps for depression. If the app was available for both Android and iOS devices, we downloaded both sets of reviews. In total, we downloaded 2,904 reviews from 24 apps (seven apps had 0 reviews). To identify reviews for the analysis, we followed the approach similar to the one used in [43]: first, the first author manually assessed all reviews, recording their sentiment (positive, negative, neutral) and whether each mentioned at least one therapeutic feature; this was then discussed with other authors. Next, to qualitatively identify underlying themes, we used thematic analysis [52] to analyse the subset of reviews that mentioned at least one therapeutic feature. Coding was done by the first author and regularly discussed with others to allow for better familiarisation with the data and reduce potential for bias. This iterative process led to codes gradually being merged into broader categories and researchers identifying overarching themes.

Study 1: Functionality analysis
Overall, within the set of 31 CBT apps for depression we identified 26 therapeutic features and 10 engagement features; four additional engagement features were also available in the broader group of apps. App functionality is described in the following sections and features are summarised in Tables 1 and 2. Detailed information about the 31 CBT apps for depression, including their user ratings, available features and expert involvement are summarised in Multimedia Appendix 1.

Therapeutic features of CBT apps for depression
The most common features available in 31 CBT apps for depression focused on dealing with negative automatic thoughts (15 apps, or 48%) and negative thinking styles (nine apps, or 29%), and provided examples of activities users could do to improve their mood (nine apps). They also allowed users to record thoughts and emotions (three apps, or 10%), schedule daily activities (three apps), offered challenges and behavioural experiments (two apps, or 6%) or enabled goal setting (two apps). The apps also offered several "non-CBT" features, including writing and self-reflection, gratitude and affirmations, various tests and scales, or relaxation tracks. All features are summarised in Table 1. We also checked whether the presence of CBT features was associated with app ratings. Ratings were available for 24 apps with the average rating of 4.1 (on a 5-point scale), ranging from 2 for Activity Diary (which had one CBT feature) to 5 for MoodMaster Anti-Depression App (which had two). There was no difference in average scores between apps with no CBT features (average score = 4.1, min=4, max=4.2, N=3) and apps with at least one such feature (average score = 4.1, min=2, max=5, N=21). The average rating for MoodTools, the most comprehensive app, was 4.3.

Expert involvement
For each app, we recorded whether any experts were involved in the development process. Fourteen apps (45%) mentioned experts on their description page: health professionals (13 apps) and university researchers (two apps). Apps with expert involvement provided between 0 and 8 CBT features (N=13, mean=2.3, mode=2), while the number of CBT features in apps that did not mention experts ranged from 0 to 5 (N=17, mean=1.6, mode=1). While health professionals were involved in the development of MoodTools (the app with the most CBT elements), health professionals were also mentioned in the descriptions of Self-Esteem Blackboard, an app that did not provide any features matching CBT elements. In contrast, one app with 3 CBT features (MoodSentry) was built by a patient who wanted to share the methods that worked for him.
There was also no link between the presence of expert involvement and user ratings. For example, MoodTools and What's Up were the most rated apps with over 2,000 user ratings each, and both had the average score of 4.3. However, the former offered eight CBT features and experts were involved in its creation, while What's Up offered only two such features and no experts were involved. On the other hand, Cloud Clinic received no ratings at all, despite expert involvement and presence of CBT features; and both the lowest rated (Activity Diary) and the highest rated (MoodMaster Anti-Depression App) apps mentioned health professionals.

Engagement Features
For all mental health apps we identified, including 31 CBT apps for depression, we recorded details of features that aim to support user engagement with therapy or the app itself. They are summarised in Table 2. Among the CBT apps for depression, 18 apps (58%) provided explicit engagement features. Eight apps (26%) enabled sharing (including sharing with friends, family and therapists). Graphs and charts to illustrate progress (four apps, or 13%), audio content (three apps, or 10%) and notification and prompts (three apps) were also available. Two apps offered personalisation features and two other apps used gamification. Peer support and professional support can also drive engagement [47]; the former was available in two and the latter in only one app.
Among the 253 mental wellbeing apps, 149 (59%) provided engagement features. Graphs and charts were the most common (available in 17% of apps), and were used to visualise users' mood and progress. Five apps provided more detailed reports and summaries that often accompanied graphs and charts. Twenty-three apps (9%) provided various notifications and prompts: from reminders to record one's mood or interact with the app in some way to weekly summary emails and the ability to set up medication reminders for antidepressants. Interactive content included: video (10%), audio (7%) or game elements such as cartoon avatars, badges or progress bars (2%). Customisation and the ability to add own pictures and videos (six apps each) helped to tailor the experience to users' needs. Three apps also used a Q&A format to make reading materials more engaging and to help users select the right activities. Fifteen apps (6%) enabled contact with therapists, including one-to-one chats, forums where health professionals can answer questions or even the ability to schedule sessions.

Conclusions of Study 1
CBT apps for depression provided a mix of features. Even though all their descriptions mentioned CBT, only half of the apps provided features reflecting core competencies of CBT [14]. Moreover, most of them provided only one or two CBT features, and three apps did not provide any such features at all. This limited evidence-base of apps is in line with existing research [31,37,38].
Available CBT features tended to be limited, focusing around mood tracking, recording thoughts and emotions, and dealing with negative thoughts. As a result, they often lacked elements of CBT used in high-intensity interventions for depression, such as addressing core beliefs [14]. The presence or absence of features grounded in CBT practice was not linked with expert involvement in app creation, which raises concerns regarding the responsibility of app creators who may be misleading potentially vulnerable users by mentioning CBT without actually providing it. There was also no clear link in terms of expert involvement and user ratings. However, our results highlighted high user ratings for all apps, regardless of whether they provided CBT features, which corroborates previous findings [37].
Doherty et al. [47] discuss interactive features, professional support, peer support and customisation as key strategies to facilitate engagement with therapy. While similar proportion of CBT apps for depression provided engagement features compared to the other mental wellbeing apps (58% vs. 59%), these features were less varied. Interactive features such as video content, graphs and charts, bot interface that allows users to talk with a 'virtual' therapist or ability to add own pictures were more prevalent in the apps for mental wellbeing, although a bigger proportion of CBT apps offered audio content and gamification. Contact with peers or professionals was almost non-existent in CBT apps for depression, possibly because peer support is not part of standard CBT and having an open forum would require moderation to reduce potential risks. However, CBT apps for depression offered more customisation options, allowing users more options to adapt the features to their needs.
To better understand high app ratings and reasons for why people use and value these apps, our second study focused on users' attitudes towards app features and their experience of using CBT apps for depression.

Study 2: User reviews analysis of CBT apps for depression
Of the initially collected 2,904 user reviews, 1,287 reviews from 23 apps mentioned at least one therapeutic feature and therefore were included in the analysis; 1,179 (92%) of these reviews were positive, 95 (7%) were negative and 13 (1%) were neutral. The thematic analysis uncovered four key themes: different contexts of use; importance of privacy, security and trust; importance of engagement features; and the attitudes towards therapeutic features not related to CBT. The themes are described below.

Context of use of CBT apps for depression
Reviews often mentioned context of use and how the apps fit into users' lives. For one group of commenters, having a "pocket therapist" often meant using the app instead of therapy. It was often motivated by not being able to afford therapy or negative experiences in the past. The apps were also seen as simply better than regular therapy. In contrast, another group of commenters used the apps as an adjunct to treatment, alongside visits to a therapist. Users often commented on apps' usefulness and how well they enhanced their treatment.

"Great app. I always email my entries to myself so that I can print it out and share it with my psychiatrist." (Cognitive Diary CBT Self-Help)
In some cases, users were using the app because it was the therapist who suggested that in the first place.

"My therapist actually recommended this app and we trialed it in session, where it was really effective." (CBT Thought Record Diary)
Regardless of why people used the apps, they generally appreciated their role in supporting their mental wellbeing. Comparisons to a "pocket therapist" or "therapy sessions at the tip of their fingers" were frequent: "Love this app, it's like having a therapist in your pocket." (Depression CBT Self-Help Guide) Reviews were also written by therapists. They showed that sometimes clients were the ones who found the apps and integrated into their work with the therapist, and at other times therapists actively recommend apps to their clients.
"I have several therapy clients who use this app rather than writing out a thought log." (iCBT) However, feedback was not always positive. Some users advised caution and warned that the apps should not be used without supervision, or even at all.
"Can't replace a therapist, especially when you're first starting this therapy, but by directing the process, this is the next best thing." (Cognitive Diary CBT Self-Help) "Good design, bad idea. Apps shouldn't be diagnosing medical conditions. Consider suggesting the user seek medical help PRIOR to using the app, and that the app should only be used in conjunction with treatment. Not as a precursor to treatment or as a reason to get treatment." (MoodTools)

Importance of privacy, security and trust
As the apps play an important role in supporting users' mental wellbeing, knowing that the service was reliable and their data was secure was important. However, this was not always the case: often apps were unreliable and losing data was experienced as 'devastating'. The analysis also revealed the importance of privacy and security, and users often mentioned them alongside therapeutic features. People appreciated the presence of password protection or security locks, and demanded these features when they were not available. Users appreciated the discreet nature of the apps, and often compared them to paper worksheets, highlighting the privacy benefits the apps bring.

"I like that it's on my phone and therefore it's always in my pocket, so if I'm stressing out over an issue I can pull it out and deal with the stress immediately. To people around me it just looks like I'm checking my phone, playing a game or writing an email." (MoodKit)
In addition, this also meant that the users were more engaged:

Importance of engagement features
Sharing was the most common engagement feature among CBT apps for depression and users appreciated the ability to share data with their therapist. Graphs and charts (more common among the wider set of mental wellbeing apps) and notifications were mentioned mostly in feature requests.
"All I would improve is maybe reminder system. Like a reminder to update journal and to exercise mentally or physically." (MoodTools) One of the apps (What's Up) provided a forum where users could talk and support each other. This feature polarised the users, who either loved or hated it: Comments also highlighted the importance of customisation. Users were happy when the app was customisable, and demanded more flexibility when it was not.
"It has a lot of helpful wording and allows you to add your own, to personalize it." (Cognitive Diary CBT Self-Help) "There's also not many customization options with statements. For instance, as an atheist, statements like "I can turn this over to God" are just not helpful to me, so I'd like to be able to hide them." (Worry Box) Therapists using an app with their clients also wanted the ability to customise it, with one observing: "I would have liked to have a way to list your own alternative coping statements along with the canned ones." (Worry Box)

Attitudes towards therapeutic features not based on CBT
Users frequently commented on therapeutic features not based specifically on CBT, especially relaxation tracks and meditation. Commenters also appreciated examples of activities they could do to improve their mood and the ability to track anxiety episodes and worry (although the majority of those comments referred to Worry Box -an app designed specifically to deal with these issues). Writing in a diary was also often mentioned, although it was not always clear whether the mentions referred to simple journaling or a structured thought diary. The majority the users valued having both therapeutic features based on CBT and other approaches.
"For a free app that provides cbt logs, meditation, relaxation training, the ability to track some of you depression symptoms & provide suggestions to get you up and moving -it's a well designed little app." (Depression CBT Self-Help Guide) However, some indicated that the use of CBT was important because of its strong scientific or evidence base and did not like more eclectic apps: "This app is presented as a straightforward cognitive therapy app, but is riddled with pseudo-spiritual New Age nonsense. Avoid." (Depression CBT Self-Help Guide) Some therapists using an app with their clients also were positive of the integration of these different types of features in a single app, with one commenting:

"I was looking for an app to use with clients in my clinical work and I really like this one. Having both the relaxation tracks and a way to challenge unhelpful thoughts was great." (Worry Box)
Furthermore, the main 'criticism' voiced by many users, together with associated feature and customization requests, was the absence of positivity.

Conclusions of Study 2
The analysis of user reviews showed that users appreciated all therapeutic features, including both the ones based on CBT as well as on other approaches, and so did professionals using the apps, which can explain the lack of correlation between high ratings and presence of evidence-based features reported in other studies, e.g. [36,37]. Moreover, users wanted those other, "non-CBT" features, especially the ones focusing on more positive experiences.
Apps were often mentioned in the context of therapy: as "pocket therapists", they often replaced or augmented therapy, allowing users to take active role in supporting their own mental wellbeing. Their discreet nature was particularly important. However, reliance on the app and the type of data users entered meant that privacy, security and trust were important. And when that trust was violated, e.g. when the app lost the data, it had serious consequences, leaving users devastated.

Principal Results
We presented two studies that investigated factors that make mental health apps engaging: in Study 1 we examined engagement features available in CBT apps for depression, and the relationship between the presence of CBT features and expert involvement and the app ratings; and in Study 2 we thematically analysed publicly available user reviews to understand user experience and contexts of use. Our results show that apps are used in a wide range of contexts, both replacing and augmenting therapy, and allowing users to take active role in supporting their mental health and wellbeing. Users, including health professionals, valued and used apps that incorporated both core CBT and non-CBT elements, but concerns were also expressed regarding the unsupervised use of apps. Positivity was seen as important to engagement, e.g. in the context of automatic thoughts, users expressed a preference to capture not just negative, but also positive ones. Privacy, security and trust were crucial to the user experience. We discuss these findings below.

Integration into different therapeutic practices
The results showed that apps were used as part of different therapeutic practices: as part of therapy as well as tool for self-management. Such a mixture of different features and varying contexts of use provides a challenge to app developers, but at the same time opens up opportunities for integrating apps into therapeutic practice. The prevalence of features related to key CBT concepts (e.g. negative automatic thoughts, mood tracking) that do not necessarily require input from a therapist and are often covered in self-directed computerised CBT suggests that apps may in particular be a useful addition to low-intensity CBT. They may be particularly effective in facilitating engagement with CBT "homework" (which often requires recording thoughts and emotions, planning activities or tracking mood), which is a desirable outcome as regular engagement with such CBT exercises increases the effectiveness of therapy [53,54]. Moreover, the results show that apps have important advantages with regard to 'integration into practice': they are always at hand, are more private and discreet than paper worksheets, and they allow to easily share data with the therapist. Taking a smartphone out when in a shopping queue or on a bus is perfectly 'normal' behaviour, i.e. it is familiar to the individual and (usually) acceptable to those around them. Therefore, people can integrate such exercises into everyday life, and reduce concerns about "getting caught" completing therapy tasks in public [55].

Engagement with therapy
Engagement with therapy or therapeutic content can be achieved through different means, including interactive features, peer or professional support, or customisation [47]. Each of these approaches was found to some degree across the apps we examined, although CBT apps for depression lacked features enabling contact with others and more interactive appbased engagement techniques such as bots or the ability to upload own content. Study 2 findings suggest that reminders to use the app, in particular, are something that users would value. However, care must be taken when designing such reminders to ensure they are effective and not annoying [56]. As with other health apps (e.g. menstruation trackers [57]), they also need to ensure privacy: it was clear from the user comments that people shared their phones with others and thus would want to keep private the fact they use an app to support mental health. In exploring this area many lessons could be drawn from existing literature on behaviour change [21], health promotion [7] or medication adherence apps [43].
It is common in CBT interventions that therapists provide personalised examples or tailor exercises based on a client's needs [14]. Study 1 showed that personalisation features were more prevalent among CBT apps for depression and Study 2 showed that both users and therapists appreciated the ability to customise apps, which reflects this personalised nature of therapy. This raises questions of how best to support customisation of exercises. One option is for the customisation to be done by the therapist, and appropriate content 'uploaded' into the client's app. A second option involves a therapist and client working together to customise the app, e.g. to set specific exercises for the client to complete between therapeutic sessions. Alternatively, the user (outside a therapeutic relationship) could make such modifications themselves, which some apps already allow. However, when customisations have the potential to impact on the therapeutic effectiveness of an app, putting such abilities in the hands of the user would require responsibility on the part of the app designer, to help ensure that such customisations are therapeutically appropriate. The means of implementing this is an important subject for future research.
Building on an evidence-base from multiple fields The analysis of our data suggests that many end users, including therapists, valued flexibility in the use of therapeutic approaches. Similar to [37], we found that many apps mixed both CBT and non-CBT features. Some users felt this was inappropriate, particularly when they thought the latter were not evidence-based. However, they were a minority and a number of commenters expressed a desire for features beyond standard CBT, such as the ability to record positive emotions. While positive logs are used in CBT, the apps tend to focus on automatic negative thoughts and negative thinking styles. Moreover, some therapists seemed to approve the combination of CBT techniques with, for example, relaxation audio.
Many users find such a blend to be more engaging, even if not faithful to a core CBT model [14]. Given the importance of engagement in achieving effective outcomes, this raises important questions for the design of mental health apps and potential benefits of non-CBT features in CBT focused apps.
We believe these findings raise interesting questions regarding the potential of eclectic approaches to the development of mental health apps, where different techniques, based on different theoretical approaches are applied in combination. Prior research on eclecticism emphasises that it should not be construed as antitheoretical. Rather, eclectic approaches should be guided by some integrating framework that gives coherence to the overall therapeutic process. Within the context of traditional, face-to-face mental health interventions, several such frameworks have emerged [60]. Debate regarding the efficacy of eclecticism versus adherence to a core intervention model has a long history and is the subject to ongoing research (see for example [58][59][60]). A detailed discussion of this literature is beyond the scope of this paper. However, it is important to recognise that prior research emphasises that eclecticism should not be construed as antitheoretical. Rather, eclectic approaches should be guided by some integrating framework that gives coherence to the overall therapeutic process. Within the context of traditional, face-to-face mental health interventions, several such frameworks have emerged [60]. A more detailed understanding of how eclecticism can be effectively supported through apps is an important subject for future research. Promising initial work has been carried out by Mohr et al. [26], who developed IntelliCare -an eclectic suite of apps that provide a wide range of features (their Thought Challenger was among the set of 31 CBT apps we identified). In addition, the most comprehensive app in our set, MoodTools, has been developed by health professionals, and is a good example of how different approaches can be blended. The potential benefits of combining CBT with approaches from Positive Psychology [61,62] is another clear message from our work. Also, while it is crucial that apps should draw on clinical theory, this alone is not enough. They also need to be based on research grounded in other domains such as HCI, which provide evidence of effective approaches to building desirable and engaging computer systems, e.g. [23,47].

Responsibility towards the users
With potential users expressing different needs, and therefore using the apps in different ways, the above discussion raises the question of the responsibility held by app designers or sellers. Regulating health apps is a challenge [34], raising several ethical and practical issues. Should designers take responsibility for the ways in which their app is used? Should they 'police' it? It is clear that some users felt that certain apps should not be used by someone untrained in CBT or outside a therapeutic relationship. Assuming this view is correct, how can designers ensure that their app is not used inappropriately? Is there potential for the identification and warning of inappropriate use patterns? Alternatively, it may be appropriate to block unsupervised use of apps, e.g. through access codes provided by professional therapists to their clients, as used by the Pesky gNATs app [63] and others.
This responsibility also extends to the stability and reliability of the app. While an app failure in general can be inconvenient and annoying, it can have serious consequences in the context of mental health apps: someone who has come to rely on an app for emotional support can find a failure 'devastating'. This is also an issue with other health-related apps, e.g. app updates can lead to a loss of scheduled reminders from medication adherence apps [43]. Therefore, reliability and backwards compatibility of upgrades should be tested more thoroughly than for other types of apps. New providers entering the market should think seriously whether they can take on such responsibility.
This responsibility arguably extends to the app stores as well. App stores encourage thorough testing of the apps prior to launch [32,33], routinely examine the technical implementation of apps and block apps that do not meet the required technical standards.
If an app is available in the health section, and offers mental health support, should the app stores be expected to police its quality from a health/clinical perspective? Perhaps such apps listings should be required to explicitly state which health professionals (if any) were involved in its design, what evidence (if any) is available for the techniques it offers, and also provide guidance as to which contexts of use the developers feel are appropriate. Guidelines for evaluating health apps [64], objective app guides [65], and dedicated health app stores (e.g. one curated by the UK National Health Service [66]) are a good first step, but they do not solve the potential issues with apps widely available in commercial app stores.

Limitations
The to assess whether the features of 31 CBT apps for depression reflect CBT practice compared them to a recognised CBT Competence Framework [14]. It is important to recognise that this competence framework was developed for assessing competency of face-to-face therapists, not for CBT apps. Given the current absence of a widely recognised competence framework for CBT apps for depression we believe this was the best approach.
Our focus on app stores means that the results reflect apps that are commonly available, rather than apps that are the current state of the art in research. We believe that by exploring the experience of current users, our work complements other research and can help to inform future designs. Similarly, given the focus on user reviews in Study 2, there is an obvious potential for selection bias towards extreme ratings and positive reviews [67,68]. However, we believe that focusing on specific features and context of use allowed us to reduce this bias while still providing relevant insights. While the reviews were coded by one author, the coding and the data were regularly discussed with the rest of the team to ensure everyone is familiar with the data, which is an acceptable approach in qualitative analysis [52,69,70]. Overall, whilst not definitive, our approach allowed access to user experience data that would otherwise be extremely difficult to obtain. It provided data on experience across a large number of different apps, which people have used over different durations, as part of their daily life. In-depth studies of individual apps do provide stronger data, but not the same breath of coverage. They also proscribe how a user in a study is expected to use the app, and so provide less insight into the development of 'in the wild' usage patterns in response to personal situation.
Conclusions and future work Drawing from a rich pool of public app reviews, our research shows that users use apps alongside and instead of therapy, with the same app being used in both contexts. It also suggests that features not considered evidence-based may be key to facilitating user engagement. The challenges and opportunities we have identified open up new avenues for research. Future work should explore approaches to integrating apps into different therapeutic practices, facilitating engagement, finding a balance between drawing form clinical and design research, and exploring different approaches towards responsibility and accountability, and the role of app stores as gatekeepers.