Usability of the experience sampling method in specialized mental healthcare: A pilot study

Background: Although mental health problems occur in interaction with the natural environment, bringing this contextualized information into the therapy room is challenging. The experience sampling method (ESM) may facilitate this by assessing clients' thoughts, feelings, symptoms, and behavior as they are experienced in everyday life. However, ESM is still primarily used in research settings with little uptake in clinical practice. One aspect that may facilitate clinical implementation concerns the use of 'ESM protocols', which involves providing practitioners with ready-to-use ESM questionnaires, sampling schemes, visualizations, and training. Objective: This pilot study's objective was to evaluate the usability of an ESM protocol in clinical practice using a mixed-methods approach. Methods: In this pilot study, we created an ESM protocol and tested its usability in clinical practice. The ESM protocol was tailored to the m-Path software platform, consisting of a dashboard for practitioners and an app for clients. The dashboard was used to configure an ESM questionnaire template we designed. Additionally, the dashboard contained custom data visualizations that were made based on end-user feedback. The app was used for completing ESM assessments. A total of 8 practitioners and 17 clients used ESM in practice between December 2020 and July 2021. Usability was assessed using questionnaires, ESM compliance rates, and semi-structured interviews. Results: The usability was overall rated reasonable to good by


Table of Contents
1) Would you like to publish your submitted manuscript as preprint?Please make my preprint PDF available to anyone at any time (recommended).
Please make my preprint PDF available only to logged-in users; I understand that my title and abstract will remain visible to all users.
Only make the preprint title and abstract visible.
No, I do not wish to publish my submitted manuscript as a preprint.2) If accepted for publication in a JMIR journal, would you like the PDF to be visible to the public?
Yes, please make my accepted manuscript PDF available to anyone at any time (Recommended).
Yes, but please make my accepted manuscript PDF available only to logged-in users; I understand that the title and abstract will remain v Yes, but only make the title and abstract visible (see Important note, above).I understand that if I later pay to participate in <a href="http

Original Manuscript 1. Introduction
Mental problems are inextricably linked to daily life, meaning they do not exist in a vacuum but are influenced by our everyday activities, environments, and social interactions.For example, for individuals diagnosed with schizophrenia spectrum disorder, work-related activities may decrease hallucinatory intensity over time (Delespaul et al., 2002).In contrast, paranoid ideation may develop in response to the social environment (Collip et al., 2011) or stress (Myin-Germeys et al., 2007).This indicates that mental health problems can best be understood when they are investigated in the context in which they occur: a client's daily life.Unfortunately, however, practitioners are often confined to the brick-and-mortar walls of their therapy room and must rely on the clients' ability to retrospectively report their feelings, thoughts, and behaviors.Unfortunately, this may provide only partial insights due to selective or incomplete recall biases (Colombo et al., 2020), as one cannot expect clients to accurately remember and share all relevant daily life experiences and associated emotions or symptoms.Therefore, complementary tools that can aid clients in reliably sharing their daily life experiences with their practitioners could be valuable for clinical practice.For instance, such tools may benefit therapy by increasing insight into the contextual variability of mental health problems as they are observed across different contexts in people's daily life.While this was not feasible for many decades due to practical restrictions, the facilitation of smartphone technologies can now enable practitioners and clients with new digital tools to bring this highly relevant information into the therapy room.
The Experience Sampling Method (ESM; Csikszentmihalyi, Larson & Prescott, 1977;Myin-Germeys et al., 2018;Myin-Germeys & Kuppens, 2022) is a structured diary technique that involves using smartphone apps to assess thoughts, feelings, symptoms, and behavior in clients' daily environments.Individuals are prompted to complete a brief questionnaire multiple times a day for several consecutive days.The questionnaire is completed 'in the moment' and typically contains questions about people's momentary thoughts, feelings, symptoms, behavior, and situational circumstances (e.g., 'Who are you with?', 'Where are you?').Individuals may also be asked to rate sleep quality in the morning or evaluate the day in the evening.
The ESM may have value for clinical practice for several reasons.First, the real-time assessment of ESM questionnaires implies that the risk of recall bias is reduced compared to traditional assessment methods such as clinical interviews or retrospective questionnaires (Stip, 1996;Vrijsen et al., 2017).Second, through self-monitoring multiple times per day, ESM may increase self-insight and emotional self-awareness (van Os et al., 2017).In a similar vein, ESM may also help identify protective factors in the environment that may increase resilience, such as social networks that can provide support in moments of high distress (Bjørlykhaug et al., 2021).Third, ESM may help practitioners explore and develop hypotheses about the underlying factors for mental health problems.Similarly, ESM could help evaluate whether the provided treatment has the desired effects on clients' everyday lives (e.g., improved mood and reduced symptoms), and, if required, to make changes to the treatment plan collaboratively with clients (Myin-Germeys et al., 2018).
Emerging evidence shows that mental health practitioners and clients recognize ESM's potential advantages and are generally interested in using it in clinical practice (Bos et al., 2019;Piot et al., 2022), although some findings also suggest that practitioners may not adopt ESM more readily than traditional assessment tools (Ellison, 2020).However, clients might be more favorable toward the actual use of ESM in clinical practice than practitioners (Frumkin et al., 2020).For instance, meta-analyses found good compliance rates to ESM questionnaires -above 75% on averageamong clinical samples (e.g., Rintala et al., 2019).However, implementing ESM in mental healthcare has proven challenging as implementation attempts so far have failed to instigate continued or far-reaching use (e.g., Bastiaansen et al., 2018;Bell et al., 2020;Daniëls et al., 2019, Frumkin et al., 2021).One possible reason for this may be that the ESM software used is designed for research purposes and created without end-user input from practitioners and clients.Using software originally made for research purposes in clinical practice could be problematic, as disregarding end-user expectations may cause usability issues.For example, practitioners mentioned in previous work that ESM software was time-intensive and not intuitive, making it challenging to use the ESM data clinically (Daniëls et al., 2019).These findings suggest that an in-depth investigation into end-user software requirements may be necessary before using ESM as a clinical tool in mental health care becomes feasible.
In response to the lack of end-user perspectives on clinical ESM software requirements, we recently conducted a qualitative focus group study with mental healthcare practitioners to understand better how they wanted to use ESM and which elements this would require (Weermeijer et al., 2023).
One important finding that emerged was the need for 'ESM templates' detailing the ESM questionnaire's content and sampling schedule so practitioners do not need to develop this for each client.At the same time, practitioners stressed that personalization should still be possible, such as creating new ESM items or tailoring a client's schedule.Furthermore, they recommended using intuitive data visualizations such as line graphs depicting mood variability over time or pie charts displaying the frequency of contact with others (e.g., family vs. friends).Finally, practitioners expressed a need for training and guidelines on using ESM templates, personalization, and data visualizations.Similar findings were recently found in a survey among 89 practitioners (Piot et al., 2022).
While these initial findings offer some insights into the requirements of software for clinical ESM applications (e.g., templates, intuitive data visualization) and the implementation strategies more generally (e.g., need for user training), it is important to consider that practitioners had no prior experience with using ESM in clinical practice.Therefore, whether accommodating these identified needs will make ESM effectively usable in routine mental health care (i.e., intended versus actual use) remains to be investigated.Hence, a practical next step is to create an ESM protocol based on these recommendations and evaluate end-user experiences by piloting the ESM protocol with practitioners and clients before wider dissemination (Hwang & Salvendy, 2010).Such a multi-tiered approach provides meaningful information about experienced barriers, facilitators, and requirements for future ESM implementation efforts.To this end, we designed a protocol called IMPROVE ('IMplementing Personalized Real-time mOnetoring in eVeryday life), in which ESM templates containing scheduled ESM questionnaire content and sampling schedules can be tailored to the needs of clients who self-monitor for a week using the ESM app.The collected information is automatically visualized through intuitive visualizations on the practitioner's dashboard.The dashboard and app were made in collaboration with the m-Path software platform (Mestdagh et al., 2022), and the user training and guidelines were based on previous work (Bos et al., 2019, Weermeijer et al., 2023).

Objective
This pilot study's objective was to evaluate the usability of IMPROVE.Specifically, we were interested in whether the IMPROVE dashboard and app were considered acceptable and easy to use and whether practitioners and clients were satisfied with the user-interface design.For clients, and while thresholds are somewhat arbitrary (Weermeijer et al., 2022), we also wanted to investigate whether they would complete at least one-third of the assessments.Additionally, we were interested in practitioners' and clients' perspectives on the ESM items and sampling scheme, options for personalization, the design of the dashboard and data visualizations, and the training material.To accomplish these objectives, we used a mixed-methods approach of which the associated analysis plan was pre-registered (Weermeijer et al., 2021) 2. Methods

Participants
The study targeted mental health professionals working at KU Leuven's University Psychiatric Centre in the Flanders region of Belgium.Practitioners were recruited exclusively via email due to COVID-19 restrictions at the time of recruitment.To be eligible for participation, practitioners had to be either certified psychiatrists or psychologists working with clients suffering from mental health problems and have good Dutch language proficiency.Practitioners recruited clients into the study based on their assessment of whether IMPROVE would be helpful for the presented client.Clients were only required to be 18 years or older, receive residential or ambulant mental health care, have good Dutch language proficiency, and own a smartphone with at least 3G coverage.We employed these broad selection criteria to ensure a comprehensive and realistic assessment of the usability of IMPROVE across clients presenting with different mental health problems and at different treatment stages.Practitioners and clients were requested to provide written informed consent, and the study was approved by the Medical Ethics Committee of KU Leuven (Leuven, Belgium, identifier S64244).

Procedure
Practitioners first provided written informed consent and completed an enrolment

IMPROVE dashboard, app, and training material
The IMPROVE Protocol used m-Path's ESM software platform (Mestdagh et al., 2022), which consists of an app to deliver ESM questionnaires and a dashboard on which ESM questionnaires can be made and data visualized.Specifically, making use of the custom-made 'ESM template' feature (developed by m-Path for this project), we provided practitioners with an ESM template that contained a ready-to-use ESM questionnaire that followed a predefined sampling scheme (i.e., 10 ESM assessments per day for six days + morning/evening assessment).The ESM monitoring period started one day after a client had downloaded the ESM app and registered with their practitioner.In addition, the template could be personalized as practitioners could add a maximum of three ESM questions from a library, adjust existing multiple-choice options to include more answer options (e.g., Who are you with?), and modify the sampling schedule to fit sleep-wake patterns.The ESM content consisted of a morning questionnaire assessing sleep quality and motivation to start the day, an evening questionnaire evaluating people's day overall, and an ESM questionnaire containing questions on mood, location, and activity (Table 1.).While the morning and evening questionnaires were assessed once daily, clients received the ESM questionnaires ten times per day for six consecutive days between 7:30 am and 10:30 pm by defeault (e.g., Myin Germeys et al., 2003).Five minutes after receiving a beep without a response, a reminder was sent to fill out the ESM questionnaire.The app was synchronized with the dashboard, and responses were automatically visualized on the dashboard.Implementation strategies included a user manual with guidelines, including video links on using the dashboard and one-on-one online training sessions.1.I thought this was a normal day 2. I thought this was a nice day 3. What was the most NEGATIVE event of the day for you?[Open text] 4. How unpleasant was this event?5. How enjoyable was this event?6. Completing the questionnaires on this app has influenced my mood throughout the day 1 Unless specified otherwise, items were answered on a scale ranging from 1 'not at all' to '7 'very much'.For additional content, please see our preregistration page.

ESM data visualizations
The data collected with the ESM app were displayed on the online dashboard using a series of custom-made visualizations.These visualizations were based on end-user input on existing visualization methods for ESM data (Weermeijer et al., 2023), including information on general psychological functioning (i.e., boxplots of positive and negative emotions) and contextual information across the ESM period (e.g., pie-charts expressing the distribution time alone vs. with others face-to-face or online, activities in daily life), fluctuations over time (e.g., time-series graphs of negative affect), and qualitative text tables with descriptions of the most pleasant event of the day.
For the time series plot, it was also possible to zoom in on a single measurement point and relate the data point of interest to contextual information (e.g., with whom someone was, where they were, and what they were doing).The frequency (pie)charts could also be made conditional on contexts or activities to provide insight into the contextual determinants of mental health problems (e.g., symptom frequency at home vs. symptom frequency at work). Figure 1 provides examples of visualization used in IMPROVE.At enrolment, clients and practitioners provided age and gender.In addition, practitioners provided information on their profession (e.g., psychiatrist, clinical psychologist).

Compliance
Compliance with the ESM protocol was assessed as the percentage of completed versus scheduled ESM assessments.

Questionnaires on Usability
To evaluate the usability of the ESM app and dashboard, we used an adapted version of the mHealth app usability questionnaire (MAUQ; Zhou et al., 2019).The MAUQ assesses different usability elements, such as whether clients and practitioners found the app and dashboard easy to use or were satisfied with the user interface (e.g., 'I found it easy to learn to use the dashboard.').Items were rated using a 7-point Likert scale from 1 to 7, with higher scores indicating higher usability of the app for clients and dashboard for practitioners, respectively.Given that these statements assess different meaningful usability aspects, we interpreted individual items and did not calculate a composite score.For practitioners, we differentiated between the usability of the dashboard during a clinical session and overall usability.We made this decision as practitioners could use IMPROVE several times with multiple clients, whereas clients only used IMPROVE with one practitioner.

Qualitative measures
Practitioners and clients were invited to participate in a semi-structured interview after using IMPROVE.The interview assessed in-depth experiences on using ESM in practice, provided training material, the ESM items and sampling scheme, personalization options, data visualization, and suggestions for improvement.Interview guides were developed and divided into thematic sections (e.g., expectations regarding ESM implementation in mental health care, technical feasibility, and the ESM template), with each section starting with a short introduction to the topic.Sections were composed of broad questions followed by more specific prompt questions.The interviewers were allowed to make minor changes to the phrasing of the questions to make them more natural but were instructed not to change the content and meaning of the questions.

Quantitative data analysis
Concerning compliance, we evaluated whether clients provided sufficient data for making reliable interferences of automated data analysis at the intra-individual level (cf.Kimhy et al., 2006).
While thresholds are somewhat arbitrary (Weermeijer et al., 2022), we assessed whether clients could complete a minimum of 33% of all provided questionnaires.For the adapted MUAQ, scores of individual items that assessed usability were visualized and interpreted using heat maps (Annex 1-3).

Qualitative data analysis
The audio recordings of the interviews were transcribed and analyzed based on inductive data-driven thematic analysis (Braun & Clarke, 2006).This involves several consecutive steps.First, the first author familiarized himself with the data by re-listening to the audio recordings.Second, the transcripts were reread and divided into meaningful text segments.Third, these segments were labeled with short summarizing and comprehensible sentences (i.e., open coding approach).
Afterward, the labeled segments were grouped into sub-themes.Subsequently, these sub-themes were grouped into overarching themes.How the different segments were grouped into sub-and overarching themes was refined through collaboration with co-authors.Additionally, we included a second coder to evaluate the reliability of the results.The second coder was provided with 10% of the unlabeled segments of the first author (randomly selected) and was asked to label them and group them into sub-themes and overarching themes.Afterward, we compared coding and evaluated the labeling and grouping agreement to assess interrater reliability.Cohen's Kappa was subsequently calculated as the percentage of agreement.

Drop-outs & missing data
Given that drop-outs may indicate poor usability, the number of drop-outs is reported for clients and practitioners.

Participants
We invited 142 practitioners to participate in the study, of which 12 initially agreed to participate, and 11 followed the optional online training session.Eight of these 12 practitioners used the IMPROVE protocol with clients in therapy.The four practitioners who did not use the IMPROVE protocol mentioned the excessive burden of trying out novel instruments amid the COVID-19 pandemic and/or clients not showing up for scheduled appointments as reasons for not using IMPROVE.
Practitioners invited 29 clients to participate (mean = 2.42 clients per practitioner), 24 agreed to participate, and 17 completed the study.Drop-out occurred at various points: one client decided to quit during baseline data collection, and six clients ended the ESM week but did not attend the clinical feedback session or did not complete the usability questionnaires.Table 2 provides the demographic information of the practitioners and clients that participated in the pilot study.

Usability of the dashboard during a clinical session
Regarding the usability of the ESM dashboard during a clinical session, mean responses to individual items ranged from 5.33 (SD = 0.91) to 6.06 (SD = 0.73).Practitioners reported the lowest agreement to the statement 'When I made a mistake, I could correct it easily and quickly.',whereas the highest agreement was reached for the statement 'I felt comfortable talking to my client about the data that was visualized on the dashboard.'.When inspecting the heat map of responses (Annex 1), we observe saturation around '6 -agree' for all statements apart from responses to the statement 'The information on the dashboard was well organized, I could easily find what I needed for this session.', for which responses were saturated around '5 -somewhat agree'.

Overall usability of the dashboard
Regarding the usability of the ESM dashboard overall, mean responses to individual items ranged from 4.00 (SD = 1.91) to 6.14 (SD = 0.69).The lowest and highest level of agreement was reached for the statements 'It was easy for me to learn to use the dashboard.'and 'I would use the dashboard again.' respectively.In the plotted heat map (Annex 2), we observe saturation for 8 out of 11 usability statements labeled '5 -Somewhat agree', '6 -Agree', or '7 -Strongly Agree'.The three usability statements which deviate from this pattern were 'It was easy for me to learn to use the dashboard.','It was easy for me to create questionnaire content.',and 'It was easy for me to (re)schedule questionnaires.'

Client ratings on the usability of the ESM app
Mean responses to the app usability statements varied between 4.18 (SD = 1.7) and 5.94 (SD = 1.5), with the lowest and highest level of agreement with the statements' The app had all features and capabilities I expected of it.'and 'I felt comfortable talking to my clinician about the data collected with the app.'.For nine out of 12 statements, the largest percentage of responses consisted of '5 -Somewhat agree', '6 -Agree', or '7 -Strongly Agree'.The remaining three statements -which mainly concerned the design and user interface of the app -were characterized by a large degree of variation.For example, 35.3% of clients answered 'Agree' to the statement 'From the start, I found the app easy to use.', whereas 23.5% answered 'Disagree' (Annex 3).

compliance to scheduled ESM assessments
On average, clients completed 55% (SD=25%; range 18%-93%) of the ESM questionnaires (excluding morning/evening questionnaires).However, five clients did not reach the predefined threshold of 33% (range 18-31%), indicating limited feasibility for reliable inference.In addition, one client stated unwillingness to share their ESM data with the research team.

Qualitative data analysis
The overarching themes were using ESM in clinical practice, training material, ESM content, personalization, data visualization, and suggestions for improvements.For each of these themes, subthemes were identified.Figure 5 summarizes the results, and Annex 4 includes example quotes.

Figure 2.
Summary of the results from thematic analysis on usability.

Using ESM in clinical practice
Practitioners appeared engaged with the software, with multiple practitioners expressing a desire for continued use (e.g., annex 4, quote 1).However, practitioners mentioned that several clients declined to participate because they considered a week of ESM with the default sampling scheme too burdensome or had no smartphone or internet (annex 4, quote 2).Some clients that did participate also voiced these concerns and indicated that the ESM sampling scheme was too burdensome or that the noise/vibration from a notification was disturbing in some situations, such as during a group therapy session or relaxation exercise (annex 4, quotes 3-4).Additionally, one client reported that it felt artificial to reduce emotions and cognitions to ratings on a scale and felt that they were expected to show variation in their responses (annex 4, quote 5).

The training material
The training material we provided was generally considered useful but not practical.One practitioner, for example, indicated difficulties following the online training on a tablet, indicating a preference for group training with colleagues present (annex 4, quote 6).Another practitioner mentioned that the manual we provided would take too much time to go through and instead relied on learning from a colleague also enrolled in the study (annex 4, quote 7).Similarly, one client said they felt confused about the necessity of the repetitive nature of ESM (annex 4, quote 8) -indicating insufficient briefing by the practitioner before the implementation of ESM.

The ESM items and sampling scheme
Mixed opinions existed on the usability of the predefined ESM questionnaire.Practitioners found the default ESM items (e.g., questions on emotions, context, and activities) relevant but were skeptical toward some of the add-on items, stating they were unsatisfied with the phrasing (e.g., a focus on 'burden' in the assessment of OCD symptoms; annex 4, quote 9).Similarly, some practitioners found a single week of ESM too brief to detect meaningful changes and indicated that brief periods of intensive ESM might be more appropriate during the initial stages of therapy (annex 4, quote 10).Similar to practitioners, clients generally found the default ESM content relevant for treatment.For instance, they mentioned that it increased insight and made them more aware of their feelings and behaviors (annex 4, quote 11).

Personalization of the ESM questionnaire
Practitioners generally found the options for personalization valuable (annex 4, quotes 12-13).However, most practitioners mentioned that they rarely personalized the ESM questionnaire, which was perceived as too complex and time-consuming (annex 4, quote 14).One practitioner, for example, stated that they might need assistance in customizing the ESM questionnaire as they were 'not good with technology' (annex 4, quote 15).

Data visualization of collected ESM data
While some practitioners liked the visualizations and indicated that they helped them and their clients to concretize the contextual nature of mental health problems (annex 4, quotes 16-17), others found it initially overwhelming and indicated the need for practice to make sense of the different visualizations (annex 4, quote 18).Similarly, while some clients considered the data visualizations informative (annex 4, quote 19), others found it overwhelming and challenging to know what was relevant (annex 4, quote 20).To illustrate, one client reported that they would not use ESM without a practitioner to help them understand how to interpret and give meaning to the results (annex 4, quote 21).

Suggestions for improvement
Practitioners and clients made suggestions for improvement related to different elements of the ESM protocol.First, regarding the ESM template, practitioners and clients indicated the need for alternative sampling formats compared to a single observation period with ten beeps daily.For instance, a practitioner suggested using ESM at different periods in the therapy to evaluate progress (annex 4, quote 22).Relatedly, clients indicated sampling one week a month with fewer beeps to make it less burdensome (annex 4, quote 23).Second, practitioners reported that additional training is necessary, which could include more case descriptions and mock sessions (annex 4, quote 24).
Third, concerning the ESM content, some clients found that the ESM questionnaires were too generic and suggested using open questions and responses (annex 4, quotes 25-26).Relatedly, additional add-on questions were requested by practitioners and clients to monitor a broad range of experiences related to substance abuse, obsessive-compulsive disorder, stress, and physical health.
Fourth, covering data visualizations, some practitioners expressed the desire for adjustable visualizations, such as making it possible to annotate and adjust visualizations (annex 4, quote 27).
Finally, clients said using more color or highlighting important parts of a question might be worthwhile to make filling out the ESM questionnaire less monotonic and more time efficient (annex 4, quotes 28-29).

Discussion
Despite the potential benefits of ESM to make clients more actively involved and better match treatments to their needs (e.g., van Os et al., 2017;Bjørlykhaug et al., 2021;Myin-Germeys et al., 2018), ESM is still primarily used in research settings with little uptake in clinical practice.To the best of our knowledge, this is the first study to pilot the usability of an ESM template in a specialized mental healthcare setting for psychiatric clients.This consisted of ready-to-use ESM questionnaires, sampling schemes, visualizations, and add-on materials.The ESM template was implemented through a dashboard for practitioners (i.e., including the setup of the template and data visualizations) and an app for clients (i.e., for completing ESM questionnaires).Our results indicate that working with ESM templates can facilitate usability, but a single generic template is insufficient to capture clients' needs and address clinical goals in practice.
Although clients were somewhat less favorable than practitioners, we observed that the technical usability of the piloted software was considered sufficient by practitioners and clients.
Practitioners believed the template easy-to-use and indicated a willingness to use it again.At the same time, however, they also expressed more difficulties using more advanced features (e.g., personalization) and displaying and interpreting data visualizations.While clients considered the template also usable, their experiences were more variable.For instance, several clients indicated that the tool did not include expected features, such as being able to provide additional momentary information in an open text field.Building upon existing research on hypothetical use (Bos et al., 2019;Piot et al., 2023;Weermeijer et al., 2023), these findings provide empirical evidence for the acceptability and technical usability of ESM in clinical practice.
Despite technical usability being sufficient, clinical usability was not self-evident.Overall, a need was expressed for more personalization.Even though practitioners stressed the importance of personalization, and the software provided the opportunity to adapt the content and sampling scheme, practitioners rarely used this possibility.There may be two possible explanations for this finding.First, it may be too complex from a technical point of view, as practitioners indicated that personalization was too difficult and time-consuming.This aligns with prior research suggesting that future implementation must strike a balance between the need for personalization and the clinical reality of the limited time practitioners have during clinical sessions (Ellison et al., 2021;Piot et al., 2023).Second, a lack of personalization may also be explained because it is not straightforward how to operationalize clinical questions in ESM templates (Daniels et al., 2022).This suggests that future implementation work should provide a better understanding of how personalization might help practitioners translate specific clinical questions into ESM templates that fit clients' individual needs.
In a similar vein, the clinical usability of the data visualizations will depend on prespecified clinical objectives.For example, when ESM is developed with a clear purpose for a specific population, higher clinical utility and acceptability might be reached (Bell et al., 2020;Kiekens et al., 2023).
In contrast to recent findings (Bell et al., 2020), where generally high compliance among voice-hearing patients was found (100% above the 33% cut-off), the level of compliance in our study was substantially lower (69% above the 33% cut-off).The lower compliance in our study might be tied to the fact that the ESM setup might not have always resulted from a collaborative process between clients and practitioners (i.e., limited use of personalization features).This may help explain why the ESM protocol did not always meet client expectations and was sometimes perceived as burdensome.This corroborates earlier work and stresses the importance of actively involving clients in goal-setting and the setup of ESM templates (Daniels et al., 2022), which will be necessary to increase patient engagement and empower clients to take an active role in their recovery process.
Taken together, the findings of this first ESM pilot study in specialized mental health care suggest that using a generic ESM template may be less practical as the collaborative clinical goal will determine the ESM content, schedule, and visualizations in practice.
Our findings have several implications for future research and implementation work.First, to guide further software development, more work is needed to determine how clinical goals translate into specific ESM questionnaires and sampling schemes.For example, during the initial stages of treatment, it may be more beneficial to have a detailed summary of daily life experiences to identify patterns or contextual determinants of a client's mental health problem (i.e., the hypothesesgenerating phase of diagnostic assessment).In such cases, intensive sampling for a week may be required.In contrast, there may be less need for such a dense sampling schedule when a client has been in therapy for an extended period, and the goal is to evaluate treatment and/or prevent relapse (i.e., hypotheses-confirming evaluation and prognosis).This is also in line with recent work (Bos et al., 2019, Bos et al., 2022), which suggested that other formats of ESM may be required for clinical use.Second, we identified a need for additional items to personalize ESM content to the needs of individual clients.While future work could resort to items used in academic research (Kirtley et al., 2020), exploring co-designing items with practitioners and clients that match the experiences they want to capture outside the therapy room may also be worthwhile.Third, although compliance is one indicator of burden (Eisele et al., 2022), future research is needed to investigate under what circumstances ESM is perceived as less burdensome.Fourth, we identified the need for other training compared to our one-hour online training sessions.For instance, as suggested by the practitioners in our study and elsewhere (Piot et al., 2023), including group training with mock clients might help increase usability.However, addressing the abovementioned issues will be a prerequisite in developing practical and concrete training programs that match clinical complexity.Such training programs could already be introduced into higher education programs that take a dimensional and recovery-based perspective on mental healthcare.

Limitations
Several limitations should be considered when interpreting the results of this pilot study.
First, and as mentioned elsewhere (Thirumlai et al., 2018), there are numerous ways to define and measure a mobile health application's usability, and not all of these elements were studied.Hence, other usability elements may still need to be studied (e.g., phone battery constraints).Second, we experienced a challenging recruitment procedure due to the unforeseen and unique circumstances of the COVID-19 pandemic.However, participant numbers (i.e., 12 out of 142 invited) exceed what usability experts consider as sufficient and recommend for pilot testing in an iterative user-centered design (Rubin et al., 2008;Turner et al., 2006).Third, our sample consists of early adopters who may not be representative of the entire population of practitioners and clients.Fourth, we did not implement any strategies to increase user engagement, such as gamification (Sage et al., 2017), which may be beneficial to consider in future work.

Conclusion
This pilot study is the first to design and implement a protocol for using an ESM template in a specialized mental healthcare setting.Our findings suggest that the ESM template and used software are easy to use, indicating that practitioners and clients are capable agents for using ESM in clinical practice.Yet, clients' readiness to use (or keep using) ESM was limited due to limitations in perceived usefulness.Hence, the piloted ESM protocol should not be readily implemented and substantial adaptions are necessary.These may include providing additional sampling scheme formats, personalization through co-developed items, and a dynamic data visualization interface.To optimize the usability of ESM protocols as a mobile health assessment tool in recovery-focused psychiatry, we encourage scientists and implementation experts to focus more on collaborating with practitioners and clients in every phase of the design, evaluation, and implementation process to make meaningful translations of clinical questions into ESM templates that truly benefit and meet the specific needs of individuals.
questionnaire assessing sociodemographic information.Following this, practitioners received a manual on how to use ESM.The manual included creating an account on the ESM dashboard, setting up their account to access the ESM questionnaire content and sampling schedule, enrolling a client, adjusting or creating questionnaires/sampling schemes, and visualizing data paired with interpretation examples.Additionally, practitioners could join a one-hour online training session with a research team member, which covered the same topics as the manual.Once practitioners were familiar with using the ESM dashboard, we requested them to use it with several clients in their clinical practice.The practitioners informed clients about the study, and if a client showed interest, they asked them to read and sign the informed consent form (as the research team was not allowed access to the hospital due to the COVID-19 pandemic).Following informed consent, clients completed enrolment questionnaires that assessed demographic and clinical variables.Afterward, clients installed the m-Path smartphone app, which was used to trigger the ESM questionnaires.The practitioners could personalize the ESM questionnaires for the client using the m-Path dashboard.After one week of ESM, practitioners and clients were requested to discuss visualizations of the client's data during the subsequent clinical session utilizing the dashboard.To assess any operational difficulties or bugs, we phoned practitioners bi-weekly for routine check-ins during which they could report technical problems or difficulties with using the software.At the end of the implementation period, clients and practitioners were provided with questionnaires assessing the usability of IMPROVE software and invited to participate in a semistructured interview.The interview allowed us to capture more rich information about the experiences of practitioners and clients regarding the use of ESM.The pilot study took place between December 2020 and July 2021 and was in accordance with the Ethical Principles of the American Psychological Association (APA, 2017).
Figure 1.Examples of visualizations used in IMPROVE

Table 1 .
Content of the ESM template 1

Table 2 .
Demographic information on clients and practitioners.