Promoting Engagement in Remote Patient Monitoring Using Asynchronous Messaging

2.1 Opportunities in Remote Patient Monitoring

Multiple survey papers have expounded the degree to which remote patient monitoring has improved patient care and enhanced the ability of clinicians to track patients in nontraditional healthcare setting [23, 70, 83]. Nearly 95% of the world’s population has access to mobile networks [79], so remote patient monitoring can support patient care within rural and developing communities where healthcare systems are not easily accessible. Active work is also being done to improve connectivity and network speeds in such communities so that remote health monitoring solutions can be efficiently delivered in these contexts [34, 58].

The health data that underlies remote monitoring platforms can take multiple forms, the most prominent being self-reported data and vital signs. While it has been possible for health data to be self-reported over web portals and phone services for over the past few decades, the prevalence of vital signs data has grown due to the proliferation of wearables like fitness trackers [93]. Such devices are able to passively and continuously capture vital signs data, leading to the generation of objective and fine-grained health data. On top of this, machine learning models can be deployed on the cloud or on the devices themselves to extract advanced metrics relevant to the disease domain. Past work has slowly progressed towards such clinically relevant endpoints [14, 37, 46, 47, 77], but their clinical adoption remains to be seen.

Although remote patient monitoring has been used to support people with medical concerns ranging from weight management [4, 29, 85, 88] and mental disorders [24] to respiratory illness [63, 65, 81] and sleep disorders [25, 56], the onset of the COVID-19 pandemic accelerated the need for scalable patient care [70]. An increasing number of publications are now emerging that describe various efforts to monitor COVID-19 patients in outpatient environments. For example, the GetWell Loop platform by Annis et al. [3] offered patients educational materials about COVID-19 while asking them to fill out daily questionnaires related to their symptoms. Their platform leveraged a virtual workforce of providers and medical students who were tasked with overseeing patient responses and triaging their concerns. The authors focused on quantitative metrics of engagement and therefore did not closely examine the nature of various interactions stakeholders had on their platform.

Another example of a remote patient monitoring system for COVID-19 is demonstrated by Gordon et al. [27]. Their platform required patients to track their symptoms and vital signs using daily questionnaires and medical devices. The authors claimed that enrollment in their system decreased the likelihood of hospital readmission, but they also did not comment on communication between patients and clinicians. While the findings of our work are grounded in a platform similar to the one developed by Gordon et al. [27], our work extends upon this literature by investigating the role that asynchronous messaging can serve in remote patient monitoring platforms.

2.2 Challenges in Remote Patient Monitoring

Despite the benefits that remote patient monitoring has to offer, it also presents numerous challenges for both clinicians and patients. Many clinical workflows already involve a number of complex software-based platforms, most notably EHR systems. Introducing yet another platform to such workflows can incur significant burden and inefficiencies, leading them to be abandoned [12]. Conversely, patient attrition can be high when the burden of generating data outweighs the benefits they experience from the platform [50, 51]. Additionally, high patient engagement can result in positive patient outcomes [57]. Ensuring that this trade-off is beneficial for both stakeholders is important to the deployment of any remote monitoring platform.

Another set of challenges revolves around data quality, particularly with respect to vital signs data. Improper device usage, sensor failure, and irregular data entry can lead to data quality concerns [75]. In the case of body-worn sensors, for instance, Baig et al. [6] note that body movement, electromagnetic interference, and sensor drift can all lead to erroneous measurements. Vital signs data can lack context as well [2]. For example, an elevated heart rate can be attributed not only to a health problem but also to physical activity or caffeine consumption. This overall disconnect between what information clinicians see and what patients are experiencing creates uncertainty [62], forcing clinicians to expend additional effort validating the data they receive before interpreting it [5]. Researchers must also ask similar questions about the data they receive from these platforms if they decide to pursue their own offline analyses or data-driven models.

A third set of challenges entails reduced face-to-face interaction between patients and clinicians [49]. Many platforms have automated processes in place to flag abnormalities in patient data, but clinicians are still tasked with assessing patient data for themselves [1, 48]. Patients may not always be consciously aware of these processes, especially when communication is limited to when there are noteworthy health concerns. This can lead to patients feeling unheard or uncertain about their health status [84].

This work aims to demonstrate that an intentionally designed messaging feature built into a remote patient monitoring platform can assuage all three of the aforementioned challenges. After showing that the messaging feature does not impose significant burden on any of the platform’s stakeholders according to platform usage statistics, we present the findings from a thematic analysis to show that our messaging feature engendered increased engagement and empathy. We then show that the dataset modifications logged through the messaging feature improved data quality in a way that translated to a performance boost in a machine learning model.

2.3 Digital Communication Between Patients and Clinicians

In recent years, advancements in digital communication technologies have paved the way for various digital means of facilitating communication between patients and clinicians outside of clinical settings. Two of the most popular telehealth communication channels are phone and video calls [8, 92]. These modalities support many of the same verbal cues as face-to-face conversations, such as tone and intonation. Video calls also support non-verbal cues that are available during face-to-face conversations, including body language and facial expressions [36, 39, 71]. Clinicians rely on these cues to convey empathy, build rapport, and make additional judgments about their patients; meanwhile, patients rely on these cues to better convey their concerns and enhance their understanding of clinicians’ determinations [36, 71]. Although phone and video calls circumvent the need for patients and clinicians to be co-located, they still require both conversational partners to coordinate and be available at the same time. In the context of remote patient monitoring, asynchronous communication eliminates the need for immediate responses, enabling healthcare providers to review and assess patients’ data at their own pace.

Text-based communication lacks the aforementioned verbal and non-verbal cues, but it pervades many digital health systems because of its convenience and low technology requirements [32]. Even though there are synchronous text-based platforms like live messaging services and mobile chat applications [28], the majority of patient-clinician conversations occur over asynchronous platforms such as email, online forums, and web portals [9, 20]. This includes the messaging features included in many prominent EHR systems, including the previously cited MyChart by Epic. Even though there has been significant discussion about the design and usage of syncronous and asynchronous messaging features in the context of patient-clinician interactions [42, 45, 67, 73], these investigations have stopped short of examining the conversations that are had with respect to remote patient monitoring.

3.1 Patient Enrollment

The COVIDFree@Home platform was designed to support patients who tested positive for COVID-19 but were feeling well enough to not require hospitalization. Given the onus on hospitals to prioritize healthcare delivery to severe cases, these patients were originally being sent home and instructed to seek medical attention if their health deteriorated. Doing so left patients’ recovery unmonitored, leaving them on their own to make decisions about their health. This also meant that useful data during cases of deterioration was not being collected to further researchers’ understanding of virus symptomatology.

To remedy these shortcomings, individuals who had recently tested positive for COVID-19 were contacted by a research coordinator to participate in the COVIDFree@Home study. On average, patients were recruited 4–5 days after their positive test result. This lag was suboptimal for both patient care and our study but was largely a consequence of the propagation of positive test results from testing facilities to the research team. Although it was possible to miss relevant health events during this time, patients were generally healthy and exhibited few symptoms at the time of recruitment.

Patients who consented to participate in the study were informed about the expectations of the study and sent instructions on installing the COVIDFree@Home app. Patients were also sent a thermometer for measuring core body temperature and a pulse oximeter for measuring oxygen saturation and heart rate. Due to supply chain challenges during the pandemic, the oximeters were sourced from multiple companies and the thermometers were phased from multi-use to single-use models. The medical devices usually arrived after patients installed the mobile app, but patients were encouraged to upload any data they could provide when possible.

3.2 Design Process

Our team was composed of partners with complementary roles: engineers who built the platform, designers who helped optimize the user experience, clinicians who provided insights into patient care, and people who had lived experience with COVID-19 to shed light on patient needs. We utilized an iterative design process with collaborative input from all team members over the course of 6 months to produce the first working version of COVIDFree@Home; new features were added beyond that point, but they are beyond the scope of this work as they do not pertain to the platform’s messaging system or the kinds of data that were collected.

The design process began with an initial set of meetings to define the platform’s goals. Within the scope of this paper, key objectives included providing patients with peace of mind while they recovered and keeping doctors from being overburdened by the system. After creating wireframes of interfaces that would be available to patients and physicians, we underwent iterations of prototyping, application demonstrations, and testing to refine our designs. Because the pandemic precluded in-person interaction, our entire design process was coordinated through virtual meetings. It was through this process that we arrived at the final design presented in the rest of this section.

3.3 System and UI Design

The COVIDFree@Home platform was designed as a patient care tool first and as a research tool second. In other words, platform adjustments and research components were kept limited in order to maximize the care patients received. For the purposes of this work, we describe the mobile app that patients used to report their health data and the interfaces that both patients and clinicians utilized for exchanging messages below.

3.3.2 Messaging Feature.

Figure 1:

View Figure

Figure 2:

View Figure

Clinicians reviewed patient data through the dashboard on roughly a daily basis, prioritizing patients with worsening symptoms or anomalous data. Patients received an automatically generated message from COVIDFree@Home whenever their file had been reviewed. Beyond receiving automated messages, patients were encouraged to use the messaging feature within their app to contact their assigned clinician(s) if they had any questions or concerns about their health. They were also encouraged to use the messaging feature for any questions they had regarding the study protocol and logistics. Patients were given the explicit caveat that using the COVIDFree@Home app was not a replacement for emergency services, and this warning was reiterated throughout the app as well. Clinicians were able to view any messages sent by their patients through a dashboard created specifically for COVIDFree@Home.

The messaging interfaces available to patients are shown in Figure 1, while the corresponding interfaces available to clinicians are shown in Figure 2. Most existing platforms for remote patient monitoring included a messaging feature similar to popular chat interfaces like WhatsApp or standard SMS [35, 86]. However, the clinicians on our team noted that this style of messaging might create an expectation for multiple rounds of synchronous discussion, which would impose added burden on those who would be remotely monitoring patients [15]. To avoid this behavior, we designed the messaging feature within the patient-facing app so that it had separate interfaces for sending and receiving messages. The goal of this design was to encourage patients to send self-contained messages, thereby minimizing the number of back-and-forth exchanges. This intention was also implied by the size of the text box on the interface for sending messages, suggesting that patients should be able to fit everything they needed to say in that area. Meanwhile, the clinician-facing dashboard enabled clinicians to examine patient messages in tandem with relevant demographic and health-related information. Doing so provided important background information about the patient alongside the messages as clinicians reviewed patient files.

4.1 Study Overview

4.2 Engagement with the Messaging Feature

Table 2 describes the number of messages that were sent through the COVIDFree@Home platform. Roughly 3.3k automated messages were delivered to patients as a result of a clinician reviewing their profile. Excluding those messages, 324 messages were exchanged on the platform, with 102 (31.5%) being sent by patients and 222 (68.5%) being sent by clinicians. In the context of the study duration, the low burden of these messages on clinicians is highlighted. On average, clinicians received 0.24 messages and sent 0.70 messages per day.

Figure 3:

View Figure

As shown on the left side of Figure 3, patients most often sent messages during mornings and evenings with a dip in the afternoon. Since patients were sent notifications if they had not uploaded data by 10 AM and 8 PM, this trend reveals patients often decided to read and send messages around these times as well. Clinicians demonstrated similar patterns, although their messages were often earlier than the automatic notifications to maximize the chance that they would be viewed by patients. The times at which clinicians sent messages were also more confined to their work hours. The right side of Figure 3 shows the time it took for clinicians and patients to reply to one another’s messages. To generate this distribution, the message logs were first divided into a series of contiguous conversations according to the topic of the message content. Any time the message sender switched between messages within a given conversation, the elapsed time between them was calculated and added to the graph. The average time it took for all platform users to reply was 7 hours; patients took 4 hours to reply on average, whereas clinicians took 9 hours.

We neither draw explicit comparisons between these averages nor report the response rates of the two groups to avoid mischaracterizing their responsiveness. Patients were told during enrollment that they would receive replies to their messages within 24 hours, and Figure 3 shows that this goal was achieved. Nevertheless, clinicians often leveraged this flexibility to balance their COVIDFree@Home duties with other priorities. Patients also occasionally sent messages at night, so although receiving a response the following morning would be reasonable, these cases produced longer response times. Regarding the response rates themselves, there were many messages that did not require a response, and this determination was not as simple as annotating whether or not the message contained a question. There were also situations when the conversational partner responded to a message in a way that was not captured by the messaging feature. For example, when clinicians requested additional data from patients, the latter would upload data without confirming that they did so in the messaging feature. Conversations were sometimes continued over the phone or a videoconferencing platform, so the final resolution of the topic was never captured on the messaging feature.

Despite these caveats, our observations show that the messaging feature was leveraged by both patients and clinicians in a way that did not impose substantial overhead. Both stakeholder groups often sent messages near the times at which the automated notifications were initiated, so this design decision may have influenced asynchronous messaging behavior.

5.1 Categorization of Message Topics

The content of the messages exchanged over the COVIDFree@Home platform was annotated using thematic analysis, more specifically open coding [80]. Using an inductive approach, two researchers independently generated codebooks on a subset of messages, after which they convened to agree on a shared codebook. The researchers then independently coded all messages according to that shared codebook. Although multiple codes could have been applied to the same message, the researchers were instructed to select a unique code that they felt was most relevant to each message in order to facilitate the calculation of an inter-rater agreement score. The tool used for this analysis was a spreadsheet that was configured to facilitate the coding process. This process led to an inter-rater agreement of 81.5% for patient messages and 83.1% for clinician messages according to Krippendorff’s alpha. After the first round of coding, the researchers deliberated over conflicting codes and reached 100% agreement in order to generate a final label for each message. A breakdown of the types of messages that were sent is shown in Figure 4, and the distribution of the message topics is shown in Figure 5. Example messages for each topic can be found in Appendix A.

5.2 Conversation Patterns

Figure 6:

View Figure

Figure 6 illustrates common sequences across the various message categories. These figures were generated by identifying pairs of messages that were deemed to be connected by the same researchers who coded the messages themselves. It is worth noting that not all messages from Figure 5 are represented in Figure 6, as messages that did not receive a response were excluded. The most common exchanges between patients and clinicians often dealt with either the study protocol or patients’ concerns about their health. As expected, most Study Questions and Health Questions sent by patients were responded to by clinicians with messages about Study Logistics and Health Information and Recommendations, respectively. Many messages extended beyond information exchange, including elements of gratitude, empathy, and rapport building. Clinicians often received Expressions of Gratitude when they provided patients with information about Study Logistics, and patients received Expressions of Gratitude themselves when they provided clinicians with updates on their status.

Messages from both stakeholders eventually led to new information that allowed patients to receive more personalized and timely care. When clinicians sent Requests for More Data and Wellness Checks, responsive patients either replied with a Prompted Health Update or simply uploaded new data directly. Patients even provided health data outside of the primary monitoring mechanisms through Unprompted Health Updates. These exchanges also helped generate a more reliable and complete dataset of patient health that can be used by researchers to model symptom progression.

5.3 Case Studies

To further illustrate the benefit of the messaging feature to clinicians and patients, we provide examples of when conversations through COVIDFree@Home directly led to data corrections.

6.1 Dataset Modifications

The dataset initially only had seven labeled cases of patient hospitalizations, but thanks to follow-up by the clinicians, nine cases that had not been logged as hospitalizations were eventually updated in the system. The clinicians also played a pivotal role in correcting symptom and vital signs data, adding 61 symptom records using a combination of messages through the platform and clinical notes left after virtual consultations. The clinicians also identified nine vital signs measurements that were later determined to be incorrect due to data entry mistakes, malfunctioning devices, or incorrect use of the app. In total, 91 modifications were made to patient-reported data, and a full description of modifications is shown in Table 3.

6.2 Model Design

Figure 7:

View Figure

Our dataset included 16 hospitalization cases out of 356 patients (4.5%), comparable to hospitalization rates observed in other studies involving low-risk COVID-19 patients [3]. Nevertheless, this rate led to an exceedingly imbalanced dataset for traditional machine learning. We instead used few-shot learning [87] to leverage the few positive labels we had to train a model. We reframed our prediction task such that the goal was to identify whether two patients belonged to the same class — in this case, those in need or not in need of hospitalization. During testing, individual patients were compared against known cases of hospitalization or non-hospitalization set aside as a support set from the training data; test patients were assigned to the class corresponding to the most similar patient from the support set. Transforming the positive labels from patient hospitalization status to whether or not two patients belong to the same class increased the number of positives available during model training. Using few-shot learning also multiplied the yield of each hospitalization case since the model used pairs of patients as input.

We used the following dimensions of data as input into the model: age, gender, oxygen saturation measurements, heart rate measurements, temperature measurements, and the presence of the seven main symptoms recorded by the app, totaling 12 features to be used for classification. Demographic information was included in our model since age and gender are known to be covariates of heart rate [78] and temperature [26]. To address the fact that patients enrolled in the study at different times after self-reported onset of COVID-19, we trained the model on data from the patient’s first seven days relative to onset rather than enrollment. For example, if a patient was enrolled on their third day after testing positive for COVID-19, we only used data from their first five days in the study — their third, fourth, fifth, sixth, and seventh days after COVID-19 onset. Each day was split at 2 PM to create two half-days. Multiple data uploads within a half-day were merged such that only the highest heart rate, the highest temperature, and the lowest oxygen saturation readings were kept. Meanwhile, multiple symptom reports within a half-day were merged such that a symptom was recorded as present if it was marked as such in any of the reports.

Figure 7 illustrates our model architecture, which was a Siamese neural network consisting of fully connected, batch normalization, and dropout layers [40]. The model was trained to predict the probability that two input patients would have the same outcome. The input vector for each patient included (12 features) × (14 half-days) = 168 features in total. Instances of missing days and reports were imputed using mean imputation within each feature.

6.3 Model Evaluation

We used 70-30% train-test splits in our experiments, meaning that 250 patients were used for training and 106 patients were used for testing. The ordering of the input patient pairs mattered since the weights of the model sub-networks were independent; in other words, the input pair (patient₁, patient₂) could have led to a different result than the input pair (patient₂, patient₁). Therefore, we generated 250 × 249 = 62.3k patient pairs for model training. The support set contained 3 hospitalized patients and 3 non-hospitalized patients who were held out from the training set. We repeated this splitting procedure 100 times with random splits to establish the robustness of our results.

We compared the model performance according to two datasets: one with the original features in the dataset, and another taking into account the feature additions and corrections obtained through the messaging feature. Although dataset corrections also increased the number of hospitalization cases available in the dataset from 7 to 16, both experiments used the final set of 16 labeled hospitalization cases to prevent the potential improvement from being attributable to the redistribution of labels. The dataset modifications impacted 28 (8.0%) patients in the training and test sets, and 2 (33%) patients in the support set. Additionally, 4 (25%) hospitalized patients had their data modified.

Figure 8:

View Figure

The model performance with the two datasets is shown in Figure 8. Although the revisions impacted fewer than 100 entries in our dataset of over 10k reports, we saw that they had a noticeable effect on the models. The model that was evaluated on the dataset without modifications achieved an average AUROC score of 0.53 ± 0.18, while the model evaluated on the other dataset achieved an average AUROC score of 0.59 ± 0.20. The difference between these distributions was statistically significant according to a paired-samples t-test (t(99) = 2.18, p <.05).

7.1 Benefits for Patients

Although remote monitoring provides an additional channel for patient care, the lack of interpersonal communication and face-to-face interactions can result in patients feeling uncertain or unheard about their health concerns [13]. This was particularly true during the COVID-19 pandemic, when there was significant uncertainty around the symptoms and outcomes associated with the virus. Including a messaging feature in the COVIDFree@Home platform mitigated some of these worries for patients, conveying that there was a team of clinicians that was closely reviewing their data. Patients received a notification whenever their profile was reviewed by a clinician, and they also received messages from clinicians asking about their health. Similar feedback mechanisms in prior work have been shown to improve adherence in remote patient monitoring [7]. We also saw patients express gratitude to their clinicians for reviewing their data, further supporting the notion that they appreciated the care they felt they were receiving.

Beyond increased engagement, we found that the messaging feature led to collaborative reflection [53]. Some patients who struggled to make sense of their health data utilized our messaging feature to leverage clinicians’ domain expertise and seek advice. The fact that patients were taking the time to contemplate the progression of their health had its own benefits, as prior work has shown that such reflection can lead patients to make more informed decisions about their own health [33, 44, 54]. Collaborative reflection through the messaging feature also made it easier for clinicians to investigate potentially anomalous data and ensure that they were not being misled into incorrect patient recommendations. As highlighted in Section 5.3, conversations over patients’ data occasionally led to the identification of faulty equipment or incorrect data reporting. Without asynchronous messaging, diligent clinicians would be able to identify and remedy such issues over the phone or a videoconferencing session. Still, the messaging feature made this process more convenient while automatically documenting the conversation for all stakeholders.

7.2 Benefits for Clinicians

One of the major benefits experienced by clinicians was the fact that the messaging feature fostered consistent patient engagement with respect to reporting symptoms and uploading vital signs data. The COVIDFree@Home platform included a few automated messages for this purpose: the reminders that were sent whenever patients did not upload data by the prescribed windows and the notifications that were sent whenever a clinician was actively reviewing the patient’s data. We initially considered providing patients with more information about which data was being reviewed or why their data was being reviewed; however, concerns were raised as we were developing COVIDFree@Home that doing so would often lead to additional patient concerns. Even though these automated messages were primarily designed to give patients a sense of comfort, we hypothesize that such notifications also added a level of accountability for them to upload data. On top of that, the messaging feature gave clinicians a channel through which they could communicate with patients who were lagging behind in their protocol adherence. In some cases, it was discovered that patients were feeling too sick to upload data; conversely, there were situations when patients felt that they had recovered and no longer saw a reason to upload new data. These explanations led to important changes in care delivery that otherwise would not have been considered.

Although not an actual benefit of the messaging feature, we closely monitored the number of messages that were exchanged through the COVIDFree@Home platform to ensure that clinicians were not overwhelmed with patient communication. This was important to us since inefficient clinical workflows are quickly abandoned for simpler or more effective solutions [61]. With asynchronous text messaging, patients were inherently rate-limited in how they communicated with their clinicians. We showed in Section 4.2 that clinicians sent and received fewer than one message per day on average, suggesting minimal overhead on their workflow.

7.3 Benefits for Digital Health Researchers

The messaging feature in the COVIDFree@Home platform imbued the researchers involved in the study with additional confidence in the data that was being collected. As part of their responsibilities for using COVIDFree@Home, clinicians reviewed new uploads by patients within a day. They were also able to follow up with patients whenever data seemed anomalous, which led to either confirmation of interesting cases or revisions of incorrect entries. In other words, clinicians’ interactions with the COVIDFree@Home platform generated more reliable labels for key events in patients’ trajectories and important context to the observed data. We demonstrated through the evaluation in Section 6 that these modifications led to non-trivial improvements in the performance of a machine learning model for predicting hospitalizations.

The fact that patients experienced added benefit from the messaging feature may have also led to greater adherence to the system, resulting in a more complete dataset with fewer missing values. The COVIDFree@Home study allowed us to curate a dataset from over 350 patients describing their road to recovery or hospitalization. Since our analyses span 18 months, our dataset encapsulates patient experiences with different COVID-19 variations and vaccination levels. This breadth and depth of data will further the research community’s understanding of COVID-19 and perhaps other influenza-like illnesses.

7.4 Design Implications for Front-End Designers

Beyond the stakeholders noted throughout our paper, our work has notable design implications for front-end designers involved in remote patient monitoring efforts or studies involving passive data collection. These kinds of studies are useful for recruiting beyond homogeneous and easily accessible populations like university students [16, 19], yet many of them experience high attrition in the absence of strong financial, social, or individual incentives for participants [41, 60]. Whereas many studies only consider the flow of data from participants to researchers, we hope that our COVIDFree@Home platform demonstrates a different model that provides other forms of incentive for all stakeholders.

One user-centered challenge to adopting a remote monitoring-style platform has to do with the complexity of the data being collected. In our case, we only incorporated a handful of simple vital signs measurements into the COVIDFree@Home platform: temperature, heart rate, and oxygen saturation. The research community has explored a host of potential biomarkers for detecting cases of COVID-19, such as cough frequency [43], cough quality [17], and speech quality [64]. Although these biomarkers have shown great promise, they are still in active development and are known to be prone to bias across different environments, devices, and languages [18]. By focusing on data that could be readily interpreted, clinicians were immediately able to have conversations with patients about their health. Patients were also able to apply their knowledge and intuition to their own health data, which can have varied outcomes depending on how they consider measurement uncertainty and extrapolate trends [38].

When designing interfaces for studies intended to develop or validate new biomarkers, front-end designers should consider whether all relevant stakeholders are able to to derive benefit from at least a subset of the data being collected. In some cases, that may mean vouching for the inclusion of validated vital signs that can instantly be appreciated like heart rate, body temperature, activity levels, or even sleep and stress scores. These validated vital signs are likely to be collected anyway as potential confounds or correlates, but giving study participants access to this data can make them actively engaged in the data collection process.

Our messaging feature also proved invaluable in sustaining engagement, and the way it was designed helped us address the various needs of our stakeholders. Had we focused strictly on patients’ priorities, we would have created the messaging feature in such a way that would have likely overwhelmed clinicians. While clinicians were eager to better serve their patients and to address their health concerns in a timely manner, they were keen on doing this in a scalable manner that did not put unreasonable expectations on their responsiveness [90]. With these considerations in mind, both patients and clinicians benefited from the design of our asynchronous messaging feature. The flexibility it afforded was particularly useful for smaller inquiries that were impractical for scheduling formal meetings yet too important to overlook during routine check-ins. For example, clinicians often utilized asynchronous messaging to send Requests for More Data after reviewing patient data on the dashboard. Conversely, participants frequently used asynchronous messaging to send both Prompted and Unprompted Health Updates. The asynchronous nature of our platform served as an outlet for these "spur-of-the-moment" inquiries that might have otherwise been delayed or forgotten.

As we were designing our messaging feature, another important consideration was how it was advertised to patients. The messaging interface on the COVIDFree@Home app was designed to subtly influence patients to send a single comprehensive message rather than a long chain of shorter messages, similar to the affordances of email versus text messaging [21]. This was achieved by separating the functionality for sending and receiving messages and including a large text box for message entry rather than the standard single-line field used in most chat interfaces. To further curb overreliance on the messaging feature, patients were informed at multiple stages of the protocol that COVIDFree@Home was not a replacement for emergency services. Establishing these expectations in concert with clinicians, hospitals, and other governing bodies was critical to reducing hospital liability and mitigating the chance that patients would not seek appropriate treatment.

7.5 Limitations

Our work was not without its limitations and shortcomings. First and foremost, our dataset contained a limited number of patients who required hospitalization after enrollment (16/359 = 4.5%). Since the COVIDFree@Home study was designed to support patients who were suitable for outpatient care, we did not recruit those who initially presented with severe cases of infection. It was therefore expected that many of these individuals would not require care escalation, which may have introduced a bias into our dataset that limits its generalizability to other healthcare scenarios. However, the focus of our machine learning experiment was not to demonstrate the performance of a deployable model, but rather the improvement we observed by applying data corrections obtained through the messaging logs. Furthermore, insights from the messaging content shine a light on the experiences of a patient cohort that still incurs significant financial burdens on healthcare systems [11].

We were also only able to report on data that appeared within the COVIDFree@Home platform. Patients were free to measure their vital signs or track their symptoms without entering the data into our smartphone app. Patients and clinicians were also allowed to communicate with one another using other communication channels. We were informed of a few such cases during our study, such as instances when clinicians sent a message to notify patients of an impending conversation over the phone.

Finally, the COVIDFree@Home study was conducted in a major metropolitan city in North America known for its multiculturalism, yet our findings are limited in that they were only derived from a single population. Patient-clinician relationships and communication practices in general are tied to cultural and socioeconomic norms [55, 68, 69]. Furthermore, differences in how healthcare is structured across countries can impact the expectations of both patients and clinicians. Future work could investigate whether the proposed benefits of asynchronous messaging generalize to other population groups.

A Examples of Messages Sent Through COVIDFree@Home