The Perceived Ease of Use and Usefulness of Loop: Evaluation and Content Analysis of a Web-Based Clinical Collaboration System

Background: Patients with complex health care needs require the expertise of many health

We employed a user-centered design (UCD) [43,44] approach to engage stakeholders (ie, clinicians, researchers, designers, developers, and end users) as early engagement is believed to promote greater uptake at implementation [26,45]. In particular, we used ethnography [46], affinity diagramming [47], cooperative prototyping, and dramatic simulation [48] to involve stakeholders in the conceptualization and generation of system requirements for Loop. Through this process, we defined the following two usability objectives to guide the development of Loop: (1) Loop will be intuitive and easy to use, requiring minimal or no instruction, and (2) Loop will be useful in the care of patients with complex care needs. We continued to engage stakeholders during the development and refinement stages via usability testing, prototyping, and pilot testing. Usability testing involves a representative sample of intended end users interacting with a system to generate insights that will optimize the design and user experience [49][50][51]. The technology acceptance model (TAM) is commonly used to evaluate new technology systems [52,53]. TAM aims to predict a user's acceptance and adoption of an information technology system based on two constructs: perceived usefulness (PU) and perceived ease of use (PEOU) [52]. PU evaluates how a tool might enhance job performance, effectiveness, and productivity [52,54,55]. PEOU assesses the perceived effort required to learn and interact with a tool [52,55]. The more useful and easier to use a tool is perceived to be, the more likely it will be accepted [52]. Original applications of TAM employed quantitative metrics; however, qualitative interviews have also confirmed that PEOU and PU are main factors affecting intention to use and explored what is meant by these complex terms [56]. Using TAM as a lens for the qualitative analysis of usability testing experiences, we aimed to understand how end users perceive Loop's ease of use and usefulness.

Study Design
The data collected during the iterative cycles of qualitative usability testing were used to evaluate and improve the Loop prototype. Once all data were collected and the prototype completed, a descriptive content analysis of a subset of usability testing data was performed to determine PU and PEOU.

Clinical Collaboration System
The core functionality of Loop includes (1) composing and posting messages that are visible to team members who are part of that Patient Loop and (2) viewing messages on a central Message Stream (Figure 2) . Loop also includes the following auxiliary features: (1) tagging messages with specific labels, which can then be used for filtering messages using the Issues feature ( Figure 3); (2) tagging specific team members so that they will receive an email notification about the new message using the Attention To feature (These message are visible to the entire team on the Message Stream; Figure 4); and (3) selecting whether a message will be visible to the whole Loop (patient, caregiver(s), and HCPs) or only between HCPs on the team using the Team Only feature ( Figure 5). The need for these features was identified in the earliest stages of conceptualization. As such, some version of the Issues, Attention To, and Team Only features appeared in all Loop prototypes. Our earlier publication describes usability testing and participant perceptions of the Team Only feature. We found that all participant types endorsed the inclusion of a separate view for HCP-only communication [42].

Sample
A total of 89 participants completed usability testing: 23 patients, 19 caregivers, and 47 HCPs. Two patients and 2 caregivers completed usability testing together. Participants were recruited from the following three populations: adolescents and young adults with cancer (AYAC) [57], adults with cancer, and children with medical complexity (CMC) [3]. These represent populations across the life span with multiple comorbidities that have high service needs and require care from multiple providers across health care settings. AYAC participants were patients aged between 15 and 25 years and receiving oncological care (n=15). Adult cancer participants were patients (n=8) and caregivers of patients (n=12) receiving oncological care. CMC participants were caregivers of children with severe functional limitations (n=7). Convenience samples of eligible participants were identified by HCPs at three academic institutions specializing in cancer, palliative, and pediatric care located in Toronto, Canada (Princess Margaret Cancer Centre, The Temmy Latner Centre for Palliative Care, and The Hospital for Sick Children, respectively). Convenience sampling is an accepted form of sampling in usability testing [50]. HCPs from various disciplines who work with oncology and CMC patients were purposefully recruited from these institutions and other community-based care organizations to maximize representation across population, provider type, and clinical environment (AYAC: n=16; adult cancer: n=19; and CMC: n=12). The study protocol was approved by the research ethics boards at each site, and informed consent was obtained from all participants.

Usability Testing Protocol
Usability testing was completed over three rounds of iterative laboratory-based testing, which included a desktop computer in a quiet room equipped with microphones, video cameras, and one-way mirrors. This was supplemented with concurrent off-site testing, which was conducted with a separate sample of participants at a convenient location, such as their office, home, or meeting rooms. Each round of testing evaluated a prototype of increasing fidelity: low (iPad and paper prototype), medium (wireframe), or high (working prototype). The high-fidelity prototype was tested and refined over a series of rapid development sprints. Participants completed questionnaires collecting demographic characteristics and technology use before testing.
Participants were presented with a set of standardized task-oriented scenarios targeting new or refined features, processes, or design elements. Table 1 lists the categories of tasks completed by the different participant types. Not all features were tested with every participant type; however, participants may have interacted with features during exploration or while completing another task. Scenarios were worded to allow for a natural flow of interactions rather than as step-by-step instructions (eg, "Your pain is now being well controlled by your medications. How would you alert the team?"). Using this approach, facilitators observed the functions participants used and how they used them as well as common navigation errors and inefficiencies. Participants were asked to think aloud and verbalize their choice of actions while interacting with the system. To supplement the think aloud component, facilitators asked questions throughout the testing session to capture users' reactions and reflections (eg, "Do you have any general thoughts about the layout of the message area? How did you find replying to a message?"). All usability testing sessions were audio-recorded and transcribed. Screen captures of testing sessions were recorded but were not used for this analysis.

Data Analysis
Demographic data were analyzed using Microsoft Excel (Microsoft Corporation, Redmond, Washington) to determine categorical frequencies. Medians were calculated for participant ratings of comfort using various technologies.
Content analysis was used to analyze the usability testing interview transcripts and generate a descriptive summary of users' PEOU and PU of Loop [58]. Three reviewers (BL, AK, and MVW) independently open coded transcripts in NVivo version 10 (QSR International, Burlington, MA). Codes were hierarchically organized and served as the coding framework, which was continually adapted and reviewed with two senior team members (AH and JS) at key points in the analysis process. This review and coding process familiarized reviewers with the data and informed focused in-depth analysis. Open codes were then categorized as PEOU or PU. Themes within these categories were identified using questions derived from the key constructs of TAM [59]: PEOU: "What elements were easy ortable difficult to use?" and "Why were they perceived to be easy or difficult?" Participant responses were analyzed to identify comments that explicitly expressed positive or negative sentiments about the ease of navigating Loop, or sentiments of confusion, frustration, or satisfaction from which ease or difficulty could be inferred.
PU: "What would people use Loop for?", "Why would that be useful?", and "How could that improve care?" During the analysis of these responses, additional questions of "What are the factors that impact Loop's usefulness?" and "What strategies could mitigate the barriers?" were also explored.
Usability testing yielded 87 transcripts. We used a maximum-variation sampling [60] approach to maximize the representation of perspectives and experiences across the following categories: population type (adult cancer, AYAC, and CMC), laboratory-based or off-site testing, user type (caregiver, patient, and HCP), type of HCP, and version of Loop tested. On the basis of the analysis of this subsample of transcripts, all themes became saturated and no new concepts emerged from the data, prompting the decision not to code any further transcripts. A total of 48% (42/87) of transcripts were included in analysis.

Participant Characteristics
All adult cancer patients were older than 50 years, and all AYAC patients were younger than 30 years, representing the oldest and youngest participants tested (Table 2). Caregivers and HCPs were concentrated around the central age ranges; 95% (18/19) of all caregivers and 96% (44/46) all HCPs were aged between 30 and 69 years. A majority of participants had access to computers and Internet at home (Table 3). Adult cancer patients were the least comfortable of all participants using the surveyed technologies, and AYAC patients were the most comfortable ( Figure 6). Of all technologies, participants in all populations except AYAC were least comfortable using social media.

Qualitative Findings
During usability testing sessions, participants provided feedback about the PEOU and PU of the system. Themes were identified within each of these categories ( Figure 7). Broadly, analysis revealed two types of problems that negatively affected the PEOU of Loop: visual design problems and incorrect use of features. With regard to PU, participants felt that Loop could improve the efficiency and effectiveness of communication in real-world use. However, PU could be negatively affected if Loop disrupted individual workflow. Participants suggested features to mitigate Loop's potentially disruptive impact and improve ease of use. These findings are described in detail below. Responses were consistent across user type (patient, caregiver, and HCP) and populations (CMC, AYAC, and adult cancer) unless otherwise noted.

Perceived Ease of Use
The majority of participants enjoyed testing Loop. They felt that the layout and design were easy to navigate and that the core functions (composing, posting, and viewing messages) were intuitive to use. One participant stated the following:

Factors Negatively Impacting Loop's Perceived Ease of Use
Negative feedback about ease of use was mostly related to the visual design and use of auxiliary features such as Attention To, Issues, and specific visual cues in the Message Stream. We observed that participants were less likely to volunteer comments if they did not encounter a problem when navigating the system.

Visual Design Problems
Some participants did not perceive icons as clickable, did not notice visual cues, or had difficulty interpreting the meaning of icons or visual cues. These errors may highlight problems with certain visual design elements. For example, some participants did not identify the clickable icons that would allow them to complete tasks related to replying to messages, viewing conversations, and editing Issue statuses. In other situations, participants were unable to identify what the visual cues were trying to convey or did not perceive these cues at all. In particular, the blue ring and icon on profile pictures indicating messages visible only to HCPs and the yellow background identifying messages directed at the user were often overlooked or misinterpreted. Problems interpreting visual cues did not impair participants' ability to read messages in the Message Stream, but useful contextual information related to each message may not have been understood. One participant stated the following:  14 (87)  10 (91)  13 (68)  6 (86)  7 (58)  5 (33)  5 (62) Female     Participants suggested adding cursor labels to describe the purpose or indicate clickability of icons and visual cues. Cursor labels (text that appears when a user hovers the cursor over an icon) were introduced in the high-fidelity prototype of Loop and observed to improve navigability when tested.

Age in years
For most visual design problems, participants indicated that subsequent use of Loop would be easier after they received orientation and instruction about Loop's icons and visual cues.

Perceived Usefulness
Participants indicated that Loop's PU lies in its potential to improve efficiency and effectiveness of communication about patient care within a team. Participants also highlighted that PU could be negatively affected if individual workflow is disrupted.

Improving Efficiency
Participants across stakeholder groups felt that a central communication space such as Loop could improve the efficiency of communication about patient care within a team. The ability to post a single message that is viewable by all team members could save patients and caregivers the time and frustration of repeating information to multiple members of their care team. This ability to access multiple providers with a single post would also be useful when patients or caregivers are not sure to which HCP they should direct their questions or updates. One participant stated the following:

One of the things that I see as being useful about this is it should cut down the amount of time that family members, patients and caregivers are spending repeating information. [Caregiver, adult cancer, ID#09, low-fidelity prototype]
Another participant stated the following: And I think as far as the patient, instead of them sending you an email, if they send something on that system, at least everybody can kind of contribute to it, if there a concern she's having. [HCP, AYAC, PMH, ID#01, low-fidelity prototype] Participants indicated that they currently receive information from a variety of technologies, such as email, paging, and text messaging, and Loop could be useful for consolidating incoming messages in one place. This was primarily voiced by HCPs but also mentioned by some patients. One participant stated the following: ...rather than posting and copying and pasting to multiple doctors, emailing them...and getting their opinion on it-I think it provides an easier and quicker way to get in contact with everybody. [ Patients and caregivers also suggested that being able to post a symptom update or question as it occurs, even if they are not expecting an immediate response, could be more efficient than remembering to ask the question or recall a symptom at their next appointment. One participant stated the following: Patients and HCPs alike felt that this increased transparency would promote patient-centeredness, patient engagement, and coordination between providers.

Patient-Centeredness
Patients and caregivers felt that including the whole team in discussions about care could broaden HCPs' understanding of the patient's needs beyond a specific specialty. HCP participants did not explicitly comment on this. One participant stated the following:

Coordinated Care
Caregivers of adult and CMC patients indicated that they are typically responsible for synthesizing, reconciling, and relaying information between individual providers. They believed that Loop could improve collaboration and cooperation directly between providers, thus decreasing the burden of coordination on the caregivers. Although HCPs did not comment specifically on caregiver burden, they did acknowledge that Loop could facilitate communication between providers, patients, and families across locations to promote collaboration on things such as discharge plans. One participant stated the following:

Factors Negatively Affecting Perceived Usefulness
Participants described several factors that may add tasks or time to their workflow and, consequently, would negatively affect the PU. Participants also identified or alluded to several features that would help to mitigate the negative impact of the factors outlined below.
Across all stages of prototyping, patient, caregiver, and HCP participants felt that including patients and caregivers on all messages would reduce the efficiency of communication within Loop.
In response to this consistent feedback, the option to restrict message visibility to user-specified subteams was introduced in the low-fidelity prototype and further refined as the Team Only setting in the medium-fidelity prototype.
Despite the introduction of the Team Only feature, participants continued to express a tension between a need for efficiency and wanting to maintain patient engagement and transparency. One participant stated the following: HCP participants also expressed concern that failing to integrate Loop with other systems such as EHRs could reduce efficiency if they are required to document or search for information in multiple systems. One participant stated the following: The only thing I worry about is information in two different places. It's the information in the chart and information here...and just both of those pieces of information are a big process. [HCP, AYAC, ID#06, high-fidelity prototype] As a pragmatic workaround to a multiple EHR environment, we introduced a feature in the high-fidelity prototype that exports messages as a PDF for upload into an EHR.
HCP participants felt that high volumes of messages could reduce efficiency by making it difficult to find information that is relevant and has high priority. One participant stated the following: I do think there's a possibility of having many, many messages that are totally irrelevant to certain members of the team and then having the whole page be things that aren't necessarily [relevant]. [HCP, adult cancer, ID#17, high-fidelity prototype] Filtering was described as a way to make specific or relevant messages easier to find by reducing the number of messages one has to sort through. The ability to filter messages in Loop was first introduced in the low-fidelity prototype and was fully functional in the high-fidelity prototype. Messages can be filtered in a number of ways: by threaded conversations, by issue (Issues feature), by messages directed at me (Attention To feature), by messages flagged by the user (Starred Messages feature), or by sender. One participant stated the following: Filtering messages is good. I think that's important because, this, over time is going to be enormous.
[Caregiver, CMC, ID#20, high-fidelity prototype] Another participant stated the following: I think it's great. I think the fact that you can...zero in on the particular issues is really important, because I suspect some of these can go on for weeks and months. [HCP, AYAC, ID#06, high-fidelity prototype] All user groups were concerned about posting messages in Loop that are unread by the intended person. As a result of these missed messages, decision makers may have incomplete or fragmented information. One participant stated the following: Participants suggested that notifications alerting team members about relevant messages and prompting them to log in would be a useful feature. This was felt to be especially important after long periods of inactivity when messages are more likely to be missed. The Attention To feature, which generates email notifications, existed in all prototype fidelities; however, as described in the PEOU section, some participants were not sure how to correctly use this feature. One participant stated the following:

Principal Findings
This study evaluated end users' PEOU and PU of a Web-based clinical collaboration system, Loop. During usability testing sessions, patients, caregivers, and HCPs were able to accomplish tasks testing the core functions of Loop, including viewing, composing, and posting messages. Participants had difficulty interpreting certain visual design elements and using auxiliary features. In these instances, participants were unable to navigate certain features as intended; however, most participants were able to understand features after a brief period of exploring Loop. With regard to usefulness, participants expressed that Loop could be a valuable system for communication between patients, caregivers, and HCPs. Understanding how potential end users perceive the ease of use and usefulness of a technology is important because these factors have been associated with users' intention to adopt a technology [52,54,61,62]. Furthermore, understanding the underlying causes of difficulty associated with features helps in identifying strategies to improve the usability of Loop.
Difficulty using the Attention To and Team Only features highlights the value of adhering to visual design conventions to improve user experience [63][64][65]. Although the concepts of message notifications and visibility are used in other social media platforms, it is possible that participants' existing mental models, beliefs about how a system will work based on previous experiences, may have conflicted with the terminology we applied [66].
Older adults reported reduced comfort using most technologies including social media. This is consistent with Pew Research Centre survey data on social media adoption and usage trends from 2013 [67]. Lack of familiarity, complex interfaces, and privacy concerns have been cited as barriers to technology uptake in older adults [68,69]. To make new technologies accessible to older and new users, interfaces should be simple and consistent, and language should be easy to understand and free of jargon that assumes users' prior knowledge [68,69].
Following a brief explanation by the usability testing facilitator about system features and the introduction of cursor labels, participants felt that Loop would be easier to navigate in subsequent uses. It is possible that embedded instructional features, such as expanding the application of cursor labels to visual cues, may also aid novice navigation and reduce the time, errors, and difficulty associated with task completion [70]. Any instructional elements will need to be unobtrusive for those more familiar with social media conventions.
Participants in this study highlighted Loop's potential to improve communication and collaboration. We are not aware of any randomized control trials that demonstrate the impact of communication technologies on cross-institutional and interprofessional collaboration or on patient outcomes, such as symptom management, quality of life, length of stay, or mortality rates. Results from a pilot randomized control trial suggest a trend of improved continuity of care with access to Loop [71]; however, this finding must still be confirmed in a full-scale effectiveness trial. Without this type of robust evaluation, it is difficult to predict how the perceived benefits of Loop will translate to real-life use.
Unlike the factors impacting PEOU, which relate to how the system is designed (looks or operates), the factors impacting the PU generally relate to Loop's perceived impact on participant workflow. Although we do not have data on actual impact, participants in this study predicted a number of workflow disruptions that have been previously noted in other evaluations of eHealth integration into clinical contexts: user frustration about not knowing whether message was received [22], greater quantity of messages [40] resulting in decreased quality of messages [22], duplication of workflow [72,73], altered communication patterns [72,74], and difficulty identifying important information because of abundance of information [74,75]. Several reports have found that actual message volume between patients and HCPs within electronic communication tools tends to be modest [39,76,77], suggesting that provider concerns may be unwarranted. Access to Web-based messaging does not appear to impact the frequency of face-to-face visits [37,38] or telephone and email volume [39,76] but rather supports these typical interactions [26].
Some of the barriers to adoption identified by the participants in this study are not represented by TAM. Indeed, TAM has been critiqued for not considering the impact of external factors, such as user workflow, organizational characteristics, and social context, on users' acceptance of technologies [54,78,79]. Financial compensation structures, lack of empirical evidence about a system's usefulness, personal characteristics such as computer experience [78,79], professional conflict [25], and power dynamics [80] are other examples of external factors that act as barriers to system adoption. An updated version of TAM, the unified theory of acceptance and use of technology, includes the variable Facilitating Conditions, which acknowledges the influence of perceived organization and technology infrastructure on uptake [55]. To be successful, any real-world implementation will need to consider a full range of internal and external factors.
Several features to mitigate negative impact on workflow were integrated in later stages of Loop's prototyping. In some cases, these features were difficult to use or were overlooked entirely. Not surprisingly, system features are only useful if they are easy to identify and use [52,61,75]. Indeed, studies have shown a statistically significant relationship between PEOU and PU of a tool [59], reinforcing the need for ongoing UCD and evaluation throughout the life cycles of development and implementation.

Limitations
Findings of this study are based on participants' subjective feedback about the system and not evaluation of objective measures, such as actual time to complete tasks or number of errors. Additionally, participant feedback about how they might use the system was based on an interaction with the system guided by a clinical scenario and not in real life. Introducing information technology into complex adaptive health care environments has additional design, development, implementation, and evaluation challenges across a range of domains: hardware and software, clinical data definitions, human computer interfaces, incentives and behavior, workflow and communication, internal policies, external regulations, and the need for ongoing monitoring [81]. Although scenario-based usability testing can address and anticipate many issues in real clinical environments, real-life complexities create a need for ongoing assessments of how tools work across settings and users. Ongoing evaluations of Loop, including a pragmatic randomized trial, continue to assess PEOU and PU as key metrics in parallel to health-related outcomes and systemic factors that impact usage and behavior.
The majority of HCPs who participated in this study were recruited from academic institutions in an urban setting.
Recruitment began with the networks of the study investigators, all of whom work in academic institutions, resulting in less representation from community or rural settings. However, we sampled patients, caregivers, and HCPs across different complex care populations, and the feedback was consistent.
Usability testing was structured to test specific functions and features that had been introduced or updated during a developmental iteration. This influenced the functions or features that participants talked about in each cycle of testing. Participants were given tasks but not instructions on how to complete them. It is possible that some of the errors observed may have resulted from misunderstanding the task rather than with the system itself. Using the think aloud approach, participants were more likely to verbalize negative feedback and less likely to comment on easily navigated features when interacting with the system.

Conclusions
Loop was perceived to have the potential to improve the efficiency and effectiveness of communication about patient care. Results from usability testing point to the importance of having intended users interact with the system at early stages of development to ensure the system is both usable and useful, thereby increasing chances of system adoption in a real-life setting. A number of issues with the system were anticipatory, concerning potential challenges with integrating the system into real-life environments and workflows rather than proximal usability problems. It is, therefore, essential to continue assessing and enhancing user experience throughout the next phase of research including real-world implementation. Future research should examine the broader sociotechnical characteristics that will influence the implementation and overall benefit of Loop in clinical care.