Usability testing of an electronic patient-reported outcome system linked to an electronic chemotherapy prescribing and patient management system for patients with cancer

Background People affected by cancer experience a wide range of symptoms which have a major impact on their functioning and health-related quality of life (HRQoL). One way to measure the impact of cancer symptoms is through the use of patient-reported outcomes. Methods An electronic patient-reported outcome (ePRO) application (ChemoPRO®) was designed to be used by cancer patients to report their symptoms and communicate with their clinical team. Usability testing sessions were conducted with people with lived experience of cancer to understand how real users interact with the ChemoPRO® system. One-to-one testing sessions were conducted to assess use of the system and identify areas for further refinement. User satisfaction was assessed using a brief satisfaction questionnaire previously used by Aiyegbusi et al. (date). Results Ten people with lived experience of cancer took part in the usability study. Symptoms and HRQoL measures, including the Euroqol EQ5D5L and the PRO-CTCAE™ were included in the ePRO system. Participants had a mean age of 62.3 years. Three critical errors and 21 non-critical errors were reported. All participants were enthusiastic about the app. Participants liked the simplicity and responsiveness of the patient-facing app and highlighted the potential for communicating with their clinical team. The overall usability and satisfaction score was 4.5 (sd = 0.09). Conclusion This usability study suggests that people with lived experience of cancer found the ChemoPRO® app acceptable and easy to use. One of the key features of this particular ePRO system that should be developed further is system functionality to facilitate communication between patients and clinicians. Future testing should include testing in a clinical setting and testing with people from ethnic minorities.


Introduction
In 2020, there were around 20 million new cancer cases worldwide and it is estimated that this number will rise to over 27 million by 2040 [1]. Similarly, cancer incidence has increased in the United Kingdom (UK) by 19% in the last decade and will continue to do so [2]. Each year, there are around 450,000 new cancer cases (605 per 100,000) [3]. It is estimated that, in the UK, around three million people are currently living with cancer [4]. These patients may experience a wide range of disease and treatment symptoms and treatment-related toxicities that greatly affect their functioning, heath-related quality of life (HRQoL) and their impact on healthcare resource use [5,6]. Patients' journeys through cancer treatment are increasingly captured with electronic systems, such as Chemo-Care® which is in use in several National Health Service (NHS) trusts in the UK [7].
ChemoCare® is an electronic chemotherapy prescribing and patient management system. The electronic platform was designed by CIS Oncology (https://www.cis-healthcare.com/), an ISO certified med tech company based in the UK (ISO 13485:2016, EN ISO 13485:2016, ISO 9001:2015, ISO 27001:2013). ChemoCare® is used for prescribing, preparing and administering chemotherapy, as well as scheduling treatment and sharing data. It allows instant access to data by the clinical team and patients, ensuring accurate and fully informed prescribing at all times. However, to date, the system does not include any patient reported outcomes (PROs), measures of health status coming directly from the patient without interpretation by a clinician or anyone else [8]. Clinical teams and patients are keen to see PROs included in the electronic platform to assess the impact of disease and treatment on patients' symptoms and quality of life from a patient perspective. Electronic capture of PROs (ePROs), using online validated questionnaires, has important advantages over paper forms [9][10][11]. The collection of ePROs removes the need for manual data entry which carries the risk of human error and facilitates prompt review by the clinical team to tailor care to individual patient needs [10,11].
PRO assessment in an oncology setting offers a range of potential benefits for individual patient care and for clinicians, regulators, healthcare management teams, commissioners, and policy makers. Benefits include improved symptom management and symptom control and enhanced patient-clinician communication, patient satisfaction and well-being [9,12]. In addition, in the US the use of PROs to monitor patient symptoms in routine cancer care has been shown to lead to clinical benefits include fewer A&E visits, hospitalisations and an increase in survival [9].

ChemoPRO®
ChemoPRO® is an app linked to the ChemoCare® platform described above. It was designed by CIS Oncology, with ongoing input from clinical stakeholders. The ChemoPRO® app helps hospitals and other stakeholders maintain contact with their patients in between visits. Patients can record symptoms and wellness which can be accessed by the healthcare team. It also allows patients to access a range of information, such as appointments and medication reminders.
The ChemoPRO® app consists of several modules: 'Appointments', 'How are you?', 'My symptoms', 'Logs', 'Info', and 'Help'. The 'How are you?' and 'My symptoms' modules refer to two PROMs (EQ5D-5L and PRO-CTCAE™, as outlined in methods), as shown in CIS Oncology are GDPR and ISO 27001 compliant to ensure that all patient data is held securely in line with current best practice and legislation.

Usability testing
It is vital that the user-friendliness and usability of ePRO systems are assessed and improved prior to implementation in a clinical setting to reduce attrition rates and to ensure that the system performs efficiently and effectively [13]. Usability testing is one of the stages of system development (Fig. 1).
Usability testing is a technique that involves the testing of systems, products or websites with participants drawn from the target population. It aims to assess how effective and efficient systems are, as well as users' satisfaction with the system [14]. It can be conducted during a single session or in a more iterative manner during several sessions [15]. In our case, it allows end users to actually test ePRO systems and can assist ePRO developers in the evaluation of ePRO user interface [15].
Therefore, the aim of this study was to assess the usability of the CIS Oncology ChemoPRO® platform with individuals with lived experience of cancer.

Ethical considerations
This usability study was approved by the University of Birmingham Research Ethics committee (ERN_22-0510).

Study participants and recruitment
People with lived experience of any type of cancer were recruited by sharing the study advert on social media (Twitter) and with the National Cancer Research Institute (NCRI) Consumer Forum and a Patient and Public Involvement (PPI) panel based in Birmingham in November 2022. Further inclusion criteria can be found in Box 1.
Box 1 -list of inclusion criteria Previous research informed our sample size [15][16][17][18][19]. It has been shown that 8-10 participants, and sometimes as low as 5 if planning to do a series of testing sessions [18], are required to detect over 80% of issues [17,19], as shown in Fig. 2.
By the time we interviewed the last participant, it was apparent that we had reached saturation, as no new issue was being mentioned.
People interested in taking part in the study contacted the researcher (CM) who emailed them to confirm their eligibility and sent them a participant information sheet and a consent form to consent to participation in the study and publication of the results. A mutually suitable date and time to conduct the usability testing session was then arranged, after obtaining written informed consent. In Fig. 2. Sample sizes for usability studies (reproduced with permission of the Nielsen Group) [19]. total, nine people with cancer were recruited.
People who completed the study were given a £10 voucher in return for their participation in the study. Clinicians were not included in the current usability study as they have been heavily involved in the planning and design of the app.

ChemoPRO® registration process
Once participants gave their informed consent, CIS Oncology emailed them a link to download and register onto the ChemoPRO® app. This invitation email included a link to GooglePlay/App Store and a numerical code to enter after downloading the app. Participants then downloaded the app and registered (Fig. 3).

EQ-5d-5L
The Euroqol EQ-5D-5L [20] comprises 5 items plus 1 visual analogue scale. It has been widely validated across a range of conditions and used to assess health outcome from a wide variety of interventions on a common scale, for purposes of evaluation, allocation and monitoring [21,22]. It is used by the National Institute for Health and Care Excellence (NICE) in health technology assessment. EQ-5D-5L, takes only a few minutes to complete and has a same day recall period.

PRO-CTCAE™
The Common Terminology Criteria for Adverse Events (CTCAE) is routinely used in oncology as the means of categorising adverse events in trials and is a descriptive terminology that evolved alongside other toxicity grading systems to become a comprehensive, multi-modality grading system for reporting the acute and late effects of cancer treatment [21,22]. The PRO-CTCAE™ was developed to characterise the frequency, severity and interference of 78 symptomatic treatment toxicities that could be meaningfully reported from the patient perspective. It has been established that symptomatic adverse events are under-detected by clinicians versus patients, who are better at identifying low-grade symptomatic events [23,24]. As such, the PRO-CTCAE™ could generate information that complements CTCAE data and can provide a more holistic insight into patient experience on treatment, which is integral for clinical decision-making. In oncology, the PRO-CTCAE™ has demonstrated favourable validity, reliability, and responsiveness [25].

Testing sessions
The one-to-one testing sessions consisted of performing a series of tasks on the app whilst talking to the researcher about their experience ('Think Aloud' technique) [26]. The list of tasks performed by the participants can be found in Box 2. Although one of these tasks was to complete the two PROMs included, we did not use the responses to the questionnaires to assess the usability of the app. This will be part of the next stage of the evaluation, the feasibility study. Effectiveness was assessed by recording the type and number of errors. Critical errors were errors that prevented participants from using the app and completing the tasks unless the researcher stepped in and advised participants on how to resolve the issue. Conversely, non-critical errors were minor errors that did not prevent participants from completing the testing session. Efficiency was assessed by providing an estimation of the time it took participants to complete the EQ5D5L PROM. Participants also identified areas of improvement and made suggestions to improve the app and the users' experience. This was followed by a short qualitative interview to capture their overall experience of using the app (retrospective probing) [27]. Finally, participants were asked to complete a short demographics questionnaire and a brief satisfaction questionnaire previously used by Aiyegbusi et al. [16] The satisfaction questionnaire included four questions to rate the usability of the system and participants' satisfaction on a 5-point scale (1 = very dissatisfied, 5 = very satisfied) [16].
Box 2 -List of tasks performed by participants.

Data analysis
Participant characteristics were summarised as n (%). Comments from all the participants were recorded in a table alongside observations from the researcher, whether participants completed the tasks independently or with support, how they found the user experience, and what could be done to improve the ChemoPRO® app. Participants' comments during the interviews were extracted as quotes and categorised under "Overall comments and impressions" ( Table 5). Participant ratings for the four usability questions were used to calculate a mean score per question.

Participant characteristics
Ten people with lived experience of cancer took part in this usability study. This included six women and four men. The mean age of the participants was 62.3 years (range = 44-76 years). Table 1 presents the participants' characteristics.

Assessment of effectiveness
There were three critical errors and 21 non-critical errors ( Table 2). The critical errors related to the email invitation to download the app which were either not received by participants (n = 2) or were illegible (n = 1). In the case of the illegible email, the researcher advised the participant to download the app directly from GooglePlay/the App store and use the login code which was in the invitation email (and easily readable). The researcher became aware that two participants who did not receive the invitation email when she checked with the participants the day before the interview. CM notified CIS Oncology staff, who sent another invitation email. The 21 non-critical errors included missing information (under Appointments and About my Hospital), not knowing how save logs, and not knowing how to change password when invited to do so.

Assessment of efficiency
Although it was difficult to assess the time it took participants to complete each PROMs (especially PRO-CTCAE™) as they stopped to talk to the researcher during the process, it is estimated it took participants between 1 and 2 min to complete the EQ5D5L.

Suggestions for improvement
Participants identified potential areas to improve the user experience of ChemoPRO® (Table 3). These issues included (i) the app content not always fitting on a small phone screen and the participants not being able to read everything, (ii) lack of clarity, (iii) removing unnecessary items to avoid overburdening the users while at the same time giving them enough information (about their upcoming appointments, for example), (iv) extending communication with clinical team, and (v) providing additional ways to view their previous symptoms. Table 4 presents participants' rating of the usability and their satisfaction with ChemoPRO®. The mean score for individual question was high and the overall average score for usability and satisfaction was 4.5 (5-point scale).

Participants overall comments and impressions of ChemoPRO® app and researcher's observations
Overall, all participants were very positive about using the ChemoPRO® app. They found the app easy to use and navigate. They liked the colour scheme, the layout and found the buttons responsive and had no issue around online safety. Participants believed ChemoPRO® had potential for improving communication with clinicians and keeping a record for patients and clinicians of their symptoms/quality of life data over time. They were particularly impressed with the Appointments section, which they thought would  • Provide additional ways to view previous symptoms.
• Provide line graphs to make it easier for some people to view and understand their data.
Wrong questions in 'Your symptoms' • Some of the questions in 'Your symptoms' did not match the symptom associated with it. Patients' overall comments and impressions of the app, as well as the researchers' observations can be found in Table 5.

Discussion
This article presents the results of the usability testing of the ChemoPRO® app used by people with lived experience of cancer. In addition to two PRO measures (EQ-5D-5L and PRO-CTCAE™), ChemoPRO® included several features designed to facilitate management of the users' condition. Participants found the app easy to use and navigate and completed the questionnaires with ease, despite encountering a few issues. The errors found were mainly due to programming issues with the app, demonstrating the value of usability testing. Participants believed the app had potential to facilitate communication with their clinical team and suggested areas for further refinement.
This study adds to the existing body of research focusing on testing the usability of a new electronic applications. Usability testing is a key factor for successful implementation [16], however, this is a stage that is often overlooked [28]. All participants used their own "Bring Your Own Device" (BYOD). The app was used successfully across all age groups. Although two participants (both aged over 60 years old) said they did not feel very confident using technology, both completed the usability testing unaided without any major issues [16,29]. This may be due to the fact that more older people are using smartphones on a regular basis and that smartphones facilitate digital communication [30].
Although it was difficult to time how long completing the questionnaires took, as the participants stopped often to talk to the researcher, it became clear that some of the participants were starting feel ePRO fatigue (shown by lack of concentration or forgetfulness). This might cause users to disengage from using the app. Although this is an important aspect to consider, previous work has shown that cancer patients have successfully completed all items of the PRO-CTCAE™ [31].
One of the ways we recruited participants was through social media. Although recruiting participant on social media has advantages, such as being timesaving and inexpensive, it can pose some problems, such as having less control over who joins the study (in the case of passive recruitment). In our case, several people who asked to join the study (n = 6) were not included in the study as the researcher (CM) had strong doubts about their eligibility and motivation for joining the study. The email messages sent to express their interest in the study were worded exactly the same, were not signed, and contained the same grammar mistakes. The first two of these people who wanted to join the study never replied to CM's request to set a date/time for the testing session. Their motivation may have been the voucher offered to participants, which, although was not included on the study ad posted on social media, was included in the participant information sheet. As a result, a lot of time was spent to identify the potentially 'fake' participants, which defeats the purpose of using social media to recruit research participants.

Implications for ePROMs developers
Some participants thought that some of the features of the app would benefit from being removed in order to make the app as Table 4 Usability and satisfaction score.

Question
Average score (sd) Ease of use and navigation 4.4 (0.51) Satisfaction with content 4.6 (0.51) Satisfaction with visual display 4.5 (0.52) Likelihood of using again or recommend it to others (when available) 4.6 (0.69) Average usability and satisfaction score 4.5 (0.09) simple as possible and avoid overburdening users. Therefore, ePRO programmers need to ensure that its use is as simple as possible for participants. Participants liked the fact that they were able to view their previous answers to the questionnaires. However, most of them thought that presenting previous answers in a visual way (i.e., line graph) rather than in a list form (as it currently is the case) would help them understand how their symptoms/quality of life have changed overtime. It is recommended that developers follow PROTEUS Consortium's existing guidance on presentation of PRO data [32,33].

Strengths and limitations
The main strength of our study is its methodology which includes a 'think aloud' technique which provides real-time feedback. We are aware that our study has limitations. The sample size limited the diversity within our participant sample. Although we actively tried to recruit a diverse sample and participants in our usability study were all highly educated. Despite reaching out to a number of PPI groups and charities, including charities working specifically with ethnic minority groups, three people from ethnic minorities declined taking part in our study. One explanation for having difficulties finding participants from ethnic minorities is the lower cancer incident rate in the Asian and Black ethnic groups, and in people of mixed or multiple ethnicity, compared with the White ethnic group, in England [2]. This will be addressed in the next stage of the study which will be a feasibility study. The feasibility study will also include a larger sample and will test the electronic platform in a clinical setting on a long-term basis.
It was difficult to assess efficiency as participants stopped to talk to the researcher while completing the questionnaires, especially PRO-CTCAE™. Although it was not possible to provide an estimate of time it took participants to complete the process, prior work with PRO-CTCAE™ suggest that on average it takes users 11.1 s to answer one question (sd = 8.4) [34].

Conclusion
Our study showed people with lived experienced of cancer can easily report their symptoms/quality of life through the use of an electronic system. One of the key features of this particular ePRO system that should be developed further is the app's ability to facilitate patient-clinician communication. Future testing should include testing in a clinical setting and testing with people from ethnic minorities and people with a lower education level.

Author contribution statement
Christel McMullan: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper. Sarah E Hughes; Olalekan Lee Aiyegbusi; Melanie Calvert: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement
This study was funded by CIS Oncology.

Ethics statement
This usability study was approved by the University of Birmingham Research Ethics committee (ERN_22-0510).

Data availability statement
Data will be made available on request.