Development and usability testing of an electronic patient-reported outcome measure (ePROM) system for patients with advanced chronic kidney disease
Introduction
Chronic kidney disease (CKD) is a long-term medical condition associated with symptoms such as fatigue, pain and pruritus which may negatively impact on patients' health-related quality of life (HRQOL) [[1], [2], [3]]. While the use of clinician-reported outcomes is essential in the management of patients with CKD, relying exclusively on these clinical parameters may underestimate the impact of the disease and its treatment on patients' HRQOL [4,5]. A patient-reported outcome (PRO) is defined as “any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else.” [6,7] Self-reported questionnaires, known as patient-reported outcome measures (PROMs), are standardized instruments designed to capture PRO information [6,7]. PROM data could complement clinical parameters and inform the management of patients with advanced CKD [4,8].
Traditionally, PROMs have been administered using a paper-based format [9]. However, in recent years, there has been a widespread interest in adapting and developing PROMs for electronic administration via telephone (interactive voice response) or screen-text devices [10] such as desktop and laptop computers, tablets and smartphones.
The use of electronic PROMs (ePROMs) may facilitate the remote monitoring of patients' symptoms/HRQOL and provide clinicians the opportunity to initiate timely interventions to delay disease progression [[11], [12], [13]]. Additional benefits may include: a lower administrative burden, increased acceptance rates, prevention of secondary data entry errors, and lower incidence of missing data [9,10,14].
In Denmark, the generic ePROM system, AmbuFlex, has been successfully implemented for tailoring the care of various patient groups including patients with renal failure [15,16] while the Advanced Symptom Management System (ASyMS) and the eRAPID system have been successfully used in the UK to monitor the side effects of chemotherapy [17,18].
It is essential that the usability of an ePROM system is formally assessed during development to ensure it is fit for purpose [10,19]. The International Organization for Standardization (ISO) defines usability as “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” [20] According to ISO, effectiveness describes the ability of users to complete pre-determined tasks during a usability test while efficiency refers to the level of resource required to perform these tasks [20]. Satisfaction relates to the subjective views of users based on their test experience [20].
When assessing these three aspects of usability, consideration needs to be given to the context of use [[21], [22], [23]]. Participant characteristics such as age and health status would therefore determine the specific methods to employ and the metrics to measure during a usability study [[21], [22], [23]]. Patients with CKD tend to be older adults [24,25] who may have age-related physical and cognitive limitations [26,27]. They may also experience a number of debilitating CKD-related symptoms such as fatigue and cognitive impairment which could significantly affect their ability to use an ePROM system [28,29]. These age and health-related issues need to be taken into account when designing and testing an ePROM system for this patient group. It is also crucial that patients iteratively [30] assess the usability of the system so that usability issues may be detected and addressed prior to full-scale implementation [31] in other to reduce attrition rates [26,32,33].
At the start of this project, a systematic review of PROMs used in patients with CKD was conducted. The review found evidence to support the use of the 80-item kidney disease quality of life-short form (KDQOL-SF) [34] and the 36-item kidney disease quality of life-36 (KDQOL-36) [35]. However, very few studies validated these two measures in our target population (stages 4 and 5 CKD) [35,36]. The review also identified the IPOS-Renal (11 items) [37], which was undergoing validation at the time.
A patient advisory group evaluated the acceptability, burdensomeness and relevance of the KDQOL-SF, KDQOL-36 and the IPOS-Renal. The patients expressed a preference for the KDQOL-36 and IPOS-Renal as they were brief and easy to understand [38]. Their preference for shorter, and therefore less burdensome, questionnaires is understandable given that patients with advanced CKD often suffer from fatigue and lack of energy [1,3], which may make completing longer questionnaires KDQOL-SF on a regular basis a significant challenge. Therefore, we adapted the KDQOL-36 and the IPOS-Renal for the renal ePROM system. In order to comply with the questionnaire developers' terms of use, we had to keep the user interface as similar as possible to the original paper versions. However, we still followed a number of recommendations for web-design for elderly users [39] and the interface was designed to be simple and straightforward to minimise patient burden. For example, we avoided the need for pull down menus, double clicking and kept the number of pages to click through to a minimum, as ability to precisely position the computer cursor has been shown to diminish with age [26,39,40]. Older individuals may also have issues with visual acuity, contrast sensitivity and colour discrimination [41]. Therefore the colour palette was restricted and the text for the questionnaires was presented on a neutral background using black Arial font, which is an easy to read sans-serif font (See Fig. 1).
The electronic adaptation was performed by a senior. Net developer from the Application Development team, University Hospitals Birmingham NHS Foundation Trust (UHB) using the DataCollector application developed in-house (See Fig. 1, Fig. 2, Fig. 3) [38]. The DataCollector has two sections - the ‘back end’ of the application is the administrative section which is used to create and manage questionnaires while the ‘front end’ is the user section which enables patients and/or staff to answer questionnaires. The DataCollector was developed using Microsoft.Net technology, mainly ASP.Net Webforms, C#, Entity framework and SQL Server. Bootstrap framework was used to make the ‘front end’ as responsive as possible to enhance its performance on electronic devices and on most of the main web browsers. The DataCollector was embedded in myhealth@QEHB, a secure electronic patient portal also developed by the Application Development and Informatics team (See Fig. 3) [42].
Section snippets
Methods
This usability study was designed and conducted according to the study protocol [38], following guidelines and recommendations provided by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [10,19], and the United States Department of Health and Human Service [43]. The study was approved by the West Midlands Edgbaston Research Ethics Committee (Reference 17/WM/0010) and received Health Research Authority (HRA) approval on 24 February 2017. Project authorisation was
Results
Table 1 presents the participant demographics. The eight participants had a mean age of 64.3 years (range: 36–87 years).
Summary of main findings
This article reports the usability testing of the Renal ePROM system in a group of patients with advance CKD. Our study suggests that patients with advanced CKD may find the Renal ePROM system easy to use and acceptable for reporting their symptoms remotely. Error levels were relatively low and mostly due to non-critical omissions. Overall, the system was found to be efficient and effective despite the few issues identified.
Findings in relation to existing literature
The opinion of study participants' that the renal ePROM system is
Conclusion
Although the digital divide between older and younger populations is decreasing [79], older individuals have a tendency to discontinue the use of health information technology [80]. In order to minimise post implementation attrition rates, we have involved patients from our target population in the design and development of the ePROM system [32]. We have also conducted this usability test with patients, who represent our target users [33] in order to assess the acceptability and usability of
Authors' contributions
MC is the guarantor. The study was conceived and designed by OLA, MC, DK, PC and TM. RA, OLA, DK worked on the electronic adaptation of the PROMs. MD and NWA recruited the participants for the study. OLA conducted the usability testing and interviews. OLA analysed the data and drafted the manuscript. The manuscript was reviewed and the final draft approved by all authors.
Conflicts of interest
None declared.
Acknowledgements
This project was funded as part of the Health Foundation's Improvement Science Programme. The Health Foundation is an independent charity working to improve the quality of healthcare in the UK.
DK is funded by the National Institute for Health Research Fellowship. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research, Health Education England or the Department of Health.
TM is partly funded by the NIHR
References (80)
- et al.
Chronic kidney disease
Lancet
(2012) - et al.
Electronic and computer-generated patient questionnaires in standard care
Best Pract. Res. Clin. Rheumatol.
(2007) Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report
Value Health
(2009)Development of an integrated electronic platform for patient self-report and management of adverse events during cancer treatment
Ann. Oncol.
(2017)Validation of electronic systems to collect patient-reported outcome (PRO) data-recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force
Value Health
(2013)“Think aloud” and “Near live” usability testing of two complex clinical decision support tools
Int. J. Med. Inf.
(2017)Validation of the IPOS-Renal Symptom Survey in advanced kidney disease: a cross-sectional study
J. Pain Symptom Manag.
(2018)Validity and reliability of KDQOL-36 in Thai kidney disease patient
Value Health Reg. Issues
(2013)Implementation of computer-based quality-of-life monitoring in brain tumor outpatients in routine clinical practice
J. Pain Symptom Manag.
(2010)- et al.
Social support and self-care behaviors in individuals with heart failure: an integrative review
Int. J. Nurs. Stud.
(2014)
The role of health care providers and significant others in evaluating the quality of life of patients with chronic disease
J. Clin. Epidemiol.
Quality of life of patients with advanced chronic kidney disease receiving conservative care without dialysis
Semin. Dial.
Symptom burden in chronic kidney disease: a review of recent literature
J. Ren. Care
Use of patient-reported outcomes to improve the predictive accuracy of clinician-reported adverse events
J. Natl. Cancer Inst.
Listening to the patient: a practical guide to self-report questionnaires in clinical care
Arthritis Rheum.
Patient-reported outcome measures: use in medicinal product development to support labeling claims
Guidance for Industry
Design and Analysis of Quality of Life Studies in Clinical Trials
Agreement between patient-reported symptoms and their documentation in the medical record
Am. J. Manag. Care
Collection of daily patient reported outcomes is feasible and demonstrates differential patient experience in chronic kidney disease
Hemodial. Int.
Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial
J. Clin. Oncol.
A patient-centred approach to measuring quality in kidney care: patient-reported outcome measures and patient-reported experience measures
Curr. Opin. Nephrol. Hypertens.
Automated collection of quality-of-life data: a comparison of paper and computer touch-screen questionnaires
J. Clin. Oncol.
AmbuFlex: tele-patient-reported outcomes (telePRO) as the basis for follow-up in chronic and malignant diseases
Qual. Life Res.
Use of patient-reported outcome (PRO) measures at group and patient levels: experiences from the generic integrated PRO system, WestChronic
Interact J Med Res
Evaluation of a mobile phone-based, advanced symptom management system (ASyMS) in the management of chemotherapy-related toxicity
Support. Care Canc.
Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) -- Part 11: Guidance on Usability
Methods for human – computer interaction research with older people
Behav. Inf. Technol.
SUS: a “quick and dirty” usability scale
SUS: a retrospective
J. Usability Stud.
Chronic kidney disease in the elderly: evaluation and management
Clin. Pract.
User-centered evaluations with older adults: testing the usability of a mobile health system for heart failure self-management
Proc. Hum. Factors Ergon. Soc. Annu. Meet.
Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved
Evid. Rep. Technol. Assess.
Considerations and challenges in selecting patient-reported outcome measures for clinical trials in nephrology
Clin. J. Am. Soc. Nephrol.
Why does quality of life remain an under-investigated issue in chronic kidney disease and why is it rarely set as an outcome measure in trials in this population?
Nephrol. Dial. Transplant.
Iterative methodology and designer training in human-computer interface design
What health topics older adults want to track: a participatory design study
Testing usability and acceptability of a web application to promote physical activity (iCanFit) among older adults
JMIR Human Factors
Kidney Disease Quality of Life Short Form (KDQOL-sf) Version 1.3: a Manual for Use and Scoring
Psychometric properties of the kidney disease quality of Life-36 questionnaire (KDQOL-36)
West. J. Nurs. Res.
Cited by (20)
Patient and Clinician Perspectives on Electronic Patient-Reported Outcome Measures in the Management of Advanced CKD: A Qualitative Study
2019, American Journal of Kidney DiseasesCitation Excerpt :Unsurprisingly, clinicians requested clear guidance on the meaning of and the appropriate response to ePROM data as recommended in literature.39,40 To be fit for purpose, the ePROM system needs to be efficient, effective, and satisfactory for stakeholders.41,42 If appropriately addressed, these study findings could assist with the design and implementation of the renal ePROM system, improve its usability, and facilitate stakeholder adoption.43