User Perceptions and Use of an Enhanced Electronic Health Record in Rwanda With and Without Clinical Alerts: Cross-sectional Survey

Background Electronic health records (EHRs) have been implemented in many low-resource settings but lack strong evidence for usability, use, user confidence, scalability, and sustainability. Objective This study aimed to evaluate staff use and perceptions of an EHR widely used for HIV care in >300 health facilities in Rwanda, providing evidence on factors influencing current performance, scalability, and sustainability. Methods A randomized, cross-sectional, structured interview survey of health center staff was designed to assess functionality, use, and attitudes toward the EHR and clinical alerts. This study used the associated randomized clinical trial study sample (56/112, 50% sites received an enhanced EHR), pulling 27 (50%) sites from each group. Free-text comments were analyzed thematically using inductive coding. Results Of the 100 participants, 90 (90% response rate) were interviewed at 54 health centers: 44 (49%) participants were clinical and 46 (51%) were technical. The EHR top uses were to access client data easily or quickly (62/90, 69%), update patient records (56/89, 63%), create new patient records (49/88, 56%), generate various reports (38/85, 45%), and review previous records (43/89, 48%). In addition, >90% (81/90) of respondents agreed that the EHR made it easier to make informed decisions, was worth using, and has improved patient information quality. Regarding availability, (66/88) 75% said they could always or almost always count on the EHR being available, whereas (6/88) 7% said never/almost never. In intervention sites, staff were significantly more likely to update existing records (P=.04), generate summaries before (P<.001) or during visits (P=.01), and agree that “the EHR provides useful alerts, and reminders” (P<.01). Conclusions Most users perceived the EHR as well accepted, appropriate, and effective for use in low-resource settings despite infrastructure limitation in 25% (22/88) of the sites. The implementation of EHR enhancements can improve the perceived usefulness and use of key functions. Successful scale-up and use of EHRs in small health facilities could improve clinical documentation, care, reporting, and disease surveillance in low- and middle-income countries.


Introduction Background
Effective and high-quality health care requires high-quality, timely health information-"Information is care" [1]. Scaling up effective care for millions of patients with HIV in resource-limited settings such as sub-Saharan Africa required the development of new paradigms for the collection, storage, viewing, and analysis of clinical data and health information [2]. Most health centers treating HIV started with only structured paper records. As the volume of patient data grew and in-country digital capacity improved, electronic tools were introduced. Many early electronic health records (EHRs) in resource-limited settings have been developed for HIV care, including those in Malawi [3], Kenya [4], and Haiti [5]. These projects demonstrated the feasibility of deploying health information systems, improvements in reporting to ministries of health (MoHs) and donors, and the ability to monitor the continuum of care. Furthermore, this initial evidence also suggested that the use of EHR systems for HIV, tuberculosis (TB) and multidrug-resistant TB treatment could improve the quality of care [2]. A critical challenge to improving the quality of care in low-income settings is the ability to achieve long-term, consistent EHR use at a large scale. To better understand the perceptions and clinical uses of EHR systems that support improved use and care in Rwanda, we conducted a quantitative user survey supplemented by free-text questions. For the purposes of this study, the terms electronic health record and electronic medical record are used interchangeably.

HIV Care in Rwanda
Rwanda is an East Central African country bordering Tanzania, Uganda, Burundi, and the Democratic Republic of the Congo. Rwanda had a per capita income of US $773 in 2018, up from US $241 in 2004 [6], and has made great progress in rebuilding its health care systems after the genocide against Tutsi in 1994. A major health challenge Rwanda has faced, along with neighboring countries in Africa, is the HIV epidemic. A 2018 to 2019 survey indicated that HIV prevalence among adults aged 15 to 49 years was 2.6% [7]. Great strides have been made in the treatment of patients who are HIV positive, including improvements in the prevention of mother-to-child transmission uptake, and reduction in the rate of loss to follow-up for patients receiving antiretroviral therapy (ART) in Rwanda. This is demonstrated by the near achievement in 2019 of the 2020 Joint United Nations Programme on HIV/AIDS 90-90-90 goal, with 84% of adults who were HIV positive knowing their status, 98% of those knowing their status on ART, and 90% of those on ART having a suppressed viral load [7]. From the beginning of the HIV treatment scale-up, the Government of Rwanda has emphasized care and prevention in rural areas as well as in urban settings; recruitment, training, and supervision of community health workers; and the use of health information systems. These information systems included national-level surveillance systems for HIV care [8,9], mobile health systems to support antenatal and primary care, and patient information or EHR systems mainly for supporting HIV care in health centers and hospitals. The 3 main EHR systems used have been OpenMRS (OpenMRS Inc) in health centers and 36 district hospitals offering HIV services, IQcare (International Quality Care, Palladium Inc) [10] in some health centers (now replaced by OpenMRS), and OpenClinic (OpenClinic GA) [11] in some hospitals. Since 2009, the MoH has moved to using OpenMRS for all HIV health centers and most hospitals in the country.

OpenMRS
OpenMRS is an open-source software platform for building EHRs, with a focus on health care needs in low-and middle-income countries (LMICs). Founded in 2004, the OpenMRS community set goals to create a public software platform to assist health care organizations worldwide in developing EHR systems that were adaptable to local needs, owned by local organizations, and programmed by local developers as much as possible [12] (Textbox 1). Textbox 1. The OpenMRS electronic health record system.
OpenMRS has an unusual modular architecture allowing modules from the core development team to be mixed with modules from other developers to create flexible and updatable systems, with typical implementations using 35 to 45 modules. This ensures the core OpenMRS code is common to nearly all OpenMRS installations. Data are stored using a concept dictionary allowing flexibility in data capture and translation to other languages [12]. This approach also supports a range of standards for data storage and exchange with mappings available for a range of coding standards such as the International Classification of Diseases, 10th Revision, and Logical Observation Identifiers Names and Codes in the master Columbia International eHealth Laboratory concept dictionary.
Adapting OpenMRS to new uses typically requires technical expertise including Java programming if new modules are required. There were limitations to the older user interface used in this project (which has now been superseded), requiring care in developing clinical workflows. OpenMRS has been adapted to support a wide range of care including HIV, multidrug-resistant tuberculosis, primary care, emergency care, heart disease, oncology, and surgery. A Server Monitoring Tool module was developed to track system uptime and downtime, daily data entry rates, and completeness of key variables. The Server Monitoring Tool was used as part of the larger evaluation study in Rwanda.
OpenMRS was developed by a collaboration among the Academic Model Providing Access to Healthcare project in Kenya [13]. Before and during deployment, OpenMRS had to be customized to support the Rwanda MoH requirements. A dedicated 9-month course led by Partners In Health/Inshuti Mu Buzima trained programmers in enterprise Java and health information system design [14]. Several graduates were hired by the MoH and created custom OpenMRS modules for HIV and primary care using OpenMRS version 1.6 core code. This is the version of OpenMRS used in the control sites for this study. Unstable internet connectivity in rural Rwanda (similar to many low-income countries) required each site to run its own instance on a local server, requiring stable power and local technical support.

Impact of EHR Systems in Resource-Limited Countries
Over the last two decades, EHR systems have been implemented in a wide range of countries, including those with the lowest income levels. The scale-up of HIV care and transition from an emergency outbreak response to a lifelong chronic care model was a major driver for the expansion of EHR system use and the development of common, shared information system tools. Countries, including Rwanda, Kenya, Uganda, Mozambique, and Nigeria, have scaled up the use of OpenMRS EHR systems for HIV care to hundreds of their clinical sites. Other EHRs, including IQcare, have been widely used in countries such as Kenya [15].
The OpenMRS community has prioritized support for effective and safe clinical care as well as reporting and research. Smaller-scale studies have evaluated the impact of EHR system improvements on the aspects of clinical care, the systems efficacy. Were et al [16] studied the addition of alerts to printed patient summaries generated by OpenMRS on a range of clinical actions for the care of children who were HIV positive in Eldoret, Kenya. In a randomized controlled trial (RCT), they showed that health care workers receiving the summaries with alerts were 4 times more likely to carry out actions such as ordering CD4 counts (a T lymphocytes test) and polymerase chain reaction tests for HIV antigen. In a larger study, Oluoch et al [17] studied the impact of improved decision support tools implemented in an EHR in Kenya on the quality of HIV care. In a cluster RCT of 13 health centers and 41,062 patients, they showed that sites with the decision support tools were quicker and more effective in responding to HIV treatment failure [17]. Critical questions remain regarding the key factors that determine individual EHR use, facilitate scaling up to tens or hundreds of smaller health facilities, support long-term use, and influence the clinical impact of these systems in routine care-the effectiveness of EHRs in LMICs [18].

Overview
The aims of this study are to evaluate the following questions in a large number of health centers in Rwanda: (1) staff and stakeholder expectations and perceptions of health information system performance; (2) staff and stakeholder expectations and perceptions around effort expended to use health information systems; (3) infrastructural, organizational, and individual conditions that are barriers and facilitators to using such tools (including training and technical support); (4) staff perceptions of technology fatigue; and (5) any differences in the experiences of staff in intervention and control sites and between clinical and technical users.

The EHR Implementation Science Study
The focus of this manuscript is the electronic medical record (EMR) user survey component of a process evaluation, which is part of a larger, 3-part implementation science study on the use of an enhanced EHR to support HIV care in 56 randomly allocated health centers that commenced in July 2018. It included the evaluation of (1) EHR use, performance, and data quality; (2) the clinical impact in an RCT; and (3) the cost of development and implementation of the enhanced EHR functionality.
For enrollment in the overall study, first, the enhanced EHR package (Textbox 2) was piloted in 2 health centers in Kigali (Kicukiro Health Centre and Kagugu Health Centre), and improvements were made in response to the user experience and comments. Next, the following selection criteria were applied: (1) the presence of ≥3 computers, 1 printer, and a local area network; (2) active HIV case numbers between 50 and 700; and (3) successful installation of the Server Monitoring Tool (Textbox 1) and evidence of regular data entry by staff. Using these criteria, a total of 112 sites were selected to participate in the clustered RCT. These sites were a mix of urban and rural health centers and some district hospitals. Of the 112 sites, 56 (50%) were randomized into the intervention sites, which had the enhanced EHR installed on the servers between June 25 and July 5, 2018. All 56 sites had the alerts for delayed patient enrollment, 28 sites also had alerts for delayed viral load testing, and 14 had the alerts for evidence of treatment failure. For the analysis of the survey, sites with at least the top-level alerts (delayed HIV care registration) were classed as intervention. Health facility staff, including clinicians, data managers, local information technology (IT) staff, local clinic managers, and district IT specialists, in all 112 study sites were trained on general EHR use and data management. Additional training was provided for staff in the intervention sites on the enhanced EHR and equivalent training on the control EHR. Textbox 2. The enhanced electronic health record (EHR) package.

The enhanced EHR package enhancements
• Upgraded OpenMRS software version (to v.1.11) and additions to the concept dictionary • Improved workflow for registering and managing patients with HIV • Improved ordering of laboratory analyses (HIV tests, CD4 counts, and viral loads) • Upgraded clinician summaries of patients showing key clinical data and alerts and reminders designed to improve care • Custom automatic reports to identify patients not receiving optimal care, implementing the same alerts and reminders • Alerts and reports designed to identify patients with care delivery issues. These were chosen to reflect the needs identified by the Rwandan Ministry of Health and based on the 2016 World Health Organization guidelines for HIV care (WHO Consolidated Guidelines HIV 2016 [19]) and included the following: • Newly diagnosed patients with HIV who have not been enrolled in antiretroviral therapy within 2 weeks of diagnosis

Study Environment
The user survey was conducted at primary health care facilities, referred to here as health centers, offering HIV treatment services, located throughout Rwanda, approximately 5 months after the installation of the enhanced EHR.

Study Design
This study used a cross-sectional, key informant structured interview design within control and intervention sites. The data were collected through structured interviews to ensure high response rates and avoid technical limitations that may have impacted a web-based survey and biased results toward better-supported sites and users. The goal is to gain insights into the adoption, functionality, use, and perceptions of EHRs by clinical staff (nurses, physicians, and social workers) and technical staff (IT staff, data entry staff, and data managers) in health centers. Care of patients in smaller health facilities in East Africa, including those with HIV, is mostly carried out by nurses or junior clinician grades and rarely by physicians. The study questions were as follows: (1) whether the actions and perceptions of staff using the enhanced EHR intervention would be different from those using the control EHR and (2) whether clinicians have different experiences with the EHR than technical staff.

Sampling and Sample Size
This study drew from the sample frame of the clustered RCT implementation study. The RCT enrolled 112 health centers from >300 that use the OpenMRS EHR for HIV care. Of the 112 sites, 54 (48.2%) were randomly selected, including 27 (50%) from the enhanced EHR sites (intervention) and 27 (50%) control sites (Figure 1). Randomization was performed with R.
A total of 100 participants were approached for the structured interview, with the goal of 1 clinician (nurse or physician) and 1 data manager at each health center. If not available, other EHR users were recruited if possible.

Data Collection Tool
The structured interview and observation (survey) tool included sections on demographics, experience with IT, EHR training received, frequency of EHR use for different tasks, overall ease of use, usefulness for specific tasks, technical and user support, and system stability and infrastructure issues. The survey tool included 5-point Likert scale quantitative close-ended questions and qualitative open-ended questions. It was adapted from a form originally used by Médecins Sans Frontières, piloted in 20 clinics in Rwanda in 2012 [13], and translated from English into Kinyarwanda.

Data Collection
The survey was conducted by 10 trained data collectors from the Rwanda School of Public Health. Survey responses were documented and recorded in a preprogrammed Android tablet using ODK [20]. Free-text comments were documented in Kinyarwanda, translated into English, and reviewed by bilingual research team members before analysis. Written informed consent was obtained, and participants' confidentiality was assured using a private interview room at each health center surveyed.

Data Analysis
Descriptive statistics were carried out using Excel (Microsoft Corp). JMP Statistical Software (SAS Institute) and Excel were used for the chi-square tests for the 5-point Likert scale responses. For the comparison of clinicians and technical users, all Likert scale questions were tested for significance. For the comparison of the intervention and control sites, the 2 groups of questions (12 and 18) most directly related to the technical improvements in the enhanced EHR were tested. P values were adjusted for multiple comparisons using the Benjamini and Hochberg method and R p.adjust [21]. Analyses were designed and carried out with assistance from a statistician and a data scientist at Brown University (see Acknowledgments).
Free-text comments, which were all short statements, were analyzed thematically using inductive coding by one author (HSF) and recoding by a second author (MM), with discrepancies resolved by discussion.

General Use of Technology
A large majority of respondents used mobile phones, with 82% (74/90) using them "about half the time," "most of the time," or "all of the time" for texting, and 82% (74/90) similarly for mobile data. Computer use outside of work was reported by 31% (28/90) and internet use by 49% (44/90). Clinicians (physicians, nurses, and social workers) reported significantly less use than technical staff (IT officers, data managers, and data entry staff; P=.009 and P=.04, respectively).

Training on EHR
Respondents agreed or strongly agreed that their training on the EHR was effective (82/84, 98%), and they were confident in using the EHR (81/87, 93%). However, 77% (66/86) of respondents disagreed or strongly disagreed with the statement "I am generally not concerned making errors in EHR." There were no statistically significant differences in responses on training between clinicians and technical staff. However, in free-text comments, 81% (73/90) of respondents requested more training. These requests included refreshers, training on new modules or updates, and more practical hands-on training. There were also requests for training in reports and data analysis. Mentorship, supportive supervision, or more technical backup were requested by many respondents.

Use of EHR Functions
Tables  (14) 13 (14) Access the internet to check email, go to websites, or any other internet activities       Staff in intervention sites were significantly more likely to use the EHR for "Updating existing patient records" (P=.04), "Generating patient summaries before visits" (P<.001), "Viewing laboratory results" (P=.04), and "Generating clinician summaries" (ie, on-screen summaries; P=.01). Clinician responses indicated that they carried out the following tasks significantly less frequently than technical staff: "Creating new patient records" (P=.02) and "Updating existing patient records" (P=.04).
A total of 42% (37/89) of respondents stated that they used the EHR always/almost always or most of the time, as opposed to 76% (68/89) for the paper records. They agreed or strongly agreed >85% (71/84) of the time (Tables 1-7) with the following statements about the EMR: "I can find where to document care," "The screen displays are easy to read," "Content lay out is understandable," and "It is easy to retrieve records in EHR." For the statement "It is easy to correct errors in EHR," agreement was 72% (64/89).
Respondents agreed or strongly agreed >90% (81/90) of the time that "the EHR makes it easier to manage patients' medical file and patient's medical follow up," "the EHR makes it easier to make informed decisions," "the EHR is worth the time and energy to use," and "quality of information has improved due to the EHR." For the statement "the EHR makes it easier to exchange patient information with other health care providers," agreement was 71% (64/90). For the statement "the EHR provides useful alerts and reminders," agreement was 78% (64/82) with significantly stronger agreement in the intervention sites (P=.01).
Answers to questions on technical and user support received mixed responses. Respondents agreed or strongly agreed with these questions with the following scores: "It is easy to report problems with the EHR," 71% (63/89); "I get feedback when I report errors or problems," 60% (53/89); "Effective help is available with the EHR," 55% (49/89); "I use EHR because of the proportion of coworkers who use it," 41% (35/86); "My supervisor is very supportive of EHR use on the job," 75% (67/89); and "In general, the MOH supported the use of EHR," 93% (84/90).

Principal Findings
Overall, the results suggest that most users of OpenMRS at Rwanda MoH health centers perceive the EHR as a valuable tool for patient care and reporting activities. The responses showed a high level of EHR use and acceptability across most health centers despite the challenges of implementing EHR systems in these environments. This finding provides foundational evidence to implementers who have an urgent need to understand how well EHRs can be scaled up to hundreds or thousands of health facilities (addressing objectives on performance and scalability). An unusual feature of the Rwanda OpenMRS implementation is the long interval since the original deployment. Some MoH health centers have used the EHR continuously for 8 or 9 years, with no major upgrades in control sites for >5 years. Therefore, this study allows assessment of the long-term performance of the EHR by typical users (objective on sustainability). Such data are not available from any existing studies that we are aware of, which have mostly focused on larger hospitals in more controlled settings with better infrastructure [11,22] or small numbers of test sites.
Responses to the 2 groups of questions most relevant to the features of the enhanced EHR package showed, in the intervention sites, more frequent use of core clinical tools, including updating records, using patient summaries, and viewing laboratory results. Significantly more respondents in the intervention sites agreed that "The electronic medical record provides useful alerts and reminders," indicating support for more advanced EHR features added in the enhanced EHR.
There were some differences in the level of EHR use between clinicians and technical staff, including core clinical activities such as creating and updating records. Clinicians, as expected, had less technical experience and were significantly less likely to use computers outside work or access the internet for a range of applications. These findings indicate the need for further improvements in usability and workflow and in both IT and EHR training for clinicians.
It is important to note that recent versions of OpenMRS have greatly improved user interfaces and general functionality [23,24] and are expected to have significantly higher scores for usability and overall satisfaction. An up-to-date version of OpenMRS was implemented in Rwandan district hospitals in 2020/2021.

Limitations
This survey was conducted through structured interviews with all participants. The less confidential nature of interviews compared with a web-based survey may have increased desirability bias as staff were aware that the study was endorsed by the MoH. There was a strong positive response on the question of MoH support and on statements that the effort to enter data and use the EHR was worthwhile. However, on other questions such as infrastructure, including power and internet connectivity, and availability of technical support, participants were more mixed in their responses, and for the question "I am generally not concerned making errors in EHR" they were clearly prepared to admit that there were problems. Many made clear that they had challenges with using the EHR, and clinicians would appear to rely on data managers and other technical staff to assist with many activities. Free-text comments provided critical insights into the actual experiences of staff, along with many other issues related to usability, use, and the need for training. The lack of significant differences in the experiences of the clinicians and technical staff regarding many questions may be partly due to the survey not being powered to show small differences between these groups. Another limitation was that the 112 sites selected for the broader study had better hardware and evidence of more consistent data entry than the others; therefore, EMR implementations described here may perform better than the full set of EMR sites in Rwanda.

Comparison With Previous Work
Previous studies of EHR users in LMICs have identified a range of experiences. Ojo [25] used the Delone McLean Information Success Model in a study of EHR users in hospitals in Nigeria and showed that system quality and use were the most important in determining EHR success [25]. Tilahun and Fritz [26] conducted a similar study on the experience of users with an EHR in hospitals in Ethiopia. Compared with the survey in this study, they showed high levels of dissatisfaction with the EHR and low use levels owing to poor service quality (power infrastructure, user support, training, and lack of computers in the wards) and the need for double entry of data into the EHR and paper records (also a problem in Rwanda) [26]. A survey of the OpenClinic EHR users at the Kigali University Teaching Hospital in Rwanda showed strongly positive user comments on satisfaction and perception of data quality and usability compared with paper records [11].

Conclusions
This survey provides evidence that EHR systems have become an accepted component of HIV care delivery in Rwanda. Staff were generally supportive of the system, although most wanted further training, technical support, and better power and network infrastructure. Staff at intervention sites were more likely to use or have positive experiences of key functionality that was improved in the enhanced EHR. As this survey is part of a larger evaluation study, the responses will be compared with results from key informant interviews, the costing and data quality studies, monitoring of server performance and use, and clinical impact in the cluster RCT. Further surveys are planned for other large-scale rollouts of OpenMRS in low-income settings, building on the survey form and findings in this study. The results are likely to be generalized to similar EHR systems in low-income settings if they are well tailored to the clinical needs and workflow. They are also highly relevant to the critical need for systems to support accurate, timely, and analyzable primary care data on patients in remote and very underserved clinics in low-income countries, replacing basic tools such as paper registers. This should improve the clinical documentation, care, reporting, and tracking of disease outbreaks, including COVID-19.

Data Availability
The data underlying this paper cannot be shared publicly because of the need for privacy of the individuals who participated in the study. The data will be shared upon reasonable request with the corresponding author.