Looking Behind the Curtain: Identifying Factors Contributing to Changes on Care Outcomes During a Large Commercial EHR Implementation

Objective: To identify factors contributing to changes on quality, productivity, and safety outcomes during a large commercial electronic health record (EHR) implementation and to guide future research. Methods: We conducted a mixed-methods study assessing the impact of a commercial EHR implementation. The method consisted of a quantitative longitudinal evaluation followed by qualitative semi-structured, in-depth interviews with clinical employees from the same implementation. Fourteen interviews were recorded and transcribed. Three authors independently coded interview narratives and via consensus identified factors contributing to changes on 15 outcomes of quality, productivity, and safety. Results: We identified 14 factors that potentially affected the outcomes previously monitored. Our findings demonstrate that several factors related to the implementation (e.g., incomplete data migration), partially related (e.g., intentional decrease in volume of work), and not related (e.g., health insurance changes) may affect outcomes in different ways. Discussion: This is the first study to investigate factors contributing to changes on a broad set of quality, productivity, and safety outcomes during an EHR implementation guided by the results of a large longitudinal evaluation. The diversity of factors identified indicates that the need for organizational adaptation to take full advantage of new technologies is as important for health care as it is for other services sectors. Conclusions: We recommend continuous identification and monitoring of these factors in future evaluations to hopefully increase our understanding of the full impact of health information technology interventions.


Introduction
Electronic health record (EHR) systems adoption in the United States has increased to rates never observed before [1], and, as a result, the literature evaluating the impact of health information technology (HIT) interventions on quality, productivity, and safety outcomes has also increased [2]. However, previous studies have produced mixed results, leaving unanswered questions as to the impact of HIT interventions [2]. The lack of consistent evidence has been attributed primarily to insufficient descriptions of study settings and interventions; the use of a narrow set of study-specific measurements; and weak research designs that do not consider the longitudinal effects introduced by HIT interventions [2]. Previous studies suggest that HIT is seldom objectively measured, and that claims regarding its efficacy are often based on self-promotion and not on scientific evidence [3]. It has been estimated that without improved research methods, around 100 hypotheses per year will continue to be tested without providing any valuable knowledge [4].
Studies from other service sectors such as retail, transportation, and finance, demonstrate that information technology (IT) adoption tends to produce positive outcomes only when accompanied by complementary changes or investments (e.g., proper training, upgrading IT infrastructure, adapting workflows) necessary to take full advantage of new technologies [5]. Such factors have not been explored in evaluations of IT adoption in the health care industry and deserve further attention from the broader medical and informatics communities [6].
In previous work, we developed a systematic methodology to detect near real-time performance changes during EHR implementations using a large set of measures identified in the literature [7] and suggested by subject-matter experts [8]. The method was tested in a large commercial EHR implementation involving 4 medium-size hospitals and 39 clinics from a large care delivery system [9]. While the proposed methodology was able to effectively detect what and to what extent changes happened, it was not designed to elucidate the dynamics surrounding how they happened. The objective of the present study is to identify factors that may have contributed to changes detected on quality, productivity, and safety outcomes during a large commercial EHR implementation in order to increase our understanding of the full impact of HIT interventions and to guide future research. To elicit those factors, we augmented our quantitative findings with semi-structured, in-depth interviews with clinical employees from 1 medium-size hospital and 10 clinics from the larger implementation previously monitored [9].

Description of the previous longitudinal evaluation
Intermountain Healthcare, a not-for-profit, integrated care delivery system of 23 hospitals and over 185 clinics covering Utah and southern Idaho, has completed the replacement of a set of homegrown legacy systems [10][11] with the commercial Millennium EHR (Cerner Corporation, Kansas City, MO, USA). The Cerner EHR implementation used a phased approach with the introduction of the new EHR across 10 geographical regions at different points in time. The implementation in each region followed a "big bang" strategy, replacing all legacy systems at once within that region. We have conducted a longitudinal evaluation of the implementation in the first five regions using an interrupted timeseries design with parallel control sites [9]. We collected monthly data from February 2013 to July 2017 for 41 outcomes including quality (11 measures), productivity (20 measures), and safety (10 measures), selected from an inventory of outcomes likely impacted by HIT interventions [8]. Data needed to calculate the outcomes were collected from Intermountain's enterprise data warehouse (EDW) containing EHR data. Data became available in the EDW after the EDW team ensured that there were no systematic differences in the data generated by the EHRs (legacy vs. Cerner). Data were analyzed using an ordinary least squares model [12] that assessed whether the outcomes monitored were impacted immediately after the introduction of the implementation (i.e., EHR "go live") and compared the average change per month in the outcome before and after the go live. Table 1 lists the quantitative outcomes from our previous evaluation [9] that represent the quantitative phase of the present mixed-methods investigation.

Design and settings
We conducted a mixed-methods study with a sequential explanatory design [13]. The design integrates interpretation of the quantitative results of the longitudinal evaluation previously reported [9] with in-depth, semi-structured interviews with clinical leaders and staff from 1 hospital (375 beds) and 10 primary care clinics from one of the most recent implementation regions (region 4) [9]. We selected this region to prevent recall bias and at the same time give enough time for participants to be exposed to the new system. In this region, data were collected monthly for two years before the go live (April 2016), followed by a 16-month post-intervention period that ended when control sites went live (July 2017). Intermountain Healthcare's Institutional Review Board approved the study.

Procedure
We selected all outcomes from the previous quantitative evaluation that detected a statistically significant change after the implementation in the targeted settings [9] (Table 1), and invited clinical leaders from the departments that represent these outcomes (e.g., for emergency department (ED) measures we invited ED leaders) to participate in an indepth, semi structured interview. The goal of the interviews was to identify factors that may have contributed to changes detected on the outcomes in question. The interviews were divided into three steps that lasted from 30 to 60 minutes: (1) presentation of outcomes from the quantitative study (see example in Figure 1 below and other included measures in the Supplemental Content, Figures S1 to S15); (2) open-ended questions to elicit potential contributing factors and covariates (see Supplemental Content, Table S1); and (3) request for referral to other interviewees.
In the first step, we provided a brief explanation of the overall objective of the interview to ensure informants conceptualize "factors" consistently (i.e., changes to processes, procedures, or resources that could have affected -positively or negatively -the outcomes discussed). Then we presented graphical data from the quantitative study to stimulate the discussion. In the second step, once potential factors were identified, informants were asked to suggest data available in electronic format that could serve as covariates in future similar evaluations. Interviews were conducted in person from July 31, 2017, to September 29, 2017. Interviews were conducted until we were not offered any new referents to interview. We attempted to interview at least two employees for each measure in order to obtain different perspectives on the same outcome/process.

Data Analysis
We conducted a systematic content analysis of the interview narratives based on Srnka et al.'s guidelines for analyzing qualitative data to derive new theory [14]. The analysis was conducted in 6 stages: Stages 1 and 2: Recording and transcription. The audio recordings from the interviews were transcribed and de-identified. Stage 3: Unitization. Transcriptions were split into units that represent informants' responses about each outcome discussed. Stage 4: Coding of contributing factors. Three authors (TKC, DB, VDC) with distinct backgrounds (informatics, medicine, nursing) independently coded relevant responses that suggest potential causes of the changes on each outcome. We initially attempted to use a combined deductive-inductive approach as suggested by Srnka et al. [14], with the sociotechnical dimensions of HIT impact proposed by Sittig and Singh [15], but found that they did not provide enough granularity and depth of the potential factors reported by the informants. We then adopted an inductive approach with each coding author independently identifying categories that explain the changes for each outcome. Multiple coding sessions were conducted. In each session, the authors collaboratively reviewed initial codes and merged them into a redefined category through consensus. The resulting codes were used in the subsequent iterations. Once all transcripts were coded, similar categories were merged based on consensus producing a final list of factors that may have contributed to the changes on each outcome. Stage 5: Coding of covariates. The same steps in Stage 4 were followed for the identification of covariates to quantitatively monitor the contributing factors identified. Stage 6: Identification of factors related to new EHR implementation. Once factors were identified using the coding scheme in Stage 4, the three coding authors had a final session to collaboratively reach consensus about the classification of the factors according to their relationship with the EHR implementation: directly related, partially related, and not related.
Coding of the investigation narratives was done with Atlas.ti Version 8.0.

Results
We interviewed 14 clinical leaders and staff who reported 14 factors that they feel may have contributed to the changes detected on the outcomes. Overall, all informants confirmed that the quantitative results presented matched their perceptions regarding the effect of the implementation. Table 2 summarizes interviewees' characteristics and Table 3 lists the factors identified. A description of each factor is given below along with example verbatim interview quotations. We also identified 17 covariates with data available in electronic format to quantitatively measure 12 of the 14 factors identified ( Table 4).

Factors directly related to the EHR implementation
Nine factors related to the EHR implementation were reported by informants.

Decrease in communication
ED leaders reported that due to the increased time spent on electronic documentation, communication between nurses and physicians decreased and interruptions increased, potentially impacting length of stay (LOS) and wait time: "Communication decreased while interruption increased, massively. Our doctors were hiding in the physician lounge." Informants also perceived that both the change in communication and in the outcomes were experienced, and expected. They reported that they had sent their own nurses to support previous go lives and had observed that most of the communication between physicians and nurses became heavily dependent on the EHR, decreasing clinicians' efficiency. No specific covariate was identified for monitoring this factor.

Figure 1:
Example graph illustrating monthly length of stay in the ED with a significant increase at the intervention hospital immediately after the go live followed by recovery to the baseline within 12 months.  A primary care provider reported that due to a partial data migration from the legacy to the new EHR, some clinical decision support (CDS) alerts were inaccurate, potentially decreasing volume of laboratory orders: "I see a lot of overdue stuff. I don't know if it's overdue, so it doesn't get ordered." Acceptance rate of CDS alerts could be a covariate potentially affecting laboratory orders.

Increase in staff
Primary care providers hired new personnel to help with electronic documentation in order to recover the volume of patient visits at baseline: "Some physicians employed scribes." ED leaders increased their nursing staff to mitigate problems in LOS and wait time: "We hired 12 more nurses over the preceding months." No specific covariate was identified for monitoring this factor.

Learning curve
The need to allow time for clinicians to be fluent with the new system hampered their efficiency in the ED potentially contributing to longer stays and wait time, as reported by an ED manager: "Nurses became efficient with their [legacy] program with time, so you have to give people time." Primary care providers also reported that their practices were less efficient, which may have affected their volume of patient visits: "The issue is people are learning how to use the system. It's not only the physician. It's also the front desk and nursing staff." According to informants, both the increase in ED measures and decrease in patient volume were clearly perceived during the implementation. We asked participants if they believed that lack of training and/or go live support contributed to these changes. They informed that despite having appropriate training resources available, clinicians felt that they only learned the new system in vivo, and that they needed more support from "technology champions": "Those resources have been deployed to help with go lives in other regions." The number of people allocated for go live support can be a covariate and/or a moderator since it may hamper clinicians' efficiency after the go live, potentially contributing to longer LOS and wait time, and lower volume of visits.

Missing functionality
A primary care provider reported that the new EHR missed a key functionality available in the legacy system used in cases when blood pressure was temporarily high, but did not demand treatment changes. The lack of this functionality could have contributed to an artificial decrease in blood pressure control: "I don't have clinical judgement. Now it's just the number so if they [nurses] don't do a blood pressure clinically perfect it's going to be high." The informant suggested monitoring documentation of acute illness and changes to hypertension treatment as covariates for blood pressure control.

Redistribution of staff or work
Primary care staff started to orient patients to arrive earlier as an attempt to recover to normal levels of patient visits, as reported by a primary care provider: "We call them and say, 'You need to make sure you are 10 or 15 minutes before your appointment'." An infectious disease specialist reported that they had to redistribute preventive tasks in order to investigate an increased volume of potential surgical site infections (SSIs) captured by the new EHR: "We had to send out other tasks", which potentially increased the number of SSIs detected. ED managers reported that a difference of clinician-patient ratio between nursing and physician staffs was the most significant factor contributing to longer LOS and wait time: "They [physicians] didn't change their patient ratios even though they were massively increasing their workload." ED Informants suggested monitoring provider-patient ratio as a covariate potentially affecting LOS and wait time.

Resistance to learn or use a new EHR
Intensive care unit (ICU) nurses reported multiple examples of colleagues who demonstrated a resistance to learn and use the new EHR, potentially increasing employee turnover: "They said, 'the day the system goes live, I quit'." This resistance was perceived as more likely to affect older employees: "It seemed to be harder on older people." Management had already identified this resistance before the implementation had started. When asked if they believed the lack of training and/or go live support could have impacted this outcome, an ICU manager reported that ICU management tried to implement diverse training strategies, but were still unsuccessful: "They didn't want to learn a new system." Informants suggested tracking employee age as a covariate potentially affecting employee turnover.

Workarounds
Reactive workflow changes affected multiple outcomes in both types of settings. Two primary care providers reported that they were not able to recheck blood pressure in some cases and started using nurses' triage measurements, which potentially decreased blood pressure control rate: "Because the log-in process was so painful, people were not rechecking blood pressures at the time." Due to the implementation of computerized provider order entry (CPOE), nursing staff had to wait for physicians to enter laboratory orders before collecting laboratory samples, which may have decreased the number of laboratory orders: "Now they [nurses] need us [physicians] to sign off before it gets done." A primary care director reported that providers were oriented to document as much as possible at the time of the visit to avoid after hours documentation: "We talked to the physicians to get the documentation done at the time of the visit." However, in most cases physicians were not able to follow the orientation and had to redesign practice workflow and schedule: "On Tuesday, I stop seeing patients at 11:30 and chart the ones from Monday until 5 o'clock." Primary care informants reported that increased volume of visits may increase documentation and suggested monitoring patient visits as a covariate for time documenting after hours.

Factors partially related to the EHR implementation
Two factors partially related to the EHR implementation were reported by informants.

Change in care pathways
A cardiovascular director reported that changes to care pathways were already being applied before the implementation, and that the new system facilitated this process potentially decreasing the rate of readmissions for heart failure patients: "Our team was updating our protocols to improve these data; we added [order sets] for admissions." Informants suggested monitoring appropriate use of medication for heart failure as a covariate for readmission rate.
Intentional decrease in volume of work A primary care director reported that primary care providers were oriented to limit their schedules after the go live: "Clinics had their schedules limited in a way that would allow us to have time to deal with the new system." This orientation may have decreased patient visits and laboratory orders, as reported by a primary care provider: "You have a drop in volume, so labs would probably go down." Informants suggested monitoring the number of patient visits as a covariate for laboratory orders.

Factors not related to the EHR implementation
Three factors not related to the EHR implementation were reported by informants.

Health insurance changes
Primary care providers reported that new requirements for coding procedures increased time documenting after hours: "We didn't have a focus on trying to capture every single diagnosis for Medicare before." One primary care provider reported that insurance companies progressively removed coverage of tests ordered in physical examinations: "The insurance change was a push back on physicians to kind of change our behavior", which potentially decreased laboratory orders. He also reported that patients are more frequently opting for health savings accounts; such patients tend to avoid chronic disease management visits, which decreased compliance to diabetes bundle and patient visits: "People don't come frequently for their diabetes control because it's out of their pocket." He also reported that their top health insurance lost a contract close to the go live, which may have decreased the number of new patient visits: "A contract with [company name hidden] was supposed to come to us but it went to [company name hidden]." Informants suggested monitoring risk adjustment factor as a covariate for time documenting after hours; type of health insurance as a covariate for diabetes bundle and patient visits; rate of laboratory tests covered for physical exams as a covariate for laboratory orders; and rate of patients per top health insurance as a covariate for new patient visits.

Patient Engagement
According to one primary care provider, two diabetes bundle items, hemoglobin A1c and eye exam, depend on patient engagement: "They [patients] have to go to an ophthalmologist." He also reported: "He [patient] is working in two jobs, eating out constantly, so his A1c is 11 now." The provider suggested monitoring each bundle item in isolation.

Seasonal pattern
The implementation happened in a period of increased ED visits: "This is seasonal… it wasn't related to the new EHR." The increased visits may have affected LOS and wait time: "The volume itself will affect length of stay and wait time." ED leaders suggested monitoring the number of ED visits as a covariate for LOS and wait time.

Discussion
To the best of our knowledge, this is the first study to investigate factors contributing to changes on a broad set of quality, productivity, and safety outcomes during an EHR implementation guided by the results of a large longitudinal evaluation. Although previous studies attempted to identify factors contributing to changes introduced by HIT adoption, they have focused on specific functionality such as CPOE [16] or specific outcomes such as medication errors [17]. The diversity of factors identified indicates that the need for organizational adaptation to take full advantage of new technologies is as important for health care as it is for other services sectors. Our findings lend support to the need for more robust HIT evaluations that consider the impact of contributing factors. Hospital outcomes were more consistently affected by factors related to the new EHR implementation. Several factors affected ED outcomes; however, our qualitative analysis revealed that the lack of go live support intensified and expanded clinicians' learning curve and may be the most plausible explanation for longer stays and wait time.
Although an increase in nursing staff led to decreased patient ratios, the ED was still less efficient overall because ED physicians faced a significant change moving from paper-based to electronic ordering. Such inefficiency following CPOE implementation has been extensively reported by previous studies [18][19]. Most informants reported that appropriate training resources were available, but perceived that effective learning happens only with the use of the new system in the clinical environment. They suggested that additional support from "technology champions" after go live was needed. This learning curve could have been controlled with proper planning of go live support and anticipation of usability problems. Although employee turnover has been rated by subject-matter experts as the least relevant measure for assessing EHR implementations [8], our findings indicate that employees may resist learning and using a new EHR and potentially quit their jobs or advance their retirement. Such resistance could have been anticipated on an organizational level with the use of validated instruments for measuring acceptance of new technologies [20]. Surgical site infections increased after the go live mostly due to the EHR's increased rate of detection of potential infection cases to investigate; however, this increase in detection was observed only after the functionality was improved, which was not anticipated and happened while the system was already operational. MRSA and CDiff infections may have decreased likely due to a system configuration that prospectively increased the number of patients in isolation by requiring providers to complete isolation orders generated automatically. Primary care informants indicated that a key functionality was not available in the new EHR and felt that they lost clinical judgement to document when patients were hypertensive. For example, patients may have a temporary hypertension caused by an acute illness, which would not necessarily characterize them as "uncontrolled." The legacy system provided a functionality to allow clinicians to bypass a blood pressure measurement and consider the patient "in control"; in the new system, such functionality was not available. Identification of missing functionality could have been controlled by stakeholders with enhanced involvement of end-users in the design and customization of the new EHR, as recommended by experts in the field [18], but frequently ignored in similar interventions [21].
Ambulatory outcomes were more consistently affected by factors not related to the new EHR implementation. The constant changes to insurance coverage and billing documentation may have decreased the volume of patient visits and laboratory orders, and, in spite of that, added an enormous documentation burden. Other studies found that clinical documentation in the United States exceeds by large amounts similar documentation in other developed countries, and one of the potential reasons is the extensive billing requirements in the United States [22]. In our previous study, time documenting after hours in the new EHR ranged from 0.8 to 2.3 hours per provider per month [9]. The same outcome has been reported elsewhere as 1.4 hours per provider per weekday [23]. Our qualitative analysis found that providers frequently blocked periods of their schedule to document previous visits during work hours, such a documentation was not captured as "after hours" by our measurements, which may explain the smaller times observed at the study sites. Although insurance changes are not controlled by stakeholders, early involvement of end-users and allocation of "technology champions" for go live support are processes that can be internally controlled and could have mitigated the documentation burden. Providers suggested that a decrease in compliance with the diabetes bundle is more likely to have been affected by a decrease in chronic disease management visits, which results from an increased use of health savings accounts.
Several quality and safety measures did not recover to the baseline levels observed before the implementation. This is partially explained by the fact that these measures are more frequently affected by factors not entirely under the control of stakeholders, such as patient engagement or health insurance changes; however, there are exceptions such as identification of missing functionality only after the system was operational. Productivity outcomes frequently recovered to the baseline levels within 1 to 12 months. These measures are more likely to be affected by factors controlled by stakeholders such as redistribution of tasks, workarounds, and intentional decrease in volume of work. These factors were frequently reported by informants as areas in which adjustments were made to facilitate recovery.

Implications for future research and EHR implementations
Our findings demonstrate that several preventive actions potentially under the control of stakeholders are important to manage not only during the active phase of EHR implementations, but also continuously thereafter as a part of the organization's quality and safety efforts. We recommend more attention to preventive actions such as allocation of "technology champions" after the go live, since users seem to learn by using the system in real clinical scenarios. We also recommend more attention to the identification of missing functionality, proactive workflow redesign, and identification of deleterious workarounds. An effective strategy to identify areas needing additional go live support, missing functionality or workflow changes, is to simulate the use of the new system in the production environment, as demonstrated elsewhere [24]. Health care leaders must try to anticipate that some employees might resist learning the new EHR and develop strategies to engage these employees as early as possible. Our findings also demonstrate that although some factors may not be under the control of stakeholders, their impact should be considered when planning a new implementation. For example, ideally, an EHR go live, should carefully consider external challenges such as coincidence with an increase in volume of patients due to seasonal variations or anticipated health insurance changes (e.g., implementation of new Medicaid billing requirements). Finally, we recommend a mixed-methods approach in future evaluations including a qualitative analysis guided by longitudinal quantitative evaluations using our previously tested methodology [8][9] and monitoring of covariates. Several covariates identified can be relatively easily added to future similar evaluations as they are mostly dependent on data available in electronic format. Examples include covariates that can potentially be controlled by providers such as volume of patients, provider-patient ratio and go live support personnel, as well as covariates that are not under the control of providers, but can be quantitatively monitored such as occurrence of acute illnesses or seasonal variation of ED visits. Such an approach is necessary to improve the capacity of health care leaders, HIT vendors, and researchers to more effectively monitor EHR implementations and hopefully increase the understanding of the full impact of HIT interventions.

Limitations
The quantitative results presented to informants at the beginning of the interviews may have biased the participants' explanations. We were able to interview only 14 informants from only one implementation region (one hospital and ten clinics), which may have compromised identification of other factors. Nonetheless, we interviewed at least two employees for each measure, and in some cases the only employees specialized in the outcomes in question (e.g., the only two infectious disease specialists), which may have led to the identification of the most prominent factors. In addition, the informants represent a wide range of roles and departments relevant to the measures explored.
Intermountain Healthcare has extensive informatics experience and the perceptions of its employees may differ from employees of other institutions. Some factors and impacts were specific to the EHRs used in the present study, implementation of different EHR systems may identify different factors. Our analysis was limited by the measures available in our previous evaluation. For example, safety measures were not available for primary care settings. Finally, we were not able to identify covariates for two factors reported.

Conclusions
We conducted a mixed-methods analysis of a commercial EHR implementation, integrating the results of a previously reported quantitative evaluation with semi-structured, in-depth interviews with individuals affected by the intervention and identified 14 factors contributing to changes on care outcomes. We also identified 17 covariates for monitoring 12 of these factors. Our findings demonstrate that several factors may affect outcomes in different ways during a commercial EHR implementation and lend support for more robust evaluations that consider the impact of these factors to hopefully increase our understanding of the full impact of HIT interventions.

Additional Files
The additional files for this article can be found as follows: • Figure S1 to Figure S15. Graphs of measures that detected a significant impact in our previous longitudinal evaluation. DOI