INTRODUCTION

In an effort to improve health care quality and decrease costs, physicians and health systems have been encouraged to adopt electronic health records (EHRs).1,2,3 Theoretically, EHRs should increase efficiency by providing rapid access to up-to-date information. However, evidence suggests that EHR implementation results in additional time spent completing documentation as well as decreased accuracy.4,5,6 One possible contributor to EHR inefficiency is physician variation—that is, differences in the content, structure, or location of patient information in the EHR that are a result of how individual physicians use the EHR rather than differences in patients’ clinical status. Commercial EHR systems are commonly designed with substantial optionality to accommodate different preferences for how users record information for an identical patient in the EHR. Although there should be variation in documentation based on a patient’s clinical status, it is potentially problematic if other factors, such as user preferences, drive documentation decisions.

The small number of studies that have examined physician EHR documentation reveal substantial variation, even for basic information such as drug allergies and smoking status.7,8,9,10,11 Ancker and colleagues found that the annual average proportion of encounters that updated a patient’s problem list ranged from 5 to 60% among 112 physicians in a network of federally qualified health centers.12 However, no large-scale studies of physician variation in documentation exist. To the extent that such variation is widespread, it is critical to understand its causes (particularly, causes beyond patient mix), the ways in which it may compromise the quality of care, and the strategies that could effectively minimize any harmful effects.

Primary care providers’ documentation serves as the foundation for care coordination, population health management, referrals, and orders.13,14,15 Therefore, it is critical to assess variation in this setting. We focused on isolating differences across physicians in the same practice because they should be treating a similar patient mix as well as working under the same organizational and geographic conditions. Any remaining variation is thus likely due to physician preferences, which may interfere with care delivery by making information difficult to use for subsequent providers. We used mixed methods to answer the following research questions:

  1. (1)

    For core categories of clinical documentation, which categories, if any, have high variation across primary care physicians in the same practice?

  2. (2)

    What are the perceived causes of such variation in EHR documentation and how, if at all, do primary care physicians and their staffs perceive that variation affects care delivery and outcomes?

  3. (3)

    What strategies could primary care practices use to prevent or mitigate the negative consequences of variation in EHR documentation?

METHODS

Summary

We used a sequential, explanatory, mixed-methods design. We first used data from a national ambulatory EHR vendor to quantify physician-to-physician variation for 15 categories of clinical documentation. Once we identified documentation categories with high variation, we conducted semi-structured interviews with physicians and staff in primary care practices to explore the causes and consequences of such variation as well as to identify strategies to prevent or mitigate negative consequences of variation. The Michigan Institutional Review Board (OHRP IRB Registration Number IRB00000246) approved this study.

Quantitative

Setting and Data

We obtained de-identified EHR log data from a commercial EHR vendor that automatically captures and stores clickstream data when users are logged in to the EHR. We worked with the vendor to aggregate the data to 15 mutually exclusive clinical documentation categories, such that a given click would represent a documentation action in the given category (see Appendix Table 4). For example, if a user entered the patient’s blood pressure, or checked a “reviewed” box after reviewing that information, the data would report action under the clinical documentation category of Collect, Update, and Review Vital Signs. Documentation actions included both structured documentation and unstructured documentation. For example, clicking an option on the pre-populated list of medications was counted under the clinical documentation category of Update Medication List, as was a free-text entry in the medication list. Viewing any part of the record without taking some type of action to add, review, or remove information was not captured.

Every documentation action was tied to a patient visit (“encounter”) ID as well as a user ID. Users included physicians as well as other billing providers, clinical support staff, and administrative staff. User IDs linked each documentation action to the user’s role (physician or staff) and specialty (for physicians only). Each encounter was also linked to the specific practice location with a Practice ID. For organizations that had more than one practice location, there was a Provider Organization ID. For each Practice ID, we also received information on the state where the practice was located.

The data set provided by the EHR vendor included all documentation actions in the 15 categories for each encounter that occurred in June 2012 in all ambulatory primary care practices that had implemented the vendor’s system. We restricted the data set to active primary care providers who had used the EHR for at least 6 months in a practice with 2 or more providers. The final analytic sample included 170,332 encounters by 809 primary care physicians who were located in 237 practices across 27 states.

Measures: Dependent Variables: Documentation Completion Per Physician

We assigned each encounter to the physician who completed documentation for the most clinical documentation categories. We then created 15 binary indicators for each encounter that described whether or not anyone in the practice completed each clinical documentation category (that is, one or more actions captured in clickstream data). This rolls up documentation from other billing providers, clinical support staff, and administrative staff to the physician responsible for the visit to ensure that our measure of variation is not simply picking up differences in division of labor with respect to EHR documentation. We opted to use binary indicators rather than measure the number of documentation actions within each documentation category because it was unclear whether more actions reflected more complete documentation, at least in part because each documentation category had a different number of potential actions. Finally, for each physician, we calculated the proportion of assigned encounters with completed documentation (by that physician or by someone else) for each of the 15 clinical documentation categories over the course of the month (that is, 15 outcome measures per physician). For example, if a patient’s vital signs were documented in 5 of 20 encounters assigned to a physician in the month, the physician’s proportion would be 0.25 for that measure.

Measures: Identifying Variables

We created a categorical variable for physician primary care specialty type: family medicine, internal medicine, OB/GYN, and pediatric medicine. We also created a set of identifiers to capture the nesting of physicians within practices, practices within provider organizations, and provider organizations within states.

Analytic Approach

For each of the 15 documentation categories, we calculated the median and interquartile range of documentation completion across the 809 physicians. Because there were no established benchmarks for high versus low variation, we looked at the magnitude of the interquartile ranges and identified 50% as a natural cutoff differentiating high- and low-variation documentation categories (see Appendix Fig. 1). For the high-variation documentation categories (IQR > 50%), we isolated the amount of variation across physicians in the same practice by measuring the variation accounted for across practices, provider organizations, and states. Specifically, we estimated a multilevel linear regression model with the physician proportion of documentation as the dependent variable, primary care specialty as the independent variable, and random effects variables to capture variation at practice, provider organization, and state levels. We calculated the ratio of remaining variation (that is, variation across physicians in the same practice) and the ratios of explained variation for practice, provider organization, and state levels to total variation. We tested whether these ratios were statistically different from zero by using bootstrapped standard errors. To counteract the problem of multiple comparisons, we applied the Benjamini-Hochberg false discovery rate control procedure.16 We interpreted a ratio statistically different from zero as a significant amount of variation.

Conceptually, this approach relied on the assumption that physicians in the same primary care specialty within the same practice would be treating a set of patients in the observed month with the same distribution of documentation needs.17 To assess the robustness of this assumption, we calculated the proportion of variation across physicians for two low-variation documentation categories to see if the ratio of remaining variation at the physician level was lower than for high-variation documentation categories (see Appendix Table 5).

All quantitative analyses were performed by using Stata version 13.18

Qualitative

Setting and Data

We identified internal or family medicine practices that used a commercial EHR in southern and central Michigan from a list of practices that had worked with the state’s Regional Extension Center to achieve Stage 1 Meaningful Use. We restricted the sample to practices with at least 2 physicians. We intentionally included practices that used EHRs from a range of vendors to maximize the generalizability of the results beyond the vendor that provided data for the quantitative analysis.19 We invited the 51 practices that met these criteria and 10 agreed to participate.

In each participating practice, we conducted face-to-face interviews with at least one physician and one other respondent who regularly used the EHR. Interviews lasted 30 to 90 min. We performed both one-on-one and group interviews, based on respondents’ preferences. All interviews were transcribed. Each respondent received a $75 gift card. Data collection occurred February through May 2016.

Our semi-structured interview guide asked respondents about perceived variation in EHR documentation, factors that caused variation, the effects of such variation, and strategies to manage variation (see Appendix 4). We piloted and refined the interview protocol in a convenience sample of 2 primary care physicians.

Analytic Approach

We developed an a priori code list for qualitative themes.20,21,22 One member of the research team applied these codes to 3 transcripts. Next, 2 other members of the research team independently reviewed the final code list and the coded transcripts to ensure comprehensiveness and consistency. The original member of the research team applied the final codes to the remaining 37 interviews. We uploaded all coded interviews to Atlas.ti23 and used the query function to group interviews by code. We synthesized this information in analytic matrices24 to identify themes that emerged in interviews across multiple practices regarding the prevalence of variable documentation, resultant challenges, and strategies for addressing variation.

RESULTS

Quantitative and Qualitative Samples

For the 237 practices in the quantitative sample, the average number of physicians was 12.6. Practices had used the EHR for over 4 years on average (Table 1). The most common specialty was family medicine (69.1%), followed by internal medicine (18.1%).

Table 1 Descriptive Characteristics of the Quantitative Sample
Full size table

The qualitative sample included 5 independent practices and 5 practices that were part of larger health systems, collectively using five different commercial vendors. We interviewed 40 individuals in varying roles across the 10 practices, ranging from 2 to 6 interviewees per practice (Table 2).

Table 2 Practice Characteristics in the Qualitative Sample
Full size table

High-Variation Categories of Documentation

Ten documentation categories had low variation in the percentage of encounters for which documentation was completed (IQR < 20%), and 5 documentation categories had high variation (IQR > 50%) (See Appendix Table 5 for low-variation categories’ IQRs). The documentation category with the most physician-level variation in the percentage of encounters for which documentation was completed—Updating the Patient’s Problem List—had an IQR of 73.1% (Table 3). Results of the multilevel model showed that 70% of the variation in Updating the Patient’s Problem List was attributable to physician variation within the practice. Physicians were somewhat less variable in their rates of documentation in Reviewing and Discussing Documents (IQR = 50.8%); however, the percentage of variation at the physician level was highest in this category (78.1%) (Table 3). Results from our robustness test for two low-variation documentation categories showed lower magnitude (and variably significant) variation at the physician level (Appendix Table 5).

Table 3 Characteristics of High-Variation Documentation Categories
Full size table

Perceived Drivers of Variation in Documentation

All practices in the qualitative sample reported variation in documentation across physicians. Most respondents attributed variation to idiosyncratic physician choices, facilitated by the multiple options available in the EHR to document each category of information. As respondents noted, different options placed different constraints on documentation, suggesting that users selected an option that had a tolerable set of constraints. One of the more common scenarios related to physicians’ preferences for structured or unstructured documentation. For example, the medical director at one practice, who was also a practicing physician, explained that when documenting the history of present illness the EHR allowed users to choose between a structured template that would generate a note and an unstructured template with a single free-text field: “it really depends on the provider whether they check more boxes or if they type more.”

Respondents identified a number of reasons for variation. The first reason was implementation procedures. Many respondents pointed to a lack of training when they first implemented the EHR. One physician suggested that people developed different documentation behaviors in her practice because their training occurred entirely on video, instead of in person: “the videos move really fast, and people are still asking [questions when the next segment begins]…a lot of the variation really comes from that.” In contrast, respondents from a practice that perceived very little variation in EHR documentation attributed the consistency to clearly articulated documentation procedures learned during implementation. Another common explanation for variation was the differences in how physicians viewed templates. Respondents suggested that physicians who sought to mimic the experience of paper records (because they felt that paper records offered a better structure for documentation) were more likely to use free-text fields instead of structured fields, which led to different documentation styles across physicians in the same practice.

Perceived Effects of Variation

Many respondents perceived variation as having substantial negative effects on the experience of documenting care, such as undertaking redundant documentation when there were multiple places to record comparable information in order to ensure that the information could be found in all potential locations. In practices where users did not take extra time to complete documentation, the consequence was extra effort to search for information after the visit. As one physician noted, although these recurring inefficiencies were “only a few seconds, it adds up.”

A subset of respondents were concerned that variation in documentation, typically when documenting patient problems, interfered with the quality of care. As one physician noted, different preferences for maintaining the problem list created longer lists with “junk” information. “You may not know of something that’s important … if there’s a lot of irrelevant information,” the physician said. “It makes it harder to know what’s a real problem versus what’s transient.” One respondent noted a similar risk of “error via misinformation.” As another physician explained, varied documentation of diagnoses could lead to confusion, which is particularly problematic because “everything is driven by the diagnosis nowadays…. It affects the way you approach the patient…. It can affect everything.” The quality improvement director from another practice noted that the same frustrations physicians experienced when addressing variation in documentation of patient problems diminished patients’ trust in the practice because they felt like their record was out of date.

Strategies to Manage Variation

Discussion of documentation during regularly scheduled staff meetings was the most commonly identified strategy to prevent variation in documentation. Respondents said that having frequent opportunities to discuss EHR documentation was useful to identify documentation strategies that could be adopted uniformly by all users. These meetings, which typically occurred monthly or quarterly, were frequently reinforced by emails about best documentation practices. Several respondents felt these follow-up communications were essential, especially if people worked at multiple practices and used EHRs from multiple vendors.

The second most commonly identified strategy to prevent variation was thorough training during implementation. Respondents believed that a clear articulation of EHR functionalities and the ways that different documentation decisions affected where and how information was displayed could help achieve consensus regarding best documentation practices. One practice’s office manager suggested that training a practice manager ahead of all other practice staff would allow the manager to provide ongoing coaching during implementation to further minimize variation.

CONCLUSION

Discussion

A primary care practice’s ability to leverage an EHR to improve health care delivery and patient outcomes depends upon how its physicians use the EHR to document care. This study is among the first to quantitatively capture the level of variation in clinical documentation across physicians and the first to do so in a large set of practices across the nation. It is also the first to explore EHR users’ perceptions of the causes and effects of variation. After combining a large task-log data set of 170,332 encounters with in-depth qualitative interviews in 10 primary care practices, we found substantial variation in documentation for 5 categories of clinical information which was perceived to result from optionality in the EHR design and varied implementation practices. Our results revealed that such variation jeopardizes the efficient and possibly safe delivery of care.

The 5 high-variation clinical documentation categories that we identified in the EHR task-log data have substantial EHR optionality, such that physician documentation choices can result in variation.15 For example, a Review of Systems is often structured as a component of a clinical note covering the patient’s organ systems, with a focus on the subjective symptoms as perceived by the patient.25 If the elicited information leads to the identification of a problem or diagnosis, that information could be documented in the Review of Systems, the Problem List, the Assessment and Diagnosis, or in all three categories. Our qualitative work reveals how preferences result in variation in this scenario. Allowing physicians to document either in free-text fields or via structured data entry gives the documenting physician more flexibility but impairs the ability of future users to search and find information.8, 26,27,28 A physician who prefers to document using more unstructured text might document a new diagnosis in the Assessment and Diagnosis section, but that new diagnosis may be overlooked by future users unless the physician also updates the Problem List. A physician who prefers templates to manage a mix of unstructured and structured text may choose to use the Review of Systems to document this same information. Our results therefore not only reveal where variation occurs but also offer a plausible mechanism to explain why we observe variation in certain documentation categories.

EHR design optionality, along with minimal organizational constraints on documentation, may relieve the strain of EHR adoption on frontline physicians by allowing them to document in the way they individually prefer.29, 30 However, their decisions may be guided not by a systematic determination of best practices but instead by an ad hoc and idiosyncratic process resulting in a documentation style that works well enough for a given user. Rather than optimizing documentation styles for the practice, the aggregation of varied individual choices incurs substantial costs over the long run. First, it compromises the retrieval of information at subsequent visits, which encumbers the delivery of high-quality care. For example, when different physicians in a practice use the same fields inconsistently (some documenting a new diagnosis in the Problem List and others documenting it in the Assessment and Diagnosis), it can lead to challenges with interpreting information and subsequently cause patient harm (for example, missing a diagnosis by looking in the wrong field). Second, many health care initiatives, such as precision medicine, require that EHRs contain complete and accurate patient data against which the latest evidence can be applied in order to identify opportunities to improve patient care.31 Similarly, the potential power of analyzing data stored in EHRs across the country undergirds the promise of the learning health system to provide ongoing feedback to both physicians and health care standards.32 Variation in EHR documentation makes it more difficult to pursue these important efforts that rely on leveraging EHR data from different institutions and settings. Together, this suggests that benefits from allowing variation in documentation may not be worth these substantial costs and that efforts to move toward more constrained, standardized documentation are therefore worth pursuing.

Our study offers insights into how practices can move toward more standardized documentation. Specifically, targeted user training during implementation to articulate the effects of documentation decisions and regular practice meetings to develop consensus around documentation are feasible and effective strategies. However, we did not find that these strategies were in widespread use because variation in EHR documentation manifests as small, frequent annoyances rather than substantial, salient problems. This makes it difficult to pursue documentation standardization as a high priority. Therefore, third-party stakeholders—in particular, payers and policymakers—may need to draw attention to the downstream costs of variation in EHR documentation and create incentives that motivate practices to pursue more standardized documentation.

Limitations

Our results should be interpreted with several limitations in mind. First, the de-identified EHR data came from a single vendor as well as lacked patient characteristics and conditions. Due to the former, it is possible that observed variation is not generalizable to other vendors. Due to the latter, it is possible that variation attributed to the physician level actually reflects differences in patients. However, while there may be some patient differences between physicians in the same practice that contribute to the variation we observe, we believe that our findings are robust to this risk because prior work describing patient panel characteristics across a sample of primary care physicians had narrow confidence intervals,17 suggesting a relatively similar mix of patients across primary care physicians in general, and differences in patient panel characteristics did not emerge as an explanation for variation in our qualitative interviews with practices using multiple vendors, which further confirmed such variation at the physician level and substantial enough to cause problems. Likewise, although data only reflect documentation during one month (June 2012), we have no reason to believe these encounters would differ meaningfully from encounters in any other month in a way that would impact variation between physicians in the same practice, and heard nothing to that effect in our qualitative interviews. Furthermore, practices using the vendor are only located in a subset of the USA; while the 27 represented states are not clustered in any particular geographic region, this nonetheless limits the generalizability of our findings. Additionally, qualitative data collection relied on respondents’ perceptions, which were not compared to data from their EHR to more conclusively determine the prevalence of certain forms of variation. Finally, the prevalence of variation by documentation category, the impacts of variation on care delivery, and the utility of identified strategies to minimize variation that we found may be different in specialty practices, which future research should examine.

Conclusion

In the first large-scale study of variation in EHR documentation, we found substantial variation in the completion of documentation for 5 clinical documentation categories. Such variation was perceived to detract from efforts to use the data subsequently and impede quality gains from the use of EHRs. Our study suggests targeted user training during EHR implementation and regular practice meetings focused on documentation could help avoid or curb variation by promoting more standardized documentation. However, this may require third-party actions to ensure practices engage in these activities in ways that result in better patient care.