Stakeholder Perspectives on an Inpatient Hypoglycemia Informatics Alert: Mixed Methods Study

Background: Iatrogenic hypoglycemia is a common occurrence among hospitalized patients and is associated with poor clinical outcomes and increased mortality. Clinical decision support systems can be used to reduce the incidence of this potentially avoidable adverse event. Objective: This study aims to determine the desired features and functionality of a real-time informatics alert to prevent iatrogenic hypoglycemia in a hospital setting. Methods: Using the Agency for Healthcare Research and Quality Five Rights of Effective Clinical Decision Support Framework, we conducted a mixed methods study using an electronic survey and focus group sessions of hospital-based providers. The goal was to elicit stakeholder input to inform the future development of a real-time informatics alert to target iatrogenic hypoglycemia. In addition to perceptions about the importance of the problem and existing barriers, we sought input regarding the content, format, channel, timing, and recipient for the alert (ie, the Five Rights). Thematic analysis of focus group sessions was conducted using deductive and inductive approaches. Results: A 21-item electronic survey was completed by 102 inpatient-based providers, followed by 2 focus group sessions (6 providers per session). Respondents universally agreed or strongly agreed that inpatient iatrogenic hypoglycemia is an important problem that can be addressed with an informatics alert. Stakeholders expressed a preference for an alert that is nonintrusive, accurate, communicated in near real time to the ordering provider, and provides actionable treatment recommendations. Several electronic medical record tools, including alert indicators in the patient header, glucose management report, and laboratory results section, were deemed acceptable formats for consideration. Concerns regarding alert fatigue were prevalent among both survey respondents and focus group participants. Conclusions: The design preferences identified in this study will provide the framework needed for an informatics team to develop a prototype alert for pilot testing and evaluation. This alert will help meet the needs of hospital-based clinicians caring for patients with diabetes who are at a high risk of treatment-related hypoglycemia. (JMIR Hum Factors 2021;8(4):e31214) doi: 10.2196/31214 JMIR Hum Factors 2021 | vol. 8 | iss. 4 | e31214 | p. 1 https://humanfactors.jmir.org/2021/4/e31214 (page number not for citation purposes) Mathioudakis et al JMIR HUMAN FACTORS


Background
Hypoglycemia is a common occurrence in hospitals and has been linked to poor clinical outcomes and increased mortality [1,2]. Patients with and without diabetes can experience acute hypoglycemic episodes in the hospital, which may result in outcomes ranging from mild distress and patient dissatisfaction with cardiac ischemia, arrhythmias, loss of consciousness, stroke, seizures, and coma [1,3,4]. Of the 8 million and rising number of patients with diabetes who are admitted to hospitals in the United States annually, up to 25% may experience a hypoglycemic episode during hospitalization [5]. Approximately half of these episodes can be explained by an underlying illness, such as severe sepsis, renal failure, liver failure, or malignancy [3]; however, the remaining half are iatrogenic in nature, usually resulting from insulin treatment [6].
Insulin remains the recommended therapy for most patients with diabetes during hospitalization [7,8]. Recent data from the Medicare Patient Safety Monitoring System found that glucose-lowering medications were associated with the highest rates of adverse outcomes for all drugs used in the hospital [9]. As insulin accounts for the vast majority of hypoglycemic events [10,11], the Joint Commission and the Institute for Safe Medication Practices have designated it a high-alert medication [12]. Insulin is typically administered as a continuous infusion for critically ill patients in the intensive care unit (ICU) and as subcutaneous injections in noncritically ill patients in general medical or surgical wards. In contrast to the ICU setting where insulin adjustments are driven by nurse-managed protocols, insulin titration in the non-ICU setting is prescriber-driven and requires evaluation of a complex set of clinical, laboratory, and pharmacological parameters.
Clinical decision support (CDS) tools in electronic medical record (EMR) systems have been increasingly used in the United States and have been shown to improve the processes of care and clinical outcomes [13][14][15]. These tools have been used to alert clinicians, suggest diagnostic or treatment recommendations, and provide contextually pertinent information to optimize care for a wide variety of indications in the hospital setting, ranging from sepsis to acute kidney injury [16][17][18]. It stands to reason, then, that CDS could be used to improve care of patients vulnerable to hypoglycemia. Prediction models using large EMR data sets have been developed to trigger alerts in patients at risk of hypoglycemia [19][20][21][22]. Several studies have found that these predictive models can decrease the rate of inpatient hypoglycemic episodes, with one review finding that it could decrease the incidence of severe hypoglycemia by up to 68% [19,23].
At our institution, we have several existing CDS tools to guide clinicians in selecting a safe and effective initial insulin dosing regimen, including a mandatory subcutaneous insulin order set and an optional insulin CDS tool [24]. Although our subcutaneous insulin CDS tool was derived from evidence-based basal-bolus insulin dosing protocols [25], even universal use of this tool would not be expected to prevent all iatrogenic hypoglycemic events in the hospital because of the impact of acute illness and other clinical factors on glucose homeostasis (eg, nutritional status, renal function, and steroid doses).

Objectives
To address the need for real-time hypoglycemia risk detection, we recently developed a machine learning algorithm using EMR data that accurately predicts iatrogenic hypoglycemia in rolling 24-hour windows following each blood glucose reading during hospitalization [22]. In planning the translation of this machine algorithm into a real-time informatics alert, we sought to obtain feedback from inpatient clinicians who are responsible for the day-to-day management of blood glucose in the hospital. The objective of this study is to obtain stakeholder input regarding the design of a real-time hypoglycemia informatics alert for use in a hospital setting.

Study Design
This was a mixed methods study that integrated quantitative and qualitative data. The initial stage consisted of an electronic survey (administered in February 2019) that was sent to physicians and advanced practice providers at 2 academic medical centers located in Baltimore, Maryland. The second stage consisted of 2 separate focus groups (conducted in September 2019) with participants recruited from the pool of survey respondents, the goal of which was to expand further on the answers to the survey and identify common themes. This study was approved by the institutional review board at the Johns Hopkins School of Medicine. For the electronic survey, consent was implied from respondent completion, and written informed consent was obtained from all participants in the focus group sessions.

Survey
To evaluate the key features and functionality of a hypoglycemia risk alert, we developed a 21-item electronic survey that was administered through SurveyMonkey (SVMK Inc) via an embedded email hyperlink (Multimedia Appendix 1). The survey required approximately 7-10 minutes to complete and was sent to hospital-based clinicians involved in glucose management, including medical and surgical residents, hospitalists, surgical advanced practice providers, and inpatient diabetes nurse practitioners (NPs). To encourage participation, the email invitation with the survey link was sent directly by residency program directors (medicine, surgery, neurology, obstetrics/gynecology [OB/GYN]) to all residents in their programs, and by program leaders in the hospitalist, surgical advance practice provider groups, and inpatient diabetes management service. Thus, the total number of recipients who received the survey link (and hence the response rate) was not known by the study investigators.
Survey questions were developed using the Agency for Healthcare Research and Quality Five Rights of effective CDS as a guiding framework: the right information to the right person, in the right interventional format, through the right channel, and at the right time in the workflow [26]. Right information refers to the content and presentation of information to the end user (accuracy, estimated risk and reasons for predicted risk, and recommended action). Right person refers to the member of the health care team who is most appropriate to receive and respond to the alert and the method of identifying this individual. Right format refers to the type of CDS used to address the clinical scenario and right channel refers to the platform for communicating the alert. We considered several formats or channels within and outside our EMR system (Epic Version 2018, Epic Systems Corporation), including the best practice advisory (BPA), glucose management print group report, patient system lists, patient header, InBasket messaging, and text messaging. A description of each of these formats or channels is provided in Table 1, with example screenshots in Figure 1. Finally, right time in the workflow, a critical component of successful CDS interventions, refers to the time when the alert is presented to the end user in their usual clinical workflow to minimize disruption and achieve desired action. Multimedia Appendix 2 categorizes the survey questions according to the Five Rights topics. Table 1. Description of formats and channels considered for alert.

Epic best practice recommendation Description Term
Identify populations of patients that clinicians need to review regularly or notify clinicians of individual patients who need their attention.
Central hub for clinicians to see patients in their unit and across facility. System list can be compiled across hospital or providers can add a column in their existing patients' lists to identify patients who meet certain criteria.
Patient list ( Figure  1A) Use for one-time events that do not recur on a regular basis. Restrict how often and to whom BPAs appear so that they appear only to users who can act on them at a time when they perform the action. Limit the number of BPAs that appear in a separate window.
Alert that appears based on a wide variety of events and actions. BPA a ( Figure 1B) Show information to clinicians that they need to review or that should be available at a glance from anywhere in the chart.
Provides patient information relevant to user's specific role along left side of screen. BPAs can appear in Patient Header so they do not interrupt a clinician's workflow.
Patient header (Storyboard; Figure 1C) Use to show information that a clinician needs to make decisions based on the total information compiled in the report.
Glucose management report contains summary of all subcutaneous insulin doses over previous 24 hours, and all glucose and insulin doses administered since admission. Allows user to review relevant information about the patient from one spot in EMR b as opposed to searching several areas to compile information.
Glucose management report ( Figure 1D) Good fit for questions or issues that do not need to be handled immediately because users might not regularly check all of their InBasket messages. Secure, closed, task-based messaging system to send and receive information about patient care, directly linking messages to patient's accounts, chart, laboratory results, and orders.
InBasket message ( Figure 1E) N/A c CORUS: Secure text messaging system developed by Johns Hopkins Technology Innovation Center allowing users to communicate within channels or groups on computer or mobile device. This communication channel is external to EMR. CORUS text message ( Figure 1F) Intended for quick coordination between members of the team Epic Secure Chat (deployed after completion of this study and will ultimately replace existing CORUS text messaging system) allows users to have conversations with a single recipient or with a group of colleagues securely on a mobile device. This communication channel is internal to EMR.
Secure chat text message ( Figure 1G) N/A Designated field to specify on-call provider in EMR. First Call ( Figure 1H

Focus Groups
To gain a more in-depth understanding of the perceptions, preferences, and perceived barriers to a hypoglycemia informatics alert, we conducted 2 focus group sessions, each consisting of 6 different participants, after the results of the survey were analyzed. Hospital-based physicians, NPs, and PAs were eligible for participation. Participants who responded to the initial survey expressing interest in further research related to the topic were invited to the focus group sessions. No participant identifiers were used during the session, and all participants were referred to by a given letter at the start of the session.
The focus group sessions lasted 60-90 minutes and were led by an experienced focus group moderator (LLB). An institutional review board approved-approved structured interview guide was used (Multimedia Appendix 3), with questions designed to delve into survey results and reasoning behind responses. The principal investigator (NM) was also presented to address or clarify any participant questions during the session. The sessions were audio-recorded and professionally transcribed. The investigators further reviewed and corrected the transcriptional errors. The participants were provided with food during the session and a US $100 gift card.

Data Analysis
Descriptive statistics were used to summarize the characteristics of the survey respondents and focus group participants. For binary, multiple-choice responses and Likert response data, the number and percentage of responses were provided. For rank items, the average ranking was calculated as the weight of the ranked position. MAXQDA (Verbi Software, 2019) was used for qualitative analysis and thematic coding of data from the focus group sessions. We used a combination of deductive and inductive methods in our analyses [27]. Before coding the data, we identified an initial set of deductive codes centered on the CDS Five Rights framework. Close reading of the transcripts then informed the development of inductive codes, reflecting the ground, or participants' experiences. When the code structure was considered final (ie, no new concepts were apparent), one researcher independently applied the finalized code structure and synthesized the data into patterns (ie, a cohesive category of responses found across our participants) and themes (ie, a broad concept or topic that aggregates patterns). Table 2 shows the characteristics of the survey respondents and participants in the 2 focus group sessions. There were a total of 102 survey respondents, most of whom were internal medicine (IM) physicians, with fairly even representation from trainees and faculty or staff. Focus group 1 comprised 6 NPs (of whom 4 were inpatient diabetes specialists) and Focus group 2 comprised 6 physicians (half of whom were medicine residents).   Figure 2 summarizes the responses to the survey questions related to the importance of the problem and the perceived benefit of the proposed alert. There was unanimous agreement that preventing insulin-related hypoglycemia in hospitalized patients is an important priority, and 47% (48/102) either agreed or strongly agreed that preventing insulin-related hypoglycemia in the hospital is challenging. Nearly all providers reported that they reviewed blood glucose data on insulin-treated patients and almost all providers felt that they recognized glycemic patterns indicative of a need to adjust insulin doses. Accordingly, 82.3% (84/102) of the respondents felt confident in their ability to safely adjust their insulin doses. Over two-thirds felt that a real-time hypoglycemic alert would be beneficial.  Table 3 summarizes the survey responses related to the proposed information alert. The top 3 preferred formats were an alert indicator in the patient header, secure text message, and alert prompt displayed in the existing glucose management report. The least desired formats were an Epic InBasket message, a patient system list, and a pop-up BPA. In terms of information included in the alert, 78.4% (80/102) of the respondents indicated that they would want to know the specific reasons why an individual patient is predicted to be at risk based on the prediction model; 64.7% (66/102) of the respondents wanted the alert to categorize the predicted hypoglycemia risk into low, medium, and high levels; and 62.8% (64/102) wanted the alert to give an estimated probability of hypoglycemia. A minority of respondents (21/104, 23.5%) indicated that they would want the alert to hyperlink to an actual prediction model, and 16.7% (17/102) indicated that they would want the validated accuracy of the model displayed in the alert. With respect to accuracy, there was a wide range of acceptable sensitivity and specificity thresholds among respondents, with most considering 70% and above to be acceptable for both.  The vast majority of the respondents (90/102, 88.2%) indicated that they would want the alert to provide some recommended action, and 59.8% (61/102) of respondents suggested that there should be an option acknowledge the alert or ignore or override the alert. Other features of the alert desired by approximately half or less of the respondents included direct hyperlinks to our subcutaneous insulin order set (54/102, 52.9%) and subcutaneous insulin decision support tool (29/102, 28.4%) and the ability to directly consult endocrinology or inpatient diabetes management service (43/102, 42.2%). Although some respondents felt that the ability to easily consult endocrinology or diabetes services was important, most (85/101, 84.2%) felt that the alert should not automatically trigger a consult.

Survey Responses
Regarding the right person to receive the alert, 84.3% (86/102) of respondents indicated the person listed as first call in the EMR should be notified. In addition to the first call provider, 46.1% (47/102) of respondents indicated that the patient's nurse should also be notified. With respect to the workflow, 46.1% (47/102) of respondents indicated they would want to be notified as soon as the hypoglycemia risk was identified, whereas 26.5% (27/102) preferred the same time every day and 23.5% (24/102) preferred being notified only when entering the chart of a relevant patient.

Focus Groups
The core themes discussed during the focus groups centered on the CDS five rights format, with the aim of receiving feedback on how a functional alert system could best suit front-link clinicians at a major hospital, as well as anticipated barriers and challenges. Representative quotes illustrating each key theme are presented in Table 4. Glucose management report • "I think I find that tool to be the one that I use the most when I'm managing a patient's glucose and insulin because it gives me a good way to review a 24-hour snapshot." [OB/GYN resident] • "I really like the glucose management report. I think for internal medicine, we use it...all the time." [IM resident] • "I think it would be great to have it there. I don't think it's mutually exclusive from the other ones, but I will say for an alert, people are not always going to look in there at the right time that you want them to know about the information. This raises the concern that although the tool is made to help specifically with glucose management, that a lack of use by some staff may lead to delay in response to an alert.

Right Information
Accuracy was a common theme. According to the surveys, most of our participants wanted the alert to be at the very least 70% sensitive and specific. One participant noted as follows: Participants raised concerns that if the specificity was low at launch, the alert would go off repeatedly and would be ignored.
Many noted that similar alert systems have been deployed at our institution before validation of accuracy; as a result, many of the alerts were initially unsuccessful at gaining end user buy-in and changing behavior. To avoid this, one participant noted that the algorithm should be continuously adjusted to increase accuracy over time.
A common theme among focus group participants was the importance of defining specific reasons that triggered an alert.
Participants felt that sharing this information would present a learning opportunity for end users and could indirectly modify behavior in their care of future patients. Participants also voiced the concern that there should be an option to report the alert if it is inaccurate: I think having an option that says this alert is inaccurate and not only having that option but that triggering someone to review that alert and see why it was triggered and hopefully, revise it so that it's more accurate. [IM resident] Finally, echoing the results of our survey, the focus group participants emphasized their desire to have an alert to recommend the appropriate action. The tendency of alert systems to interrupt workflow without recommending an appropriate action leads to clinicians feeling overwhelmed. Participants noted that the best part of the other alert systems was the ability to help with workup and management. One participant noted as follows:

Right Person
There were mixed opinions among the focus group participants as to the right person to be notified of the alert. Many believed that contacting the individual or service listed as first call in the EMR would be the most sensible approach. However, many voiced the concern that the only way a first-call system would be efficient was if the EMR was accurately updated, which may not always be the case. Other participants voiced that the right person to be alerted should always be a nurse. One participant noted that with other alert systems, the nurse is responsible for triaging the alert and determining whether contacting the provider is required: I think the way the TREWS [sepsis alert] is set up is that the nurse gets the first alert and they're responsible for contacting first call or whoever the primary team is. And so, the benefit of that is they will know how that service is set up whether or not they use the first call. [IM resident]

Right Format
There was substantial heterogeneity in preferences regarding the alert format, and our discussion focused on the pros and cons of each format. Focus group participants noted that an alert indicator in the patient header would be a very suitable place for a hypoglycemia alert; however, concerns were raised that the patient header is already a very crowded space: Others really valued the idea of using the glucose management report, which is a tabular report summarizing insulin and glucose data in a temporal way that facilitates pattern recognition and insulin dose adjustments but cautioned that this passive approach may not achieve the desired action. A suggestion raised during the focus groups session that was not considered in the electronic survey was made to place an alert symbol next to a patient's glucose laboratory result to notify them of impending risk; this alert would be distinct from an abnormal laboratory result value to distinguish predicted risk from overt hypoglycemia. Participants felt that a results flag would be the most effective in increasing situational awareness:

Right Channel
The CORUS text messaging system was the unanimously desired channel for receiving the alert. All participants agreed that it was the quickest and most efficient way to be notified. A few even mentioned that they were too heavily reliant on CORUS, but noted it was the appropriate app to get the staff's attention:

Right Timing
The majority agreed that the alert should be direct and in real time to allow for appropriate action. One concern that many had was the fact that the alert might go off in a time when the staff member could not respond. One participant offered the suggestion of a snooze button to address this concern: What do you think about the idea of-

Barriers to Alert Systems
The focus group sessions elicited feedback about barriers and challenges related to hypoglycemic prevention and the proposed informatics alert. Some of the themes that emerged were: (1) iatrogenic hypoglycemia is not serious enough to justify an alert, (2) patient factors that cannot be controlled (eg, snacking without knowledge of the treating team or resistance to taking recommended insulin doses), (3) provider factors, including knowledge gaps, and, most importantly, (4) concerns regarding alert fatigue. Although the survey respondents almost unanimously agreed that preventing hypoglycemia was an important problem, some of the focus group participants seemed to question its significance. For example, one participant commented as follows: Many participants voiced concern that the alert system must be structured to prevent alert fatigue. Participants noted that other alert systems were constantly popping up inappropriately leading to people clicking through them and interrupting the workflow.

Principal Findings
Overall, the findings from this study indicate that iatrogenic hypoglycemia was perceived as an important problem that could potentially be addressed with a real-time informatics alert. We identified several key design requirements and preferences for the alert: (1) sufficiently accurate, (2) recommendations for care, (3) nondisruptive, and (4) communicated in near real-time to the responsible clinician. Several EMR tools, including an alert indicator embedded in the patient header, glucose management report, and flagged laboratory results, were deemed acceptable formats for consideration of hypoglycemia alerts. Although clinicians indicated that pop-up (interruptive) BPAs would be considered unacceptable because of alert fatigue and disruption of clinical workflow, we were surprised by the acceptability of the use of the hospital's text messaging system as a channel of communication for the alert because real-time text messaging could be interrupted.

Comparison With Prior Work
To our knowledge, there are no previous studies soliciting stakeholder input regarding inpatient hypoglycemia informatics alerts. A systematic review and meta-analysis of computerized CDS systems (CDSS) found that push notifications, the ability to execute action, and an interruptive element are the most prevalent features of CDSS that have been evaluated in controlled clinical trials [14]. Absolute incremental improvements in clinical processes or outcomes were demonstrated for CDSS that offered the ability to execute the desired action, and a trend toward improvement was shown for those that allowed for acknowledgment, were behavior-targeted, embedded in the EMR, and interrupted [14].
We recently developed a machine learning algorithm using EMR data that would be considered sufficiently accurate (sensitivity and specificity of approximately 80% and c-statistic of 0.90) based on the input received from stakeholders in this study [22]. As we progress toward translating our machine learning model into a real-time alert, we will seek to consider the key findings from this study to optimize end user buy-in and adoption. Specifically, we intend to develop a near real-time alert that provides specific and actionable recommendations to the ordering provider without disrupting their clinical workflow. The top candidate CDS formats we will consider based on the results of this study are a nonintrusive BPA embedded in the patient header, laboratory result, or existing glucose management report. The selection of the final CDS format will depend in part on our institutional requirements and programing feasibility.

Limitations
It is important to note that there have been three key system changes at our institution since we completed this study, which may have affected our findings. First, our text messaging system (CORUS) has been replaced by an alternative text messaging platform that resides within the EMR and is accessible through mobile devices (SecureChat). The main difference in the text messaging platform change is that the communication resides within the EMR and is accessible via a mobile app. Communications delivered via SecureChat would allow the user to quickly access other relevant information within the patient's chart without the need to access an external system. Second, a new feature called Storyboard was released in an Epic software upgrade, which shifted the patient header information from the top of the chart to the left side of the screen. The Storyboard includes patient identifiers and key information, including allergies, infection or isolation, and best practice advisories. At our institution, the Storyboard is already being used to summarize active alerts for a given patient (eg, sepsis) and would be a potential consideration for the development of hypoglycemia alerts. We believe that the inferences of this study related to the patient header would apply to the new Storyboard format. A third system change at our institution has been made in which all services are required to either list an individual or a service pager name (for surgical services) in the first call field. It is expected that the first call field will serve as the source of truth in identifying the responsible provider for a given patient at any given time.

Conclusions
Alert fatigue is a commonly cited limitation in CDSS [28][29][30]. Despite the fact that avoidance of alert fatigue emerged as one of the most important themes in our analysis, some incongruencies were identified among participants in this study. On the one hand, clinicians strongly opposed a pop-up style BPA; on the other hand, real-time alerting via text messaging was identified as a preferred channel of communication regarding hypoglycemia risk. Although a text message can perhaps be more easily ignored than a pop-up BPA, both forms of alerting could distract the clinician from their current clinical work. On the basis of the overall input we received from our stakeholders, we place greater value on avoidance of alert fatigue and disruption in clinical workflow than immediate notification of the risk of hypoglycemia. Therefore, we will explore several strategies to minimize alert fatigue, such as communicating increased risk in a nonintrusive fashion in the patient header, laboratory result section, or glucose management report.