Nudging Health Care Providers’ Adoption of Clinical Decision Support: Protocol for the User-Centered Development of a Behavioral Economics–Inspired Electronic Health Record Tool

Background The improvements in care resulting from clinical decision support (CDS) have been significantly limited by consistently low health care provider adoption. Health care provider attitudes toward CDS, specifically psychological and behavioral barriers, are not typically addressed during any stage of CDS development, although they represent an important barrier to adoption. Emerging evidence has shown the surprising power of using insights from the field of behavioral economics to address psychological and behavioral barriers. Nudges are formal applications of behavioral economics, defined as positive reinforcement and indirect suggestions that have a nonforced effect on decision-making. Objective Our goal is to employ a user-centered design process to develop a CDS tool—the pulmonary embolism (PE) risk calculator—for PE risk stratification in the emergency department that incorporates a behavior theory–informed nudge to address identified behavioral barriers to use. Methods All study activities took place at a large academic health system in the New York City metropolitan area. Our study used a user-centered and behavior theory–based approach to achieve the following two aims: (1) use mixed methods to identify health care provider barriers to the use of an active CDS tool for PE risk stratification and (2) develop a new CDS tool—the PE risk calculator—that addresses behavioral barriers to health care providers’ adoption of CDS by incorporating nudges into the user interface. These aims were guided by the revised Observational Research Behavioral Information Technology model. A total of 50 clinicians who used the original version of the tool were surveyed with a quantitative instrument that we developed based on a behavior theory framework—the Capability-Opportunity-Motivation-Behavior framework. A semistructured interview guide was developed based on the survey responses. Inductive methods were used to analyze interview session notes and audio recordings from 12 interviews. Revised versions of the tool were developed that incorporated nudges. Results Functional prototypes were developed by using Axure PRO (Axure Software Solutions) software and usability tested with end users in an iterative agile process (n=10). The tool was redesigned to address 4 identified major barriers to tool use; we included 2 nudges and a default. The 6-month pilot trial for the tool was launched on October 1, 2021. Conclusions Clinicians highlighted several important psychological and behavioral barriers to CDS use. Addressing these barriers, along with conducting traditional usability testing, facilitated the development of a tool with greater potential to transform clinical care. The tool will be tested in a prospective pilot trial. International Registered Report Identifier (IRRID) DERR1-10.2196/42653

1 K23 HL145114-01 2 MPOR (OA) RICHARDSON, S 1 K23 HL 145114 -01 RICHARDSON, SAFIYA RESUME AND SUMMARY OF DISCUSSION: This is a new K23 application from Dr. Safiya Richardson in which she has proposed a research career development plan focused on the development and testing of a computerized clinical decision support for pulmonary embolism risk prediction. The Candidate is exceptional. She has five first authored publications and strong letters of support. She has significant research experience in developing health information technologies to assist provider decision making. The Mentors are outstanding, have the necessary expertise, and have established relationships with the Candidate. The Research Plan seeks to improve "evidence-based" care at the point of decision making by using behavioral "nudging" to increase adoption of clinical decision support tools. The research is based on a strong premise and potentially impactful; however, there was no discussion of alternative strategies. Overall, this is an outstanding application from an exceptional young investigator. Reviewer enthusiasm was high despite some weaknesses in the research strategy.

DESCRIPTION (provided by applicant):
With the long-term career goal of becoming a leading independent researcher transforming health information technology, Safiya I. Richardson, MD, MPH, proposes a mentored research project and specific career development plan which will prepare her to use behavioral science to develop effective clinical decision support (CDS). The disparity between usual and evidenced based clinical practice is responsible for a third of hospital deaths and waste estimated at 380 billon dollars each year. Computerized CDS has the potential to narrow this gap by bringing meaningful and relevant evidence to health care providers at the point of decision making. However, moderate improvements in care seen with CDS are significantly limited by consistently low provider adoption, estimated at 10%. This project uses behavioral theory and key principles of behavioral economics to illuminate and address barriers to provider adoption of CDS. Using the Capability Opportunity Motivation Behavior (COM-B) framework Dr. Richardson will examine barriers to provider adoption of a pulmonary embolism risk prediction CDS tool. The use of CDS to assess pretest probability before computed tomography pulmonary angiography reduces testing by 25% without any missed pulmonary emboli. Routine use by providers would result in 600,000 fewer scans, 84,000 fewer cases of contrast induced nephropathy and prevent 3,000 malignancies as well as 2,000 cancer deaths in the United States every year. The overall objective of this training application is to develop and evaluate the feasibility and preliminary efficacy on provider adoption of a new tool that incorporates nudges designed to address barriers to tool use. Nudges are applications of behavioral science used by behavioral economists, defined as positive reinforcement and indirect suggestions which have a non-forced effect on decision making. The proposal aims to: 1) develop nudges designed to address identified behavioral barriers to adoption, 2) build and conduct iterative usability testing on prototypes of the new tool, 3) evaluate the feasibility and preliminary efficacy on provider adoption of the new tool compared to the current tool, in a pilot trial. This project uses a multistage mixed methods framework. It is the first to evaluate the impact of nudges on provider adoption of CDS. This research is complemented by career development activities, including formal training in health informatics, behavioral science, mixed methods and clinical trial design. With the guidance of an experienced mentoring team, the proposed research and training activities will lead to the development a competitive R01 grant application to assess the effectiveness of the new tool to improve health outcomes.

PUBLIC HEALTH RELEVANCE:
Computerized clinical decision support attempts to close the gap between usual and evidence based clinical practice by bringing relevant evidence to health care providers at the point of decision making, however, the moderate improvements in care seen with these tools have been significantly limited by consistently low provider adoption. This proposed study aims to use the principles of behavioral economics (i.e. the "nudge") in the development of a pulmonary embolism risk prediction clinical Overall Impact: Dr Richardson is a clinician scientist in her second year as junior faculty. She is working in the field of implementation science to understand the behavioral reasons why providers don't utilize clinical decision support. Much of her underlying work has focused on both the usability testing and the quantitative survey data surrounding a lack of adoption for a CDS tool to reduce the number of CTs for PE rule out.
The proposed training plan is integrated deeply with the research plan and includes a certificate in clinical informatics; mixed methods/ qualitative and behavioral economics and research methods and clinical trials. The research aims seek to develop nudges to address the identified barriers to CDS adoption (semi structured interviews done for low inter and high tool users). To do iterative testing on the prototypes of the new tool using traditional usability testing (Think aloud and near live). The third aim is testing the new tool in a pre-post design at 2 hospitals among 150 providers for acceptability adoption/ uptake and satisfaction. This is the aim that is methodologically the weakest and could be strengthened if using a difference in difference design or even doing a small cluster RCT at the 2 sites with the provider as the unit of randomization. This would likely be feasible during the K time because of the automated nature in flipping on a CDS tool at the provider level and the big data collection using informatics to collect all clinical outcomes at the patient level for those seen with chest pain or SOB etc.

Candidate: Strengths
 Fantastic candidate with clear career trajectory and dedication to improving clinical care.  She has had foundational research training and experiential learning through the NIH supplement.  The reduction of unnecessary testing is an important concept which is often more difficult to change in providers than asking them to perform additional tests or add a behavior to meet a quality metric. The use of behavioral economics to understand these concepts would make Dr Richardson a leader in this field and her work would contribute to this understanding. Weaknesses  None noted.

Career Development Plan/Career Goals & Objectives:
Strengths  Good collaborative mentorship team who have a record of working with Dr Richardson.  There is a clear plan for learning and advancement, although didactics are heavy in years 1-3.

Weaknesses
 I am uncertain of the time commitment need for the completion of the Certification program in Health IT. I can see how it may be beneficial; however, it appears to be a big portion of time in year 2 when much of the research will also be starting up and underway. The table notes only about 5-6 hours a week but that seems unlikely for a formal certificate program.

Research Plan: Strengths
 CDS is an important tool for health systems to remain agile and increase their ability to apply standardized and evidenced based knowledge at the frontline of patient care.  The understanding of the behavioral approaches to adoption and implementation/uptake of the CDS is of paramount importance to its success.  The use of mixed methods to iterate the CDS is a strength and would add to Dr Richardson's skills.

Weaknesses
 The efficacy trial to assess adoption of the PERK tool compared to standard CDS without the behavioral nudge is a strength; however, the pre-post design is not optimal. Consider a difference in difference design if there are enough timepoints pre and post; or consider randomizing at the level of the provider. preliminary efficacy on provider adoption of a new tool that incorporates "nudges" designed to address barriers to tool use. The research plan addresses a significant scientific gap and may inform developments of new methods to increase provider's adoption of CDS tools; thus, improving quality of care while reducing costs. There are a few minor methodological concerns that do not significantly diminish the enthusiasm for the project. The overall impact is high.

Candidate: Strengths
 The candidate obtained her medical degree and MPH from Columbia University, New York City, NY, followed by a residency in internal medicine and a fellowship in general internal medicine.  The candidate is currently a primary care physician and a faculty member at Hofstra Northwell School of Medicine in New York.  The candidate has significant research experience focusing on developing high impact health information technology to assist provider decision making.  A current recipient of a Research Supplement to Promote Diversity in Health-Related Research from the NIAID.  Long-term career goal is to use the principles of behavioral science to develop and implement high impact health information technology and Clinical Decision Support (CDS).  Published 9 peer-reviewed manuscripts (5 as first author).  The letters of recommendation are uniformly strong.  There is evidence of previous collaboration with the mentors. Weaknesses  None noted.

Career Development Plan/Career Goals & Objectives:
Strengths  The didactic coursework is tied specifically to proposed research and career goals.  The training plan includes sufficient detail linking proposed coursework and the contribution of each proposed course to the candidate's career objectives.  The plan includes appropriate training in the responsible conduct of research.

Weaknesses
 Methods of evaluation and training milestones are not sufficiently addressed.

Research Plan: Strengths
 Computerized clinical decision support (CDS) has the potential to narrow the gap between "usual" and "evidence-based" care at the point of decision making.  The central hypothesis is that the addition of a "nudge" [a positive reinforcement and indirect suggestions which have a non-forced effect on decision making] designed to address psychological barriers to CDS use will improve provider adoption.  The project is underpinned by a strong central premise. The study is conceptually innovative, as it proposes to use a behavioral framework (COM-B) to examine barriers to and facilitators of provider adoption of CDS and to evaluate the impact of nudges to increase provider adoption of CDS.  The first aim is to develop "nudges" designed to address identified behavioral barriers to adoption. The investigator proposes to create wireframes of the new CDS tool and to incorporate four nudges designed to address the identified barriers to tool use into the current tool. The use of semi-structured interviews in groups of varying use rates will provide valuable insight.  AIM 2 is to build and conduct iterative usability testing on prototypes of the new tool, "PERK".
Iterative usability testing will employ a convergent parallel mixed methods design, 1 K23 HL145114-01 6 MPOR (OA) RICHARDSON, S simultaneously collecting both qualitative and quantitative assessments of ease of use. Completion of this aim will allow for finalization of the prototypes and development of the final version of "PERK".

Weaknesses
 AIM 3 is to evaluate in a pilot trial the feasibility and preliminary efficacy on provider adoption of the new tool compared to the current tool. However, a higher adoption rate than the current tool may not be meaningful clinically. Is there a specific adoption threshold for a clinically-meaningful difference?  The success of the research is predicated on increasing adoption rates for the new tool. There is insufficient discussion regarding negative findings or alternative approaches.

Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s): Strengths
 The primary mentor is Dr. Thomas McGinn. Secondary mentors are: Michael Diefenbach, Ph.D. and Renee Pekmezaris.  The mentorship team also includes expertise in informatics and biostatistics.  The roles and responsibilities of each mentor or collaborator are clearly described.

Weaknesses
 None noted. Overall Impact: This is an interesting application by a promising candidate, with a reasonable track record of success, who makes a compelling case for a pivot. After a basis in epidemiology, they propose a pivot to a cleverly combined program of informatics to do clinical decision support and behavioral economics (lightly) to help inform making that decision support actually useful. This seems like a highly productive union.

Environment and Institutional
The program itself is an effort to design a clinical decision support program to reduce low yield CT scans for PE an important goal, and a reasonable initial model system. The Specific Aims of the grant as presented are a little frustratingly un-specific, but when one digs in, one can see a thoughtful build up. In some places greater depth of the motivating behavioral social theory and the link to intervention 1 K23 HL145114-01 7 MPOR (OA) RICHARDSON, S selection is desired, but presumably those are what the training program and Dr. Diefenbach's comentorship are for. In order to successfully compete for an R01, a much more nuanced grasp of specific behavioral/social theories will be needed, and the intervention choice will need to be rooted in more than the plausibility arguments presented here but the proposal gives reasonable confidence that such growth will occur to offer a promising research career.

COMMITTEE BUDGET RECOMMENDATIONS: RECOMMENDED AS REQUESTED
Footnotes for 1 K23 HL145114-01; PI Name: Richardson, Safiya NIH has modified its policy regarding the receipt of resubmissions (amended applications). See Guide Notice NOT-OD-14-074 at http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-074.html. The impact/priority score is calculated after discussion of an application by averaging the overall scores (1-9) given by all voting reviewers on the committee and multiplying by 10. The criterion scores are submitted prior to the meeting by the individual reviewers assigned to an application, and are not discussed specifically at the review meeting or calculated into the overall impact score. Some applications also receive a percentile ranking. For details on the review process, see http://grants.nih.gov/grants/peer_review_process.htm#scoring.

MEETING ROSTER NHLBI Mentored Patient-Oriented Research Review Committee
Heart, Lung, and Blood Initial Review Group NATIONAL HEART, LUNG, AND BLOOD INSTITUTE MPOR (OA) 06/21/2018 -06/22/2018 Notice of NIH Policy to All Applicants: Meeting rosters are provided for information purposes only. Applicant investigators and institutional officials must not communicate directly with study section members about an application before or after the review. Failure to observe this policy will create a serious breach of integrity in the peer review process, and may lead to actions outlined in NOT-OD-14-073 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-073.html and NOT-OD-15-106 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-106.html, including removal of the application from immediate review.