Effectiveness of an Emergency Department–Based Machine Learning Clinical Decision Support Tool to Prevent Outpatient Falls Among Older Adults: Protocol for a Quasi-Experimental Study

Background Emergency department (ED) providers are important collaborators in preventing falls for older adults because they are often the first health care providers to see a patient after a fall and because at-home falls are often preceded by previous ED visits. Previous work has shown that ED referrals to falls interventions can reduce the risk of an at-home fall by 38%. Screening patients at risk for a fall can be time-consuming and difficult to implement in the ED setting. Machine learning (ML) and clinical decision support (CDS) offer the potential of automating the screening process. However, it remains unclear whether automation of screening and referrals can reduce the risk of future falls among older patients. Objective The goal of this paper is to describe a research protocol for evaluating the effectiveness of an automated screening and referral intervention. These findings will inform ongoing discussions about the use of ML and artificial intelligence to augment medical decision-making. Methods To assess the effectiveness of our program for patients receiving the falls risk intervention, our primary analysis will be to obtain referral completion rates at 3 different EDs. We will use a quasi-experimental design known as a sharp regression discontinuity with regard to intent-to-treat, since the intervention is administered to patients whose risk score falls above a threshold. A conditional logistic regression model will be built to describe 6-month fall risk at each site as a function of the intervention, patient demographics, and risk score. The odds ratio of a return visit for a fall and the 95% CI will be estimated by comparing those identified as high risk by the ML-based CDS (ML-CDS) and those who were not but had a similar risk profile. Results The ML-CDS tool under study has been implemented at 2 of the 3 EDs in our study. As of April 2023, a total of 1326 patient encounters have been flagged for providers, and 339 unique patients have been referred to the mobility and falls clinic. To date, 15% (45/339) of patients have scheduled an appointment with the clinic. Conclusions This study seeks to quantify the impact of an ML-CDS intervention on patient behavior and outcomes. Our end-to-end data set allows for a more meaningful analysis of patient outcomes than other studies focused on interim outcomes, and our multisite implementation plan will demonstrate applicability to a broad population and the possibility to adapt the intervention to other EDs and achieve similar results. Our statistical methodology, regression discontinuity design, allows for causal inference from observational data and a staggered implementation strategy allows for the identification of secular trends that could affect causal associations and allow mitigation as necessary. Trial Registration ClinicalTrials.gov NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064 International Registered Report Identifier (IRRID) DERR1-10.2196/48128

HITR PATTERSON, B RESUME AND SUMMARY OF DISCUSSION: This R18 health services research grant resubmission from Dr. Brian Patterson, from the University of Wisconsin-Madison, proposes to adapt the design of an automated screening and referral intervention for implementation in three diverse emergency departments (EDs) settings, and test the effectiveness of the automated screening and referral intervention on both completed referrals to a multidisciplinary fall prevention clinic and rates of injurious falls. The study will also evaluate implementation of the automated screening and referral intervention in three diverse ED sites using a mixed methods approach. The reviewers agreed on the significance of this study and noted that falls are an extremely significant problem in the community and the ED is a good place to identify patients in danger and refer. Identifying patients at high risk for falls when they present in the ED could prevent falls thus improving patient outcomes and reducing costs. They also noted that this is a well-written resubmission that will build upon pilot work to address the implementation of a clinical decision support tool using human factors and implementation science methods. The use of multiple datasets (collected data, Medicare claims, etc.) minimizes the risk of lost patients and may improve the algorithm. It includes a complimentary team of researchers with prior history of successfully working together. The resubmission is responsive to the previous review critique including issues with unspecified approach data collection and analysis methods. However, the reviewers also noted some very small concerns. There are some missing details about the EHR data analyses and implementation outside the investigators health system. The reviewers also noted an unclear description of the intersection between the clinical decision support (CDS) and the provider and why integration into workflow was previously a problem. Overall, the reviewers recommended this application for further consideration with an outstanding level of enthusiasm.

DESCRIPTION (provided by applicant):
Falls are the leading traumatic cause of both injury and death among older adults. American emergency departments (EDs) see over 3 million fall victims yearly, yet they play little role in primary or secondary fall prevention. The ED is an ideal site to identify patients at risk of future falls, however in this setting preventive care cannot be implemented at the expense of the primary mission of the ED: the provision of emergency care in a time-pressured environment. As the population ages, and the ED continues to expand its role as the primary site for delivery of acute unscheduled care, there is an urgent need to create a scalable intervention to assess older adults for fall risk and link them to appropriate risk reduction interventions after discharge without adding additional workload for nurses or physicians. Through an AHRQ K08, our study team has developed and validated an innovative automated screening and referral intervention for fall risk. This intervention harnesses existing data to select and connect patients to appropriate primary and secondary prevention services after ED visits without adding burden to nurse or physician workloads. This intervention features smart use of automation for screening and referral tasks maintaining physician decision autonomy, as well as the unique ability to adjust referral rates based on clinic availability. This intervention features smart use of automation for screening and referral tasks maintaining physician decision autonomy, as well as the unique ability to adjust referral rates based on clinic availability. Based on our work, UW Health is currently piloting the intervention, and has committed to implementing it at three diverse ED sites. This study will adapt the intervention for implementation at additional sites, and investigates the implementation and effectiveness of the automated screening and referral process in all three EDs through three specific aims: 1) Adapt the design of an automated screening and referral intervention for implementation in three diverse ED settings, using a human factors approach. 2) Test the effectiveness of the automated screening and referral intervention on both completed referrals to a multidisciplinary fall prevention clinic and rates of injurious falls using EHR data generated during implementation. 3) Evaluate implementation of the automated screening and referral intervention in three diverse ED sites using a mixed methods approach. This grant proposal builds upon our previous innovative work developing both CDS and risk-stratification algorithms to improve the quality and safety of care delivered to older adult ED patients. We will address the urgent and growing need for a scalable strategy for fall risk reduction from the ED by demonstrating the effectiveness of our novel HITR PATTERSON, B approach in a study spanning diverse hospital types and patient populations. Furthermore, knowledge gained from this work will inform other use cases which could benefit from automated risk-stratification and care coordination in the ED and beyond.

PUBLIC HEALTH RELEVANCE:
This proposal directly addresses the public health burden of the rising rates of significant falls in older adults which are the leading cause of traumatic morbidity and mortality in the elderly. Specifically, the use of an automated screening tool harnesses existing data resources and information systems to use the ED to better coordinate care between the ED and existing preventive resources in the health system. Beyond improving care for older adults with falls, this study will demonstrate the promise of using information technology to deliver public health interventions in the ED setting without diverting provider resources from the core ED mission of providing quality acute care.

CRITIQUE NOTE:
The sections that follow are the essentially unedited, verbatim comments of the individual committee members assigned to review this application. The attached commentaries may not necessarily reflect the position of the reviewers at the close of group discussion, nor the final majority opinion of the group. The above RESUME/SUMMARY OF DISCUSSION represents the evaluation of the application by the entire committee.

CRITIQUE 1
Significance: 2 Investigator(s): 1 Innovation: 1 Approach: 3 Environment: 1 Overall Impact: Strengths • Well-written grant that will build upon pilot work to address the implementation of a clinical decision support tool using human factors and implementation science methods.

Weaknesses
• Some missing details about the EHR data analyses and implementation outside the investigators health system.

Significance: Strengths
• Falls are a common source of morbidity in older adults and efforts to reduce falls through primary care has had limited success.

Weaknesses
• Investigators gloss over the future challenges associated with the dissemination of their approach outside their health system, which if this cannot be addressed would decrease the significance of their system.

Investigators: Strengths
• Team seems to have the right combination of expertise necessary to complete the project.
• The PI is a K08 recipient and this work is a clear extension of his previous research.
• Team members have collaborated in the past. HITR

Weaknesses
• Concern that Dr. Liao may have too little effort to assist with the implementation of the intervention at 2 additional sites within their health system.

Innovation: Strengths
• Interventions aimed at targeting patients in the ED for prevention of future health outcomes is relatively novel, especially given that the intervention will need very little effort from limited time of ED personnel. • Providing absolute risk predictions with the ability to titrate the level at which the intervention is triggered is not generally done and provides flexibility. Weaknesses • None noted.

Approach: Strengths
• Nice use of existing theoretical models (i.e. RE-AIM and SEIPS) to drive the methodological techniques for evaluating the design and implementation of the tool. • Robust use of Human Factors Design methodology to obtain feedback from clinicians, interdisciplinary team members, automated referrals, UW patient and Family Advisory council, etc. • Investigators appear to have adequately addressed a previous reviewers concern about the distance between one of the sites and the Falls clinic. • CMS claims data will be used to address previous concerns about injurious falls occurring outside the health system.

Weaknesses
• Some missing details about EHR data handling and future implementation.
• How were missing EHR data points handled in the regression modeling and how will missing EHR data points be handled when implemented in future patients? • Lack of details about how the risk model calculation, which appears to occur outside of the EHR is "piped back into the EHR". Does it use FHIR resources or other APIs, exactly where do the results appear in the EHR and in what format? Is there an ETL process in place for storing the results in the EHR back up database? • What happens when the cloud based system "goes down"? • No apparent method for addressing the competing risk of death due to other causes during the follow up period. The project does not appear to take use regional death registries or tools like the National Death Index. • Measures of SDoH are limited. For example, EHR addresses could be geocoded and mapped at the block group level with ACS data.

Environment: Strengths
• Excellent academic university with high volume EDs that serve heterogeneous populations.
• Existing resources such as the ICTR, falls clinic, and advanced data science group. • Established EHR system with data mapped to i2b2.
• Current use of a cloud computing system that can provide real-time calculations at the point of care.

PATTERSON, B
Protections for Human Subjects: Acceptable.

Strengths
• Appropriate data safeguards.

Single IRB for Cooperative Research
• NA

Degree of Responsiveness:
Addresses public health burden of falls in older adults. Improves coordination of care. Demonstrates the use of IT to deliver better quality care.

Inclusion of Women, Minorities, and Individuals Across the Lifespan: Acceptable. Strengths
• Includes women and minorities.
• Appropriately focused on older adults at risk for falling.

Inclusion of AHRQ Priority Populations: Acceptable. Strengths
• Includes an ED that serves a rural population and lower income individuals in both urban and rural settings.
Budget and Period of Support: Appropriate, no concerns.

Data Management Plan:
Adequate.

Resubmission Applications: Strengths
• Addressed issues regarding distance of ED with falls clinic.
• Addressed issue regarding falls occurring outside the health system.

Weaknesses
• Did not address comments about missing EHR data and EHR connections.

CRITIQUE 2
Significance: 1 Investigator(s): • Identifying patients at high risk for falls when they present in the ED could prevent falls thus improving patient outcomes and reducing costs. • Use of multiple datasets (collected data, Medicare claims, etc.) minimizes the risk of lost patients and may improve the algorithm.

Weaknesses
• COVID-19 may impact the number of patients who come to the ED and thus patients receiving/completing referrals. • It is not clear how readily the algorithm can translate to other facilities with a different EHR.

Significance: Strengths
• Falls are a significant issue for older patients and the healthcare system, leading to morbidity/mortality and high costs for care. • Using the ED for prevention could shift care dynamics and prevent more costly care in the long term.

Investigators: Strengths
• Team is diverse in expertise and has strong experience.
• Statistician on the team.
• Established collaboration between team members. Weaknesses • None noted.

Innovation: Strengths
• The adjustment threshold for referrals is interesting and novel.
• Use of the ED for prevention is unique.
• The predictive algorithm is fairly novel.

Weaknesses
• Methods are well established.

Approach: Strengths
• Site selection supports capturing data from diverse populations.
• Linking Medicare claims data will help minimize lost data.
• Prior work suggests feasibility and appropriateness of the proposed methods.
• Potential problems and alternative strategies are described.

Weaknesses
• There is a potential that ED visits decreased due to COVID-19 as people seek telehealth and other options. It is not clear if that will have an impact on the proposed work.

Environment: Strengths
• The environment is well positioned for the proposed work to be successful.
• The Mobility and Falls Clinic is a unique site for the intervention.
• EDs see a large number of patients. Weaknesses HITR PATTERSON, B • None noted.

Strengths
• Protections are appropriate for potential participants. Weaknesses • None noted.
Single IRB for Cooperative Research: N/A.

Degree of Responsiveness:
Highly Responsive.

Strengths
• The application seeks to improve patient safety related to potential falls through predictive use of data.
Inclusion of Women, Minorities, and Individuals Across the Lifespan: Acceptable.

Strengths
• Participants will be recruited to ensure inclusion of women and minorities. Weaknesses • None noted.

Strengths
• Focus is on the elderly population.
Budget and Period of Support: Acceptable.

Strengths
• Budget and period of support are reasonable for the proposed work. Weaknesses • None noted.

Data Management Plan:
Adequate.

Resubmission Applications: Strengths
• Concerns about missing data/patients lost to follow-up have been addressed.
• Greater detail is provided about qualitative analysis.
• Letter of support now indicates capacity for number of potential referrals. Weaknesses • None noted.

CRITIQUE 3
Significance: Overall Impact: Strengths • Falls are an extremely significant problem in the community and the ED is a good place to identify patients in danger and refer. • Complimentary team of researchers with prior history of successfully working together.
• PI has successfully completed related research and has extensive knowledge of the available information system sources. • Application is responsive to some the previous review critique related to unspecified approach data collection and analysis methods.

Weaknesses
• Vague description of the intersection between the CDS and the provider and why integration into workflow was previously a problem. • Not clear how the provider will be alerted and respond to a CDS alert for fall risk and referral.

Significance: Strengths
• Addresses a significant problem, falls in the elderly, where not screening for in the ED is a missed opportunity. Weaknesses • None noted.

Investigators: Strengths
• Complimentary team of researchers with prior history of successfully working together.
• PI has successfully completed related research and has extensive knowledge of the available information system sources.

Innovation: Strengths
• Innovative to screen and refer in the ED using existing data.

Weaknesses
• Better method for screening and referral follow-up could be described.

Approach: Strengths
• Responsive to previous critique regarding need for better specification of data collection and analysis. • Detailed explanation of qualitative coding strategy.
• Detailed timeline of activities provided.
• Adequately address human subjects recruitment and protections.

Weaknesses
• Not clear how the CDS tool will be implemented in the workflow. Even though this is to be developed based on participant feedback, some idea of CDS/provider interface would be helpful.