Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research

Background Suicide is the 10th leading cause of death in the United States, with >47,000 deaths in 2019. Most people who died by suicide had contact with the health care system in the year before their death. Health care provider training is a top research priority identified by the National Action Alliance for Suicide Prevention; however, evidence-based approaches that target skill-building are resource intensive and difficult to implement. Advances in artificial intelligence technology hold promise for improving the scalability and sustainability of training methods, as it is now possible for computers to assess the intervention delivery skills of trainees and provide feedback to guide skill improvements. Much remains to be known about how best to integrate these novel technologies into continuing education for health care providers. Objective In Project WISE (Workplace Integrated Support and Education), we aim to develop e-learning training in suicide safety planning, enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The research aims include identifying strategies for the implementation and workflow integration of both the training and safety planning with patients, adapting 2 existing technologies to enhance general counseling skills for use in suicide safety planning (a conversational agent and an artificial intelligence–based feedback system), observing training acceptability and nurse engagement with the training components, and assessing the feasibility of recruitment, retention, and collection of longitudinal self-report and electronic health record data for patients identified as at risk of suicide. Methods Our developmental research includes qualitative and observational methods to explore the implementation context and technology usability, formative evaluation of the training paradigm, and pilot research to assess the feasibility of conducting a future cluster randomized pragmatic trial. The trial will examine whether patients hospitalized for medical or surgical reasons or traumatic injury who are at risk of suicide have better suicide-related postdischarge outcomes when admitted to a unit with nurses trained using the skill-building technology than those admitted to a unit with untrained nurses. The research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. Results Project WISE was funded in July 2019. As of September 2021, we have completed focus groups and usability testing with 27 acute care and 3 acute and intensive care nurses. We began data collection for research aims 3 and 4 in November 2021. All research has been approved by the University of Washington institutional review board. Conclusions Project WISE aims to further the national agenda to improve suicide prevention in health care settings by training nurses in suicide prevention with medically hospitalized patients using novel e-learning technologies. International Registered Report Identifier (IRRID) DERR1-10.2196/33695

1 K23 MH118361-01A1 3 SERV DARNELL, D trials research, including the responsible conduct of research with patients at-risk for suicide, and statistical methods for low base-rate outcomes and nested longitudinal clinical trials data. This K23 application addresses the NIMH Strategic Plan by developing strategies incorporating information technology and pragmatic feedback systems for suicide prevention efforts in real-world practice, reaching the full breadth of patients presenting to the health care system after injury.

PUBLIC HEALTH RELEVANCE
Each year in the United States approximately 30 million individuals require emergency department visits; 1.5-2.5 million are so severely injured that they require inpatient hospitalization and over 650,000 patients present for self-inflicted injury. Acute care medical settings, such as emergency departments and inpatient trauma units, present critical opportunities for suicide prevention, which is stymied by a lack of scalable and sustainable methods for training providers in high quality suicide prevention interventions. The proposed K23 harnesses advances in artificial intelligence, natural language processing, and machine learning to improve the scalability and sustainability of suicide prevention training among acute medical care providers to better reach the population of patients at-risk for suicide with potentially life-saving interventions. Overall Impact: This second submission of a K23 application that seeks to launch a research career aimed at developing scalable and sustainable training methods for healthcare providers delivering suicide prevention interventions. The three-phase proposed research plan leads to a pilot feasibility trial of a collaboratively developed training program for front-line nurses to deliver suicide safety planning intervention (SPI). The career development plan includes technology-focused team science, SPI and SPI fidelity assessment, and acute care suicide prevention and intervention clinical trials research. The application comes from an outstanding and productive candidate with a clear commitment to research and the patient population. She is supported by a highly accomplished mentoring team -each of whom fulfill a specific niche in support of the candidate's career development. The career development plan addresses the relevant components for the proposed research, and overall, the application seems likely to launch an independent career, with outcomes from the third phase of the research plan intended to inform a full-scale RCT. While the initial review was overall quite enthusiastic of the candidate, career development and plan research aim, the major weakness was the concern that of the research and career development plans were quite ambitious, with many "cutting-edge" elements potentially not feasible in the proposed timeframe. In addition, aspects of the training program and career development plan were not presented with enough detail. The resubmission clarifies that proposed conversational agent ("Client Bot") and automated quality assessment system ("CORE-MI") are both well-validated and ready to use in the clinical setting. Further, the complexity and practical burden of Aim 1 is reduced to focus on the local implementation context as opposed to surveying 25 sites participating in an ongoing multisite clinical trial. There is additional justification for the training plan and add training in implementation science models and methods, as well as provider education and behavior change with an in-depth application of an implementation science framework through selfreport nursing surveys during the trial and end-of-trial nurse focus groups. Finally, the resubmission has a reorganization of the training goals around three core areas to better articulate the justification for training activities. The application discusses that this K23 award would not be focused on a new 1 K23 MH118361-01A1 4 SERV DARNELL, D Suicide Prevention training program, but rather the key innovation is for the candidate to learn how to harness innovative technologies to facilitate skills practice and provide feedback and coaching in an applied context. Overall, the resubmission responds to the major concerns although the career development and research plan is still ambitious.

Candidate: Strengths
• The candidate is a clinical psychologist with a strong prior research training and commitment focused on patient-focused research to improve outcomes in substance abuse and victims of trauma.
• Highly productive with more than 22 publications, has received Diversity Supplement and T-32 award.
• Candidate has strong potential to develop as an independent and productive researcher, as indicated by candidate's prior training and research experiences (including academic productivity to date), letters of reference, and endorsement and support from mentoring/advisor/consultant team. Weaknesses • None.

Career Development Plan/Career Goals & Objectives: Strengths
• Career Development Plan and Goals are appropriate for the scientific development of the candidate in the proposed content areas of interest.
• Responded well to previous concerns and didactic plan maps nicely with career aims.

Weaknesses
• Still fairly ambitious goals but somewhat simplified; an implementation goal was added.

Research Plan: Strengths
• Research aims are simplified, with a focus on local implementation instead of surveying 25 sites.
• Better evidence to support the effectiveness of SBI is presented.
• Nursing leader will also be a champion for the project.

Weaknesses
• This is a minor weakness, but it might have strengthened the application to discuss the time needed for each nursing intervention since trauma nurses are very busy at a Level One Trauma center. Exactly when in the visit (inpatient vs ED) the intervention occurs is not clearly specified. The feasibility question will still need to be proven.

Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s): Strengths
• The mentor/consultant/advisor team appears well positioned to support Dr. Darnell in both her training and research plans, with each team member committing to contribute to a specific aspect or niche content area outlined within her plans.
• Many members of the mentoring team have histories of collaborating with one another, and some have previously worked with Dr. Darnell directly. Weaknesses • The number of mentors and advisors will require significant program coordination and time. The mentoring plan does not appear to be significantly simplified in the resubmission. • Harborview Medical Center, the host site for this project, is home to a Level 1 Trauma Center managed by the University of Washington School of Medicine and sees 5,500-6,500 trauma admissions per year. Weaknesses • None.

Protections for Human Subjects:
Acceptable Risks and Adequate Protections Data and Safety Monitoring Plan (Applicable for Clinical Trials Only): Acceptable.

Inclusion of Women, Minorities and Children:
• • Acceptable.

Comments on Subject Matter (Required):
• Acceptable.

Comments on Faculty Participation (Required; not applicable for mid-and senior-career awards):
• Acceptable.

Comments on Frequency (Required):
• Acceptable. Overall Impact: This resubmission of a K23 application is from a qualified candidate who is surrounded by a qualified team of mentors and consultants. The application is somewhat responsive to the prior review. Some of the proposed training is online -in particular, the natural language processing and machine learning. The machine learning will be complex, and it is not fully clear if this is necessary for the proposed training, since the CORE-MI and Client-Bot already are developed and only the output will be used in this research. The study procedures for Aims 1-3 lack adequate details about the timing of assessments. The analytic plan seems to be one small paragraph, which also lacks adequate details and rigor. Similar concerns were raised in the prior submission but did not seem to be adequately addressed in the current submission.

Candidate: Strengths
• The candidate has been very productive as noted by her publications.
• The K23 application builds upon a solid foundation in psychology and mental health services research.

Career Development Plan/Career Goals & Objectives: Strengths
• The training goals of technology-focused training, implementation science, and suicide prevention clinical trials are appropriate.
• The training is tied to the research.

Weaknesses
• It is not fully clear that auditing the biostatistics course would be sufficient, and it is not clearly described who the mentor would be to advise the candidate when implementing these analytic methods.
• The machine learning online course is probably not the right course for the proposed research.
Machine learning is a very technical application, and it does not seem that this will be used in the proposed research.

Research Plan: Strengths
• The first two aims are well positioned to understand how to best integrate the technology in practice.
• The use of existing technologies -CORE-MI and Client Bot -is a strength. These have been used and the candidate has experienced mentors to assist her with implementation of this technology. Weaknesses • The research plan lacks detail of how and where nurses will be recruited or any anticipated response rate. Much of the details appear to be embedded in the Human Subjects Section. At least 30 nurses will participate in Aim 1, 3 in Aim 2, and 20 in Aim 3, with those participating in Aim 3 naïve to the study. It might have strengthened the application to discuss what is the base pool from which the trauma nurses are going to be recruited and how likely it will be to recruit the desired sample.
• The feasibility of organizing 3 focus groups of 10 nurses each is not adequately discussed. The challenge is getting a time and location where all can attend.
• The setting from which the nurses and patients will be recruited is not fully described, and there is insufficient discussion about the study setting and where study procedures will take place.
• The timing of the assessments for Aim 3 appears to be baseline, 1 month and 6 months.
Patients' complete assessments at 1 and 3 months, and there are patient encounter recordings that will happen soon after the standardized patient assessment. There are a lot of moving parts of this trial, and it is not fully clear how data collection is organized and mapped out.
• It appears that patients are recruited during the inpatient stay, and all study procedures are completed before discharge (these individuals are in the facility for 3 months). The Human Subjects form has more detail and states that patients complete all study procedures prior to discharge. Without information about this particular setting, it is difficult to place this in context with respect to a patient's length of stay, etc. On page 174, there is mention of contact numbers for follow up, so some individuals presumably will leave the facility.
• There seems to be nothing mentioned about any incentives for participants. The same can be said for the nurses.
• The research assistant will require substantial training, but it is not fully clear when and who will do this.
• There is limited discussion about the possibility for missing data and how the candidate will handle this.
• The data analysis plan seems particularly sparse. It is not clearly described how all of the assessments collected during the randomized trial will be operationalized and entered into the analytic models and whether the main outcome is baseline to 6 months for the nurses and baseline to 3 months for the patients.
• The end of study focus group is mentioned, but there is insufficient detail about this component of the study, who will participate, and how it will be structured and analyzed.

Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s): Strengths
• The mentors and consultants are accomplished researchers with expertise in the key areas of the training and research plan.

Environment and Institutional Commitment to the Candidate: Strengths
• The institutional environment is strong.

Study Timeline: Strengths
• The clinical trial is designed as a feasibility and acceptability pilot, and it is adequately over a year and one-half to recruit 20 nurses and 3 of their patients.

Weaknesses
• • A DSMP does not seem to be included in the application.
Inclusion of Women, Minorities and Children:

Training in the Responsible Conduct of Research:
Acceptable.

Comments on Format (Required):
• Format is varied and adequate.

Comments on Subject Matter (Required):
• Subject matter is comprehensive

Comments on Faculty Participation (Required; not applicable for mid-and senior-career awards):
• Preceptors are involved in the training.

Comments on Duration (Required):
• The duration is adequate

Comments on Frequency (Required):
• Frequency is also adequate and meets standards. Overall Impact:

Authentication of Key Biological and/or Chemical
The application outlines major hurdles to scalable and sustainable methods for training routine healthcare providers in suicide prevention. Innovative technologies include conversational agent and AI-based feedback of counseling performance. This K23 application will utilize computer technology for the purposes of training general medical providers in suicide prevention and monitoring the quality of skills ongoing. The candidate is a clinical psychologist that has focused on patient-centered research to improve outcomes in substance abuse and victims of trauma. She has been very productive and is well situated for a K23 patient-oriented career award. The application documents the need for training 1 K23 MH118361-01A1 9 SERV DARNELL, D for medical personnel in suicide prevention. The training plan includes certification in User-Centered Design, two online courses to gain basic knowledge of the computer technology underlying the Client Bot and CORE-MI technologies, and two courses at the University of Washington on Dissemination and Implementation Science Clinical Trials Biostatistics. The mentorship team provides a good supportive structure and expertise to meet the proposed training and research goals. The research aims include the development of a technology-based training for a Suicide Safety Planning Intervention (SPI) using a conversational agent and compare it to a web-based didactic only. This resubmission is responsive to the previous review regarding feasibility of the research and training plans.

Candidate: Strengths
• The candidate is a clinical psychologist that has focused on patient-centered research to improve outcomes in substance abuse and victims of trauma. She has been very productive and is well situated for a K23 patient-oriented career award.

Career Development Plan/Career Goals & Objectives: Strengths
• Includes certification in User-Centered Design, two online courses to gain basic knowledge of the computer technology underlying the Client Bot and CORE-MI technologies, and two courses at the University of Washington on Dissemination and Implementation Science Clinical Trials Biostatistics.

Research Plan: Strengths
• Innovative technologies include conversational agent and AI-based feedback of counseling performance. • Responsive to previous review in regard to feasibility.

Weaknesses
• Limited analytic plan.

Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s): Strengths
• The mentorship team provides a good supportive structure and expertise to meet the proposed training and research goals.

THE FOLLOWING SECTIONS WERE PREPARED BY THE SCIENTIFIC REVIEW OFFICER TO SUMMARIZE THE OUTCOME OF DISCUSSIONS OF THE REVIEW COMMITTEE, OR REVIEWERS' WRITTEN CRITIQUES, ON THE FOLLOWING ISSUES:
PROTECTION OF HUMAN SUBJECTS: UNACCEPTABLE. In Study 4, recruitment of patients at the bedside may breach confidentiality, since these are not private rooms. The application does not adequately discuss what would be done to minimize this.

INCLUSION OF CHILDREN PLAN: ACCEPTABLE.
Children are appropriately excluded.

COMMITTEE BUDGET RECOMMENDATIONS: The budget was recommended as requested.
Footnotes for 1 K23 MH118361-01A1; PI Name: Darnell, Doyanne Aspen NIH has modified its policy regarding the receipt of resubmissions (amended applications). See Guide Notice NOT-OD-14-074 at http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-074.html. The impact/priority score is calculated after discussion of an application by averaging the overall scores (1-9) given by all voting reviewers on the committee and multiplying by 10. The criterion scores are submitted prior to the meeting by the individual reviewers assigned to an application, and are not discussed specifically at the review meeting or calculated into the overall impact score. Some applications also receive a percentile ranking. For details on the review process, see http://grants.nih.gov/grants/peer_review_process.htm#scoring.

Mental Health Services Research Committee National Institute of Mental Health Initial Review Group NATIONAL INSTITUTE OF MENTAL HEALTH SERV 02/27/2019
Notice of NIH Policy to All Applicants: Meeting rosters are provided for information purposes only. Applicant investigators and institutional officials must not communicate directly with study section members about an application before or after the review. Failure to observe this policy will create a serious breach of integrity in the peer review process, and may lead to actions outlined in NOT-OD-14-073 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-073.html and NOT-OD-15-106 at https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-106.html, including removal of the application from immediate review.