Development of an Implementation Science Telehealth Toolkit to Promote Research Capacity in Evaluation of Telehealth Programs

Background: The field of telehealth is rapidly growing and expanding access to quality health care, although there have been varied implementation outcomes in telehealth modalities. Dissemination and implementation (D&I) research can provide a systematic approach to identifying barriers and facilitators to telehealth implementation processes and outcomes. Methods: An interdisciplinary research and clinical team developed an implementation science telehealth toolkit to guide D&I evaluations of new and existing telehealth innovations. Results: The toolkit includes a separate section to correspond to each step in the D&I evaluation process. Each section includes resources to guide evaluation steps, telehealth specific considerations, and case study examples based on three completed telehealth evaluations. Discussion: The field of telehealth is forecasted to continue to expand, with potential to increase health care access to populations in need. This toolkit can help guide health care stakeholders to develop and carry out evaluations to improve understanding of telehealth processes and outcomes to maximize implementation and sustainability of these valuable innovations.


Background
][10][11] Thus, it is vital to understand implementation processes, characterize determinants of successful telehealth implementation, and define strategies for quality telehealth care across settings.
The field of dissemination and implementation (D&I) research has been showcased as key to accelerating the dissemination, implementation, and sustainability of evidence-based interventions into health care practice. 12,13D&I research is typically guided by frameworks, theories, and models with interdisciplinary research, clinical, and administrative teams using multiple methods to describe a variety of implementation processes and outcomes related to health care innovations.The multitude of approaches to D&I research can pose barriers to engagement in this field, but educational resources such as toolkits have been leveraged to increase engagement. 14The Agency for Healthcare Research and Quality (AHRQ) defines a ''toolkit'' as a collection of resources that can guide users in implementation of evidence-based practices or development of plans to reach evidence-based recommendations. 15hile there are numerous existing D&I toolkits in the literature to support D&I evaluations of evidencebased interventions to health care practice, no D&I toolkits specifically focus on telehealth interventions. 16he field of D&I research offers many benefits to telehealth including providing a systematic approach to understanding telehealth outcomes and processes, demonstrating the value of telehealth programs, identifying challenges and success in telehealth delivery, facilitating utilization of data to overcome barriers and challenges, and guiding implementation strategies to improve patient and system outcomes.
Due to recent rapid telehealth expansion and benefits of the D&I field to contribute to efficient and effective telehealth implementation processes and outcomes, a team within the Telehealth Center of Excellence (COE) created a D&I telehealth toolkit to guide interprofessional telehealth teams to design and conduct D&I telehealth evaluations.One of the goals of the COE is to develop resources for telehealth providers, staff, and researchers to improve quality of telehealth innovations.8][19][20][21][22][23] The telehealth toolkit is intended to be a dynamic adaptable resource that can be utilized in a variety of telehealth settings by a broad group of telehealth stakeholders.
The objective of this manuscript is to describe the telehealth toolkit development process, content in-cluded in the toolkit, and applications to telehealth D&I evaluation.The toolkit and other COE resources can be accessed publicly at no cost online via the Telehealth Center of Excellence website.

Methods
The development of the telehealth toolkit originated with an interdisciplinary team review of literature related to the evolving resources and tools in the D&I field.This literature review included an inventory of existing published D&I toolkits and examination of key components of D&I evaluation including common study designs, existing frameworks and models, implementation outcomes, and measures to assess those outcomes.Our team then framed the conceptualization of the telehealth toolkit with a series of interdisciplinary team (physicians, health services researchers, nurses, telehealth research coordinators) discussions to outline the format and content, after which two researchers (E.E.J. and K.R.S.) developed the content for each toolkit section.Throughout the process of content development, we followed the AHRQ toolkit checklist for content and usability to ensure sufficient information was presented in a clear format. 15The process of toolkit development was considered exempt from the university IRB.
After the initial toolkit draft was completed, our internal interdisciplinary telehealth team (N = 5) provided comprehensive review and feedback, which was used to update the toolkit content and formatting.Next, we completed an external review to assess toolkit acceptability and usability by telehealth experts, including three reviewers knowledgeable in the telehealth field.Participants reviewed the toolkit and completed an electronic survey.The survey included three items to rate usefulness of the toolkit to different audiences (scale of 1-4), seven questions assessing usability of the toolkit (scale of 1-100) and two questions rating content and clarity for each section of the toolkit (scale of 1-4).Open-ended feedback about each toolkit section was also collected.

Toolkit content
Figure 1 illustrates the sections of the D&I telehealth toolkit and Table 1 describes the content within each section.Each toolkit section corresponds to a step in the D&I evaluation process: refine research question, determine study design, define implementation outcomes, select implementation framework, develop data collection tools and measures, and select implementation strategies.Each section includes consistent content to demonstrate the corresponding D&I evaluation step with introductory materials, practical questions for the research team, telehealth specific considerations, case study examples, and external articles and resources.The section on selecting implementation strategies is unique in that the steps in this section can be utilized throughout the D&I evaluation process to overcome identified barriers to implementation.Three case study examples from South Carolina telehealth programs were included throughout the toolkit, with one illustrative example in each section to further elucidate that specific D&I evaluation step.
The first case study was an evaluation of a schoolbased telehealth asthma program.][22][23] The second case study was an evaluation of a remote patient monitoring program to support diabetes management.While remote patient monitoring programs have exhibited success in improving patient diabetes outcomes, a program in South Carolina demonstrated variable implementation success across clinics. 27The objective for this D&I case study was to characterize clinic delivery strategies for a South Carolina remote patient monitoring program and to examine barriers and facilitators to program implementation in underserved and/or low-income community settings. 19he third case study was an evaluation of a program to treat perinatal and postpartum mental health and substance use disorders.9][30][31] Although evidence-based treatment is available for these conditions, many barriers to care exist.Thus, the objective for this D&I study was to characterize barriers and facilitators to implementing a mental health and substance-use disorder telehealth program in community obstetric and pediatric clinics. 17,18ternal and external review The internal toolkit review guided changes to content, the order of sections, and overall formatting.Table 2 illustrates the feedback from the toolkit external review, which was also utilized to update the toolkit content and format.The toolkit content, clarity, and flow received high rankings from all three reviewers.In addition, the toolkit received high rankings in usability and recommendations for use by others.However, the ability of beginner D&I researchers to independently utilize the toolkit without insight from mentors or other resources received lower endorsement scores.These scores on usability for beginner researchers were considered acceptable as our team intends for this toolkit to serve as a supplemental guide to D&I telehealth evaluation and not replace expert mentorship.
Based on the qualitative suggestions for improvement, we reordered the introduction section of the toolkit and reformatted all tables within the toolkit for consistency.We also added additional clarification in the implementation framework and implementation strategy sections to define terms and concepts more clearly.Lastly, after incorporation of external reviewer feedback, our internal telehealth team reviewed the  toolkit again, revisions were finalized, we created a dissemination plan, a final draft was sent to the formatting team, the formatted version was reviewed by our team, and the toolkit was released as an interactive PDF to the HRSA Telehealth Center of Excellence Website. 32

Discussion
4][35] Moreover, patients and providers appreciate the convenience telehealth affords. 3,36These benefits, however, have neither been universal nor experienced equitably.Research has shown variation among practices in terms of their readiness and the resources available to them to implement telehealth.Similarly, despite telehealth being an effective tool to support health care access in rural communities, rural patients were less likely to engage in telehealth during the COVID-19 pandemic compared with urban patients. 9,37There also have been reported disparities in use of telehealth among different patient populations due to various demographic characteristics and social determinants of health. 8,9,11,34ith the Centers for Medicare and Medicaid Services making permanent many of the telehealth flexibilities authorized during the pandemic and many state Medicaid and private payers following suit, it is evident telehealth will continue to play an important role in the US health care system and the evolving landscape of evidence-based interventions. 38,39D&I research has great promise to help stakeholders understand global processes and complexities related to telehealth clinical practice and support more equitable access to novel telehealth programs across care settings and among diverse populations.
Telehealth researchers and administrators have the opportunity to leverage D&I research to better understand the barriers and facilitators to telehealth implementation and sustainment and how to best adapt telehealth programs to meet their specific contexts.Despite the utility of D&I, many have noted barriers to engagement in D&I research due to lack of training and the diversity of frameworks, approaches, and tools within the field. 14Resources are needed to support Future directions for this telehealth toolkit include continued modification of the toolkit content and format by our interdisciplinary team to meet the dynamic D&I research field.Based on the external review, we can add introductory content to the toolkit in the future to make it more functional for a beginner D&I researcher.Our team is currently tracking ongoing dissemination of the toolkit by monitoring counts of toolkit downloads from the COE website and will plan future evaluations of usability and outcomes generated from toolkit use.

Conclusion
With telehealth becoming a mainstay in the US health care system, it is vital to identify barriers and facilitators to implementation and sustainment of telehealth innovations.The field of D&I offers structured methods to assess implementation processes and outcomes.Developed by a multidisciplinary team of D&I scientists and telehealth experts, the toolkit described in this manuscript provides stepwise resources to guide telehealth teams and stakeholders in D&I evaluation.

Table 2 .
External Review of Telehealth Toolkit uptake of D&I research within different scientific communities.Thus, a primary strength of this telehealth toolkit is that it meets a gap in the literature to provide D&I guidance specifically to the field of telehealth.The toolkit can be utilized in a variety of settings as a template that will inform development, implementation, and sustainment of telehealth innovations.The toolkit can also identify resource and logistical strategies that may be utilized to mitigate telehealth disparities among rural communities.With tools like this to support the development of D&I research questions and rigorous methods to evaluate telehealth outcomes, ultimately additional telehealthspecific D&I research can advance the development and implementation of appropriate and efficacious strategies to reach and provide quality care through telehealth.