Patterns and Correlates of Mis-implementation in State Public Health Practice in the United States

Background: Much of the disease burden in the United States is preventable through application of existing knowledge. State-level public health practitioners are in ideal positions to affect programs and policies related to chronic disease, but the extent to which mis-implementation occurring with these programs is largely unknown. Mis-implementation refers to ending effective programs and policies prematurely or continuing ineffective ones. Methods: A 2018 comprehensive survey assessing the extent of mis-implementation and multi-level in�uences on mis-implementation was reported by state health departments (SHDs). Questions were developed from previous literature. Surveys were emailed to randomly selected SHD employees across the Unites States. Spearman’s correlation and multinomial logistic regression were used to assess factors in mis-implementation. Results: Half (50.7%) of respondents were program managers or unit directors. Forty nine percent reported that programs their SHD oversees sometimes, often or always continued ineffective programs. Over 50% also reported that their SHD sometimes or often ended effective programs. The data suggest the strongest correlates and predictors of mis-implementation were at the organizational level. For example, the number of organizational layers impeded decision-making was signi�cant for both continuing ineffective programs (OR=4.70; 95% CI=2.20, 10.04) and ending effective programs (OR=3.23; 95% CI=1.61, 7.40). Conclusion: The data suggest that changing certain agency practices may help in minimizing the occurrence of mis-implementation. Further research should focus on adding context to these issues and helping agencies engage in appropriate decision-making. Greater attention to mis-implementation should lead to greater use of effective interventions and more e�cient expenditure of resources, ultimately to improve health outcomes.


Background
Recently, there has been an increasing emphasis in public health practice on use of evidence-based decision-making (EBDM) in chronic disease prevention and control (1,2).While the eld has made strides in the expected use of evidencebased practices and programs, there is still a gap in the way decision-making practices occur related to ending non-evidence based programs and continuing evidence-based programs (EBPs) (3,4).Since much of chronic disease burden is preventable (5)(6)(7), gaps in delivery of EBPs hinders effective public health practice to improve health.
Mis-implementation is an emerging area of interest for public health practitioners and researchers (8-10).The term refers to the inappropriate termination of evidence-based programs or the inappropriate continuation of non-evidence based programs (8).An example of inappropriate termination of an evidence-based policy in the United States is notable with the rollback of Bush and Obama era healthy school lunch standards (11), which were relaxed despite evidence they increased school-aged children's consumption of healthy foods (12).Alternately, an example of inappropriate continuation of nonevidence-based programs is the continued use of health fairs for community screenings, interventions and education.While they may help increase visibility of services to subsets of the community, there is limited evidence that they increase screening follow-up, enhance sustained health knowledge, or improve health outcomes (13,14).Previous studies have suggested that between 58% and 62% of public health programs are evidence-based (15,16).Even among programs that are evidence-based, 37% of chronic disease prevention staff in state health departments reported programs are often or always discontinued when they should continue (10).However, these studies did not further explore the contributing factors to these decision-making processes and did not assess the degree to which mis-implementation is occurring in chronic disease public health practices.
Exploring the evidence-based decision-making (EBDM) and related literature suggests that a mix of individual, organizational, agency and policy related factors are at play in organizational decision-making, including whether or not to begin or continue implementing programs, and program outcomes (1,10).EBDM which is an approach to decision-making that combines the appropriate research evidence, practitioner expertise, and the characteristics, needs, and preferences of the community, can have a signi cant impact on health-related outcomes (1,17).Speci cally, leadership support in applying EBDM frameworks can enhance an organization's capacity for improved public health practices (1,18,19).Concurrently, contextual factors, cost burden and characteristics of early adopters are also factors in mis-implementation outcomes (16,20).These concepts support the factors that inform our original mis-implementation framework (8).In a cross-sectional U.S. study of local health departments, higher perceived organizational supports for EBDM were associated with lower perceived frequency of inappropriate continuation (21).In cross-country comparisons of mis-implementation involving Australia, Brazil, China, and the United States, leadership support and political contexts were common factors in whether chronic disease programs continued or ended inappropriately across four countries (22).
State health departments (SHDs) are a signi cant driver of public health programs within the United States.Most federal funds for chronic disease prevention are directed through state health departments, and they provide resources and guidance to local level implementation of public health programs (23).This dynamic of the SHD as the pass-through organization means that their organizational dynamics are key in the successful outcomes of these programs.And while an estimated $1.1 billion dollars ow through state public health chronic disease and cancer prevention programs annually, a majority of these funds focus on secondary prevention (e.g., cancer screening), leaving a scarcity for primary prevention resources (24,25).With this scarcity in prevention funding, it is essential that every dollar being directed towards chronic disease programs have maximum impact.
This study seeks to: 1) assess the extent to which mis-implementation of chronic disease programs is occurring in state health departments, and 2) identify the most important factors associated with mis-implementation among programs overseen by SHDs.

Methods
This study is a cross-sectional assessment of decision-making practices within state health departments.We surveyed current SHD employees across the U.S. to gather quantitative data to identify the perceived frequency and correlates of misimplementation within SHD chronic disease units.Human subjects approval was obtained from the Washington University in St. Louis Institutional Review Board (#201812062).

Survey Development
To develop a survey informed by the literature and addressing knowledge and survey gaps, we undertook an extensive literature review.Survey development was also guided by the study team's previously described conceptual framework to ensure measures included EBDM skills, organizational capacity for EBDM, and external in uences such as funding and policy climate (8).
A literature review of several databases (i.e., PubMed, SCOPUS, Web of Science) was conducted to search for existing survey instruments regarding organizational decision-making.Identi ed measures were summarized according to setting, audience, psychometric properties, and survey question themes.From our review of 63 surveys, we ended up selecting items from 23 measures to examine in relation to our conceptual framework (8- 10,18,[26][27][28][29][30][31][32][33][34][35][36][37][38][39][40].Questions pertaining to political in uence and mis-implementation decision-making were iteratively developed and re ned as there was little published literature available at the time to inform these questions.Drafts for questions in each domain (individual skills, organizational/agency capacity, mis-implementation decision-making, external in uences) were updated, and underwent three separate reviews by the study team and a group of practitioner experts to develop a nal draft of the study instrument.
The nal draft survey underwent cognitive response testing with 11 former SHD chronic disease directors.Reliability testretest of the revised draft with 39 current SHD chronic disease unit staff found consistency in scores and only minor changes to the survey were needed.

Measures
Survey measures addressed the following topics: participant demographics, EBDM skills, perceived frequency of misimplementation, reasons for mis-implementation, perceived organizational supports for EBDM, and external in uences.
External in uences included perceived governor o ce and state legislative support for evidence-based interventions (EBIs), and perceived importance of multi-sector partnering.Exact item wording is provided in the national survey located in Appendix 1. Survey questions for EBDM skills, organizational supports, and external in uences consisted of 5-point Likert scale responses.Response options ranged from either "Strongly Disagree to Strongly Agree" or "Not at all" to "Very great extent".
Perceived frequency of mis-implementation was assessed with two questions: "How often do effective programs, overseen by your work unit, end when they should have continued"; and "How often do ineffective programs, overseen by your work unit, continue when they should have ended."The response options were: never, rarely, sometimes, often, and always.These variables will subsequently be referred to as inappropriate termination and inappropriate continuation, respectively.

Participants
Participants for the survey were recruited from the National Association of Chronic Disease Directors (NACDD) membership list.The NACDD membership lists consists of SHD employees working in their respective chronic disease units.Participants were randomly selected from the membership roll after individuals from territories and non-qualifying positions (administrative support & nancial personnel) were removed.Emails were sent out in June 2018 inviting a randomly selected sample of 1239 members to participate in a Qualtrics online survey.Participants were offered the option of a $20 Amazon gift card or to have us make a donation to a public health charity of their choosing.A follow-up email was sent two weeks after the initial email with a reminder phone call a week later.Non-respondents could have received up to three reminder emails and two reminder voicemails or a single phone conversation to address questions.The online survey closed at the end of August 2018.

Data Cleaning and Analysis
Respondents who answered any of the questions beyond demographic questions were included in the sample.State-level variables, such as population size, funding from the Centers for Disease Control and Prevention (CDC) (the major funding source for state chronic disease control), and state governance type, were added to the data set from other publically available datasets such as the CDC grant funding pro le, Association of State and Territorial Health O cials (ASTHO) State Pro les and Public Health Accreditation Board data (24,41,42).Dichotomized versions of Likert scale variables were created given the limited distribution of responses across the original scale and to facilitate interpretation.Responses that included Agree or Strongly Agree were coded as 1 while all other remaining responses were coded as 0.
Descriptive statistics were calculated for all variables in SPSS version 26.To assess associations, a Spearman's correlation was calculated between each non-dichotomized mis-implementation variables and the individual demographics, individual skills, organizational capacity for EBIs and external factors.Multinomial logistic regression was then used to assess how variables were predictive of mis-implementation outcomes.The dependent variables (inappropriate termination & inappropriate continuation) were re-categorized to 1) often/always 2) sometimes and 3) never/rarely (reference category).
Multinomial regression was used as the assumption of proportional odds was violated with an ordinal regression.The independent variables were dichotomized (as described above).Two separate models were t: the rst assessing inappropriate termination among programs overseen by SHDs and the second assessing inappropriate continuation among programs overseen by SHDs.We decided two separate models were appropriate as inappropriate termination and inappropriate continuation are two different phenomena within the overall mis-implementation concept.An initial model for each of the two dependent variables was run for each domain with all their respective variables included.All variables shown to be signi cant in these rst runs of the model were then added to a nal version of each model (inappropriate termination and inappropriate continuation).

Demographics
The nal response rate was 48.3% (n=643).There were respondents from every state, but the number of responses per state was not proportional to state population size.In the interest of con dentiality, responses were grouped by ASTHO de ned regions (41), and there was a relatively even distribution of participants across regions (Table 1).Half (50.7%) of the respondents were program managers and on average had been in their position for over six years.Most respondents worked across multiple health areas with cancer as the most represented program area.Thirty-ve percent of respondents had a master's or higher degree related to public health.

Mis-Implementation Patterns
When asked "How often do effective programs, overseen by your work unit, end when they should have continued," 50.7% of respondents indicated sometimes, often or always (Table 2).Respondents were asked to choose the top three reasons for effective programs ending (but not in a ranked order).The most common responses were: funding priorities changed/funding ended (87.6%); support from leaders in your agency changed (38.9%); support from policy makers changed (34.2%) and program was not sustainable (30.2%) (Table 2).
Regarding inappropriate continuation, when asked "How often do ineffective programs, overseen by your work unit, continue when they should have ended," 48.5% of respondents indicated sometimes, often or always.Respondents were also asked to choose the top three common reasons for ineffective programs continuing (not in ranked order).The most commons responses were: funder priorities to maintain program (43.4%); policy makers' request or requirements to continue (42.9%); agency leadership requests to continue (37.9%); and standard is to maintain status quo (36.5%) (Table 2).

Mis-Implementation Correlates
The number of years a participant had been working in their current position (r= -0.11), years they had been working at their agency (r= -0.09) and years they had been working in public health (r= -0.10) were shown to have small negative signi cant correlations with inappropriate continuation (Table 3), meaning more years of experience were associated with lower likelihood of inappropriate continuation.None of the individual skills were shown to have a statistically signi cant association with either inappropriate termination or inappropriate continuation.All of the organizational capacity variables were shown to have a small negative signi cant association with both mis-implementation variables, meaning higher perceived organizational capacity was associated with lower perceived frequency of mis-implementation (Table 3).External variables related to lawmakers' priorities and support were shown to have small negative signi cant, associations with both the inappropriate termination and inappropriate continuation variable.
In the nal model for inappropriate termination (Table 4) the largest effects were shown for having individual skills to modify EBIs from one priority population to another (OR=3.24;95% CI=1.19, 8.85); reporting that the number of layers of authority impedes decision-making (OR=3.23;95% CI=1.61, 7.40); and leadership preserves through ups and downs of implementing EBIs (OR=0.16;95% CI=0.07, 0.34).In the nal model for inappropriate continuation, the largest effects were shown for reporting that the number of layers of authority impedes decision-making (OR=4.70;95% CI=2.20, 10.04); use of economic evaluation in decision-making (OR=0.35;95% CI=0.17, 0.73); and leadership competence in managing change (OR=0.26;95% CI=0.13, 0.53).

Discussion
A set of organization/agency capacity factors demonstrated more consistent association with mis-implementation outcomes than individual skills of staff.These factors demonstrated an inverse relationship with mis-implementation outcomes (e.g., as agency capacity increased, the association with mis-implementation rates decreased).These ndings are consistent with our earlier study among US local health departments, which found organizational supports for EBDM were associated with lower perceived frequency of inappropriate continuation (21).This suggests agency culture and capacity are signi cant protective factors against mis-implementation in multiple public health organizations rather than the skills of individual staff.Importantly, the agency-level variable reporting that the number of layers of authority impedes decisionmaking about programs continuation or ending was found to be strongly associated with both inappropriate termination and continuation.This suggests that highly vertical organizations may be more vulnerable to ineffective decision-making around program continuation or ending.
Outside of funding, the primary correlates for inappropriate termination or continuation were changing support from leaders and policymakers.We saw more variability in the reasons for inappropriate continuation versus termination.Inappropriate termination was heavily skewed towards funding priorities changing or ending, which is to be expected given the predominance of state public health programs based on time-limited grant funding (43).The top four reasons for inappropriate continuation were more spread out across multiple domains.This variability in reasons could demonstrate that the processes that result in an ineffective program continuing may tend to involved multiple domains, but this also allows for more opportunity for modi ability.
The two factors most strongly negatively correlated with inappropriate continuation related to leaders' exibility were: work unit's leaders are competent at managing change (r=-0.30);and leadership reacts to critical issues regarding the implementation of EBIs (r=-0.30).The two factors most strongly correlated with inappropriate termination were: work unit leadership react to critical issues regarding the implementation of EBIs; work unit leadership encourages planning for sustainability of programs (r= -0.28, -0.27 respectively).Again, this suggests agency culture and leadership are strong drivers of mis-implementation outcomes but more speci cally how leadership can be related to the importance of EBI use and exibility in program implementation and adaption.
Our ndings are largely consistent with the literature in EBDM.Reviews found organizational climate, leadership support, staff commitment and skills, adequate sta ng and low staff turnover, organizational resources, and partnerships affect EBI sustainability (4,44,45).Policy and legislation are also associated with sustainment of programs in community, clinical and social service settings (45).Engaging community leaders and other policy makers throughout programmatic decisionmaking can increase likelihood of program sustainment (44).While de-implementation of ineffective clinical tests or services has been studied, there is sparse parallel literature on de-implementation of ineffective public health programs (8, 46-48).

Limitations
Our response rates across states was varied enough that we were not able to study state-level correlates in detail.In the absence of other organizational and administrative data, this study relied on self-report surveys of individual and perceived organizational characteristics.While we asked respondents their level of involvement in decision-making, they were not always in the position to be privy to the reasons about decision-making or they joined the agency after a decision about a program had concluded.
Compared to previous pilot work, perceived frequency of mis-implementation in SHD was higher in this study (36.5% vs 50.7% for inappropriate termination and 24.7% vs 48.5% for inappropriate continuation), although some of this difference may be attributable in part to updates to the mis-implementation survey item de nitions and changes in the approach to categorization of responses (9,10,21).In earlier studies, the recoded dichotomized mis-implementation variables only included the often/always response.After examining the distribution of the mis-implementation variables responses, we thought it was important to include the "sometimes" response in categorizing mis-implementation because "sometimes" still captured the phenomena occurring and that excluding it could potentially leave out nuances in the data.

Future Directions
These results provide a rst look at factors that may be related to the phenomena of mis-implementation in public health practice.Later phases of this study include eight case studies highlighting lessons learned around mis-implementation and agent-based modeling to identify the dynamic interactions between the individual, organizations and contextual factors and disseminate them back to the state health departments (8).These models should provide decision-making tools to better facilitate evidence-based decision making.
There is also a need to explore mis-implementation in other public health settings.While our study focuses on SHDs, local health departments and non-pro t settings are signi cant implementers of public health programs.There is also sparse information on how mis-implementation may vary across public health program areas (e.g., chronic disease, infectious disease, maternal and child health).

Conclusion
While our understanding of mis-implementation in public health practice is in an early stage, our ndings provide practitioners and applied researchers some actionable ndings.For example, based on our study and related literature (18, 49, 50). it appears that e ciency and effectiveness may be gained via attening of public health agencies along with an organizational culture that supports EBDM.Given the emergence of evidence that chronic diseases are a signi cant moderating factor in outcome of timely disease concerns i.e.COIVD-19 and Cancer (51, 52), suggestions like these could help maximize dollars spent on public health programs ensuring that appropriate evidence-based programs are contributing to improved health outcomes and bene ting the communities they serve.
Abbreviations SHD= State Health Department

Table 2 .
As de ned by Association of State and Territorial Health O cials (41) Formal public health degrees include: BSPH, MPH/MSPH, DrPH or PhD in a Public Health eld Reported frequency and reasons for mis-implementation a survey of U.S., 2018 a b Could indicate more than one response c a Respondents could choose up to 3 reasons so percentages will add up to more than 100%.List is arranged by top responses in descending order of frequency a reason was selected; complete questions are shown in Appendix 1.

Table 3 :
Practitioner, Organization, and External Correlates of Mis-Implementation in U.S. SHD chronic disease units, 2018 (N=613).It is important for my work unit to develop partnerships with both health and other work sectors to address out state's health issues.5-point Likert scale used.1-Strongly Disagree, 5-Strongly Agree b 5-point Likert Scale used.1-Not at all, 5-Very great extent a

Table 4 :
Mis-Implementation Predictors among programs overseen by U.S. state health department chronic disease unit staff, (N=613).