Introduction

It is widely recognised that adoption of national guidelines remains variable and guideline adherence programmes have had limited success in improving it.[1–3] There has recently been “a wave of optimism in implementation science” that applying theory will lead to programmes with greater chance of success.[4] Similarly, there is a strong drive for theory-driven evaluation of approaches to get evidence into practice, to the extent that describing programme theory is required for peer-reviewed publication of quality improvement projects.[5] In practice, however, many ‘real-world’ programmes (i.e. where guidelines are implemented outside a research context) rarely develop programme theory prospectively.[6, 7] Opportunities to capture learning from such initiatives are limited, partly because they may not meet standards for peer-reviewed publication without articulated theory in either the programme or its evaluation. Moreover, in many apparent theory-driven evaluations, theory has not in fact shaped evaluation questions, informed methods or interpretation in any visible way.[8] Therefore questions remain about whether and how a theory-driven approach can productively influence evaluation and/or implementation initiatives in real-world circumstances.

In this paper we describe a retrospective evaluation of a ‘real-world’ guideline implementation programme using a theory-driven approach. We then use this case to provide generalizable learning and reflections on the feasibility and value of theory construction in evaluating real-world, guideline adherence programmes and how theory construction may, or may not, lead to greater programme success.

Materials and methods

Design

This study was started after the completion of all YiL activities. We combined a range of qualitative and quantitative methods to address the study’s objectives.(Fig 1)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Mixed methods design to address evaluation objectives.

https://doi.org/10.1371/journal.pone.0174086.g001

Results

We first describe programme theories identified from planners’ ideas ex ante and their iteration over time, focusing on three core elements of theory: programme components, expected outcomes and ideas about mechanisms for change. After setting out two testable theories derived from these ideas, we report findings of each quantitative analyses. Finally, we discuss the evidence drawn from all the data for three possible candidate explanations for observed effects post-implementation.

Testable theories, timeline and major contextual changes

The two testable theories from the ‘early’ and ‘evolved’ ideas are described below and shown on a schematic timeline, together with evaluation periods and major contextual changes.(Fig 2)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. YiL implementation timeline, evaluation periods and programme theories (early and evolved).

Triangles = programme events or blocks of activity (hollow = preparatory, solid = during the programme), crosses = contextual events.

https://doi.org/10.1371/journal.pone.0174086.g002

2a. Testing the hypotheses supporting the theories using quantitative analysis

Early theory.

‘Early’ theory predicts that YiL would result in greater reductions in EHA in YiL boroughs than in other comparable boroughs over the same timescale. These reductions would lead to cost savings.

There was no evidence that YiL reduced emergency admissions more than comparator areas. Rates were not significantly different in YiL boroughs than comparators before, during or after the programme (Table 1). Results were similar in all alternative analyses. (Table A and Table B in S1 Supporting information)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Rate ratios and 95% confidence intervals for YiL boroughs versus comparators in each time period.

https://doi.org/10.1371/journal.pone.0174086.t001

Evolved theory.

‘Evolved’ theory predicts effects would be greater where there was greater practice participation in YiL activities. It extends ‘early’ theory by specifying outcomes of implementing the aspects of guidelines that YiL prioritised, i.e.:

• improving the quality of diagnosis by confirmation with the use of post-bronchodilator spirometry
• encouraging use of rescue packs by distributing a self-management plan leaflet
• increasing pulmonary rehabilitation referrals for patients with severe COPD
• reducing prescribing of inhaled corticosteroids without other medications.

In common with ‘early theory, it predicts that YiL would reduce EHA and thus reduce associated costs.

YiL Participation by practice: Overall, 15/183 (8%) practices participated in all possible educational activities and had the template installed, with 30% having medium-high participation and significant variation by borough (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Practice participation in YiL: distribution of scores by borough.

Number of interventions each practice undertook. Measured by: evidence of attendance by practice staff at YiL educational events (Spirometry training, Nurse Mentorship programme, Master Classes) or whether e-template for COPD was uploaded on practice computer.

https://doi.org/10.1371/journal.pone.0174086.g003

Adherence to NICE guidelines COPD diagnosis and management: Greater participation in YiL was associated with greater improvements in appropriate diagnosis and management advised by the guidelines, as measured by recorded post-bronchodilator spirometry, provision of a self-management plan and pulmonary rehabilitation referral.(Table 2)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Difference in differences results comparing ‘before and during’ vs ‘after’ periods of YiL for practices that scored 2–4 vs those who scored 0–1.

https://doi.org/10.1371/journal.pone.0174086.t002

YiL implementation insufficient for clinically meaningful change

In theory 2, planners believed active programme participation was necessary for providers to have the skills and tools to adopt the COPD guidelines in their clinical practice. Consistent with this theory, providers in seven out of 10 practices recalled YiL and described impacts of the educational activities, not only in terms of increasing their skills but three practitioners described it enabled them to apply their skills and three practices continued to use the YiL self-management leaflet and template:

We’ve had our nurse do some training, so we’re certainly picking up more cases. (GP, Practice 7)

There’s so much to spirometry … when I went and did the COPD course with XXX and then on that basis I did speak to the GPs that really we need to increase our prevalence then we need to get our own spirometry, so we did. (Nurse, Practice 1)

Again, consistent with planners’ evolved theory that sufficient active programme participation was needed for clinical impact, providers that recalled the programme but opportunities to participate were limited (e.g. practice staff could not attend training in work time or where they were not able to persuade practices to purchase spirometers), and reported benefits were not sustained. In addition, there were low numbers of practices with high active participation and many practices where we requested or obtained an interview did not recall the programme at all.

However, incompatible with this theory, providers and planners and documentation indicated programme participation did not always lead to higher confidence or motivation to adopt guidelines. Practices with similar levels of YiL participation expressed varying levels of confidence in managing COPD, and none of them considered YiL as the sole, or even the main determinant of better COPD management. In fact, some practices expressed considerable scepticism about the value of the feedback based on routine data, partly because of data quality concerns, but also because of suspicions about how analysis originated:

We received your analysis of the COPD performance indicators at our practice and would like to know where the information was extracted from as it is contradictory and certainly is not equivocal [sic] to [our IT system]. I don’t think we have met, and no one here seems to know you, could you please let us know which company you are from? (Practice email to YiL planners 6/2011)

Distributing feedback on performance via email in particular was not sufficient to prompt change and may actually have alienated some practices, at least initially:

By the end of the role, people were desperate for the data. They wanted to look at it but to begin with I had a lot of meetings where they were really put out that we’d measured them without asking them. (Planner 1, clinician)

Planners’ and providers’ beliefs about mechanisms of change differed

In both early and evolved theories, planners sought to reduce costs through reducing emergency hospital admissions in COPD patients. Providers were generally committed to this aspiration, but considered that factors other than primary care quality were more important influences on the risk of emergency COPD admissions.

Several providers described patients’ health behaviours as most important to alter their disease trajectory to reduce the need for healthcare. This view was also echoed by one planner after the programme had finished:

You can get nurses and doctors delivering better care but until you change the behaviour of the patient all you are doing is recording data. Because until the patient stops smoking and actually does pulmonary rehab and understands their condition, they’re not going to get any better. (Planner 1, clinician)

YiL focused on improving COPD management in primary care but providers viewed social and psychological factors as major reasons why patients attend A&E when they become unwell.

Some people keep presenting because, well, they’re lonely and they’re unwell too, but they’re anxious and they need reassuring.(Nurse, Practice 5)

Providers also described how community and hospital services influenced the likelihood of admissions:

I have said to the respiratory nurse at XXX Hospital, because his, his argument was ‘you need to stop them coming in’ as in me. And I'm like ‘well you don’t have to admit them, you know, just because they walk in your door and they know what symptoms to tell you about and how to breathe, does not mean you have to admit them.’ (Nurse, Practice 9)

Therefore, while providers supported the programme’s overall aim, they did not perceive their role in providing primary care in line with guidelines was key to achieving it.

Other drivers of performance swamped programme impacts

YiL was a local programme but during 2011–2014, there were widespread efforts to improve COPD nationally[9] and regionally which raised the profile of COPD. There was evidence some practices were more strongly influenced by these drivers than YiL. For example, one entire borough declined the YiL mentorship component because “they were doing other COPD projects” and some providers that failed to recall YiL recalled other London-wide COPD interventions. Most practice staff reported the national GP contract (Quality and Outcomes Framework (QOF)) was a major influence on their time and effort in COPD:

Wednesday morning is my QOF clinic. I don’t see patients, I spend time just looking at the registers. (Nurse, Practice 1)

Planners sought to use these national drivers but were also aware of where their goals were not aligned. For example, the clinical lead and implementer both described how annual reviews, for which practices were remunerated by QOF, did not on their own, motivate practices to include elements recommended in guidelines:

There was no QOF payment for doing the right thing, it would still be a QOF payment for doing [a review] and no one really worried too much about it. (Planner 2, clinician)

Just as there were drivers beyond YiL towards better COPD care, there were some barriers to improving care outside YiL too. National healthcare reforms changed NHS funding and organisation and constrained stretched practice nurse capacity:

We are very thin on the ground with practice nurses in north-east London and they’re like, it’s easier to get another doctor into the practice than it is to get a decent practice nurse who’s trained up. (GP, Practice 3)

Practices along with everywhere in the public sector, it’s being squeezed and squeezed, in the last two, three, four years, we have been expected to do more and more and more. (Nurse, Practice 10)

As a result, several providers reported struggling to maintain skills and sustain the specialised activities needed to diagnose and monitor COPD patients according to guidelines.

Discussion

Learning for evaluations

We sought to derive theory through making explicit the tacit assumptions and ideas of those close to the programme in interviews after the programme and using contemporary programme documentation. This approach is advocated by Chen and others to ensure the evaluation framework is meaningful to stakeholders and measures what is important in programmes.[8]

Coryn et al’s review of 45 evaluations found that many claiming to be theory driven do not use theory “in any meaningful way” for formulating questions or informing evaluation design.[8] In contrast in this evaluation, we formulated questions based on both early and evolved constructed programme theories. It directly shaped the measures of programme participation which we used to generate comparator populations to test hypotheses based on the second theory. It also enabled us to cost the programme and its possible impacts comprehensively, a recognised limitation of previous guideline implementation studies.[22] This meant we could address some of the challenges of conducting robust evaluations of guideline implementation strategies in service settings: lack of clarity about programmes’ components and aims; evolving delivery; differential participation; without prospectively defined control groups.[12, 23]

The theory we derived however, was subject to some limitations. It was difficult to distinguish whether evolved theory captured the programme’s iteration during its lifetime or only reflected planners’ views formed after the programme had finished. Without this knowledge, it is not possible to ascertain the extent to which planners could have modified the programme in light of these realisations. Dixon-Woods et al note the limitations of constructing theory without process evaluation fieldwork data on programme evolution.[17] We contend that in many projects (like YiL) formal process evaluations are unlikely and illustrate how theory can be derived and critiqued where those close to the programme have had the courage to share documentation that open their early ideas and actions to scrutiny. Another approach would have been to construct programme theory retrospectively by applying prior theory as other evaluators have done.[8] Nilsen and others suggest middle-range theories could be used for this purpose but also recognise one theory may only provide partial understanding. Moreover, as Coryn and others warn, deriving programme theory from established theories (e.g. Theory of Planned Behaviour) may skew evaluation towards outcomes within pre-existing theory rather than what was intended by the programme.[3, 8]

There were also weaknesses in our evaluation methods that a theory-based approach could not eliminate. When the programme was implemented a control group was not included in the design (e.g., areas included in the study but not receiving the programme). Therefore it was not possible to judge the extent to which the lack of impact on admissions was due to insufficient implementation across the practices, determinants of admissions outside of YiL, or that major impacts from national system reforms/incentives swamped YiL’s impacts. This is a common problem in system-wide programme evaluations conducted in ‘real-world’ settings where experimental designs are not appropriate or feasible. To address this, we employed a difference-in-differences design, using comparator practices that had little or no participation in the YiL programme. This was still less robust than a control group because all practices had some exposure to YiL. Inaccuracies and incompleteness in routinely collected primary care data limited the conclusions we could draw from quantitative analyses. As a result, it was not possible to distinguish whether providers actually changed patient care to adopt NICE guidelines or just ensured that existing activity was better recorded. Our YiL participation index, based on registers of educational activity attendance, did not capture other important but undocumented aspects of YiL involvement (e.g. practice discussions), nor did it capture variations between staff within practices.

Value of theory in designing interventions: Learning for planners/designers

It has been widely argued that articulation of theory should lead to more effective programmes.[4, 13] There are frameworks and process models developed for such articulation[4] but programme designers can be reluctant to develop theory and may struggle to see its value.[7] Moreover, there is little empirical evidence from programmes developed outside research that theory does lead to better programmes.

Our study did not set out to test whether prospective theory articulation could improve programmes’ chances of success because it was a retrospective study. However, it suggests ways in which prospective theory might or might not have improved this programme that may be generalizable to other initiatives.

On one hand, retrospective theory construction made clear that the programme started with a lack of theory, particularly concerning mechanisms of change. In particular it highlighted an important flaw in early theory concerning programme implementation (i.e. unanticipated problems with using routine data for feedback to drive change). Planners recognised the flaw soon after the programme’s start. They changed implementation tactics (improving data quality, extraction and feedback mechanisms) and altered their underlying ideas about how change would occur (recognising the need for training and tools to enable practices to improve care). In this case, prospective theory articulation could have forced planners to make explicit the mechanisms by which they thought change could occur. There is a wealth of existing evidence on data and feedback that could have informed this aspect of the programme.[24, 25] Moreover, existing implementation frameworks or process models could prompt programme planners or designers to break down the steps involved in implementing programmes[7] and consider factors such as organisational readiness for change[26] to anticipate and address implementation problems before a programme starts.

On the other hand, formally setting out theory at the beginning of YiL–even using existing implementation models—was unlikely to have altered planners’ aspiration to reduce emergency hospital admissions. This aspiration looked reasonable at the time YiL was instigated; it was assumed in national guidelines and COPD strategy documents that reduced admissions through better primary care was ‘likely’.[27, 21] This assumption—based on professional consensus not empirical data–has since been questioned, in that few initiatives have succeeded in reducing emergency admissions.[28–33]

NICE COPD guidelines are not unusual in lacking underpinning empirical evidence. Guidelines are based on robust empirical evidence of patient benefit where possible, but they are still formed where empirical evidence is lacking. Even where underpinning evidence is available, an evidence base drawn from one population may not be applicable to a different target patient population.[34, 35] Applying implementation theories or frameworks, which generally do not go beyond guideline implementation to consider health outcomes, would not prompt planners to challenge the feasibility of achieving health outcomes either.

Renger et al speculate that programme designers may also fear “making things explicit” through articulating programme theory.[6] For YiL, reducing emergency hospital admissions was central to several planners’ motivation to initiate the programme, above and beyond guideline adherence. This was (and remains) an NHS policy priority.[36, 37]. Therefore, planners could have been reluctant to challenge how and whether it was possible to achieve it. More broadly, our evaluation suggests that loose, informal theory may have kept several planners with different ideas involved in YiL while they all learnt more about the context and implementation of the programme. Confronting a lack of consensus early in a programme where there are many unknowns could alienate some planners and put a programme in jeopardy.

Conclusions

In the UK, it is expected that local services will promote adherence to national guidelines but programmes to implement such guidance often have limited success. Moreover evaluations of programmes conducted in real-world settings are often not captured in evidence reviews. Our theory-based evaluation of a real-world programme indicates that reconstructing theories used by programmes planners can provide the basis for more robust evaluation.

It is also widely argued that applying theory would lead to better programmes. Our evaluation supports the call for planners to prospectively articulate theory, drawing on existing evidence of successful implementation, theories and models to strengthen implementation of guideline adoption efforts. However, planners’ motivations to initiate such programmes (and providers to implement guidelines) depend on the belief that implementation will improve health outcomes. Where this supporting evidence is weak or uncertain, prospectively articulating or using existing theory may not lead to more successful programmes.

Supporting information

Acknowledgments

We would like to thank

• The evaluation steering group for their invaluable input in commenting on the design and methods of this evaluation throughout and providing project documentation (Prof Mike Roberts, UCL Partners & Lead for the UK & European COPD Audit Programmes, Rob Meaker, Director for Innovation, Barking & Dagenham, Havering & Redbridge CCG cluster, Dr James Mountford, Dr Phil Kozcan, Professor Martin Marshall, UCLP, Dr Rami Hara, GP & Barking and Dagenham CCG respiratory lead).
• All those who agreed to be interviewed for this evaluation and took part in a CLAHRC North Thames workshop on interim findings.
• Barking & Dagenham & Havering Breathe Easy Groups for their perspectives on COPD care services and general practice.
• Stuart Bond (Health Analytics) and Simon Lam (Barking & Dagenham, Havering & Redbridge CCG cluster) for providing linked routinely collected data for the ‘evolved’ theory quantitative analyses.