A mixed-methods process evaluation of an integrated care system's population health management system to reduce health inequalities in COVID-19 vaccination uptake

Phase 1 – mixed methods evaluation of PHM usage in the context of reducing inequities in COVID-19 vaccine delivery

The national COVID-19 vaccine programme required all health and care systems to work rapidly to mobilise vaccine delivery mechanisms, and collaboratively monitor and ensure high uptake across their regional population (Table 1). North Central London Integrated Care System (NCL) developed the COVID-19 Vaccination Dashboard in December 2020 to help promote uptake and reduce inequities in delivery of the COVID-19 vaccine (Watson et al., 2022). We selected this Dashboard as the evaluation's focus because NCL was keen to understand whether and how PHM helped them reduce inequities in vaccine delivery and thus reduce health inequalities. There is good evidence that vaccine delivery protects against COVID-19 acquisition, and against severe disease, hospitalisations and mortality in those who do contract it (Dagan et al., 2021). Therefore, it is credible to propose that reducing inequities in COVID-19 vaccination delivery could reduce inequalities in both healthcare use and health outcomes. Figure 1 gives an illustration of the Dashboard, with more information on its construction and design in the protocol (Watson et al., 2022).

Qualitative data collection and analysis

Our objective was to describe how (or whether) staff across organisations used evidence of inequities in uptake available in NCL's COVID-19 Vaccination Dashboard to shape delivery or care.

Interviews were conducted with health and care staff across NCL who used the COVID-19 Vaccination Dashboard, either directly or indirectly via an intermediary. Potential participants were initially identified through NCL's PHM programme leads' recommendations of staff that would be expected to use the Vaccination Dashboard in their role. Then, a snowball approach was taken with those who agreed to take part supplemented with additional advertising through a local GP news bulletin and COVID-19-related working groups.

All interviewees gave written consent to take part. Four members of the evaluation team conducted interviews. Normalisation Process Theory (NPT) was used as a sensitising framework to structure a topic guide to capture staff perceptions and experiences of the Dashboard (Murray et al., 2010). After piloting, three versions of the topic guide were produced for use with different participants based on whether they used the tool directly or indirectly, and whether they had access to personal identifiable data. Interviews were conducted on MS Teams, recorded and transcribed. Additionally, interviewers made brief notes about their interviews, capturing their impressions in “pen portraits”.

Transcripts and pen portraits from interviews were analysed alongside contextual documentation comprising correspondence from NCL about the dashboard, and personal emails from participants about the Dashboard shared with interviewers. These data were analysed using the Framework method (Gale et al., 2013). We used an initial coding framework based on the four constructs of NPT which was adapted inductively in response to the data (May and Finch, 2009) (Supplementary Table S1). Two members of the team conducted coding on all transcripts (GW, CM). The wider evaluation team read key transcripts and advised on coding refinements and descriptive analysis.

Quantitative methods

Objective: To describe staff usage of NCL's COVID-19 Vaccination Dashboard, including differences over time and by organisation.

Data: CM sought anonymised staff usage data already stored within NCL's PHM system from the time of its launch in December 2020 until November 2021. NCL analysts extracted data on all staff registered to access the Dashboard from NCL's PHM platform. Multiple iterations of the dataset were produced by CM in partnership with the NCL analysts, for example, to correct organisation names, to assign Dashboard users to consistent organisational categories, to assign each user to the correct user access category (i.e. that took into account their rights to view individual- or population-level data) and to clarify how “activity” recorded in the PHM system's metadata corresponded to users' views of, and actions on, the Dashboard.

Analysis: Data were cleaned in Excel and imported to Stata 15 for analysis (StataCorp, 2017). Local and national policy documentation were used to identify key time points in the vaccination programme and in the development of the Dashboard (Table 1). We sought to describe usage by presenting the numbers of staff using the Dashboard as a proportion of those given access (i.e. those eligible to use it). This was not possible because data were missing for many staff that been granted access to the Dashboard but had not used it. In addition, it became clear that staff were given access to the Dashboard at different time points, and some would have lost access during the lifetime of the Dashboard (e.g. because of leaving the organisation) but date of granting or losing access for each user was not captured in the system. The variables constructed for the analysis are described in Supplementary Table S2.

Phase 2 – development of programme theory through mixed-methods synthesis and interpretation

An initial programme theory was developed through reviewing the analysis of both the quantitative and qualitative components of the case study evaluation and through discussing findings with NCL staff and stakeholders between December 2021 and June 2022.

A workshop was held in June 2022 with individuals working in PHM nationally and internationally to extend the programme theory for how integrated health and care systems could use PHM to reduce health inequalities. Initial discussions with the NHS England PHM development programme, national and international experts (n = 10) identified leaders in PHM using different approaches in different contexts. It also provided examples of how integrated systems were seeking to use PHM to inform individual patient care, service delivery and inter-organisational commissioning. Workshop participants (n = 50) were identified and invited through several routes including the NHS England PHM development programme, regional research networks and national public health networks. The workshop was held simultaneously in person and on Zoom and recorded, with Sli.do software used to capture participant comments and questions (Sli.Do, 2022). The recording of stakeholder discussions and poll results were used in conjunction with the local evaluation findings to develop a generalised programme theory.

Phase 1 sample

Interviews were conducted with 19 health and care professionals across a range of organisations in the ICS involved in the planning or delivery of the COVID-19 vaccination programme. Participants represented a range of professional groups working in primary care, acute trusts, public health teams and the Clinical Commissioning Group (CCG) across all five boroughs in NCL (Table 2). The final quantitative dataset is summarised in Table 3.

Phases 1 and 2: programme theory

A model of the programme theory is illustrated in Figure 2. It is structured under the headings of context, mechanisms and outcomes, with arrows showing the directional relationships between these constructs. The themes and supporting evidence are described in detail below.

Context

There were three contextual conditions which actors identified as important for the success of PHM: the importance ascribed to the health inequality in question, the importance of investment and commitment to data sharing and the role of underlying personal and organisational relationships.

Mechanisms by which PHM may be implemented

Programme delivery

Participants described multiple strategies to increase awareness or readiness to accept existing vaccination opportunities, including communication strategies, for example investing in translating leaflets and videos into languages spoken by populations with lower uptake; targeting communication campaigns via WhatsApp and social media; ensuring that campaigns used images reflecting the ethnic groups they were aiming to reach; workforce strategies, using staff reflective of communities and utilising trusted community voices to deliver vaccination information and reaching out to communities via door-knocking; working closely with faith or community leaders to share accurate information including about uptake.

“We’ve got that information and we’ve got it at a fairly granular level to allow us to then effectively target those communities or engagement events. So we’ve done a lot of engagement events on social media and stuff like that – again, those have been targeted at audiences that we know are hesitant using languages that we know they can speak.”

Participant 36, Senior Public Health Analyst, Deployer

Participants also described physical changes made to the location of vaccine delivery units and reported that these changes may not have been made without the evidence provided by the Dashboard, as the more mobile and flexible delivery model was more resource-intensive than centralised vaccination centres. For example, the Dashboard's capacity to display maps showing areas with very low uptake was used in the business case for, and design of route of, vaccine buses which were deployed in areas of low uptake. Additionally, NCL commissioned “pop-up’ vaccination events, for example arranging vaccination drives at sports stadiums aimed at young people, at universities aimed at students or employing vaccine link workers.

“[We] use it to set the groups who need to have vaccine link workers for so for example, identify the need to have a Polish speaking vaccine community worker.”

Participant 10, Consultant in Public Health, Deployer

Participants ascribed value to the Dashboard providing “targetable” information, namely that which enables the user to reach a specific population group in a tangible way for example via an identified location. Participants highlighted that targeting, in turn, was enabled by the dashboard's ability to layer and combine characteristics, such as ethnicity and language spoken, enabling an intersectional and granular approach. Finally, the short duration between initiating an action and seeking a change in the data gave them confidence in both the data and their approaches.

“We saw low uptake in White Other populations, and our first vaccine bus after doing specific door knocking in those communities, we had our highest number of vaccines administered in the White Other populations that are Romanian, Bulgarian, and Turkish. And, again, the same in our Caribbean population, again low uptake targeted with door knocking, targeting community leaders and that was our second highest uptake from our vaccine buses.”

Participant 1, Public Health Analyst, Deployer

Many participants described that the Dashboard provided strategic intelligence which helped them formulate the question of how to reach groups with low uptake. In NCL, users reported ad hoc engagement with local community groups and leaders on a case-by-case basis to determine the best mechanisms to improve uptake. Stakeholders indicated that some PHM localities are prospectively building community engagement into their programmes in a more structured way.

“… gives you the data to then be able to have proper discussions with people who do know those populations.”

Participant 27, Lead in Vaccine Programme, Deployer

In contrast, there were parts of the ICS where participants did not feel they had the support to understand the reasons behind low uptake or the capacity to respond to it. This acted as a disincentive to looking at the data in PHM tools:

“I think unpicking the reasons why people are not coming is also difficult to do and time-consuming … at some points we just didn't have capacity to set up a popup or run out or going to speak at local community centres or religious venues. … So there's almost no point in delving into the data.”

Participant 6, GP, Questioner

Intersectoral programmes and direct care

From our local evaluation we learnt little about how PHM could be used for direct care. Some of the evidence suggested the COVID-19 Vaccination Dashboard had minimal impact on direct clinical care, in part because general practice staff that were not providers of vaccinations and were therefore less able to respond to indicators of low uptake, but also because PHM may not have reached its potential in primary care. In the workshop, however, several stakeholders discussed the potential for PHM to influence direct clinical care, and particularly how data could be used to support shared decision making and the potential to differentially benefit disadvantaged patients.

With respect to intersectoral care, workshop participants also gave examples of how data from PHM has been used to decide on the commissioning of new programmes or influence referral pathways to address health inequalities. For example, they described a programme whereby patients with conditions exacerbated by cold weather that had high emergency department attendances were referred to receive energy efficient home improvements (Anon, 2016).

Strengths and limitations

This study's main strength lay in its blended team of embedded and external researchers. The internal position of the embedded researchers, who had substantive roles in the health and care system, made the study possible. They were able to obtain interviews with participants through their networks and were permitted to access data on PHM staff usage. Their contextual knowledge of participants, familiarity with the local and regional policy context and with PHM also strengthened the interpretation of data. Experienced external researchers were able to navigate research ethics and governance procedures and enhanced the rigour of the data collection and analysis and transferability of the findings.

The case study analysis was based on Normalisation Process Theory, which focuses on the work actors do together to make sense of and embed innovations (May and Finch, 2009). In developing the programme theory we have gone beyond ‘the work’ involved in adopting and embedding an innovation and therefore diverged from NPT in our structure of reporting our findings. However, constructs of NPT emerged through analysis and in elements of the programme theory. For example, the construct of coherence was central to NCL staff's receptivity to PHM in the particular context of COVID-19 vaccinations, and prompted discussion with stakeholders on where coherence might be lacking in other contexts and its implications for PHM's adoption. In addition, coding to the construct of collective action helped to surface some unexpected ways in which actors used the dashboards and collaborated based on the findings, for example direct users summarising findings for indirect users.

There are three major limitations of this study. First, the evaluation of PHM centred on one region. The national stakeholder workshop, however, helped to understand what themes might be transferable to other geographical areas. Second, we were limited in the quality of the quantitative data we were able to obtain on staff system usage in the mixed-methods case study, which significantly constrained the analysis we were able to conduct. However, the qualitative findings also indicated that quantitative measures of direct usage would not have captured the powerful indirect ways in which the PHM could influence decisions. Third, many staff contacted did not respond to invitations to participate, and several of those who gave a reason for declining to participate cited insufficient experience of the Dashboard. There is therefore a risk that our findings do not fully capture where and why PHM may not be used. Nevertheless, some of those who were interviewed were forthcoming about what deterred them from using it.

Comparisons with other literature

As Stein et al. identified, the COVID-19 pandemic highlighted the fragmentation in health and care systems and the impact of this on exacerbating inequalities (Stein et al., 2020). They noted barriers in accessing system-wide data as a challenge to integrated responses to health and care challenges post-COVID-19. Our study suggests that where system-wide datasets are present and there is readiness to implement change, PHM can help integrated systems in addressing health inequalities.

There have been comparable approaches used to those described in this study to increase vaccine uptake overall and to address inequalities in uptake, such as tailoring communication messages (Halvorsrud et al., 2022). In common with our study, Halvorsrud et al. (2022) also found other London areas were open to recognising inequities arising from service provision and to support more costly forms of programme delivery. While many areas across London commonly targeted population groups by ethnicity, it was rarer for local areas to target inequalities by language spoken, and there were no examples of targeting geographical areas. Our data suggested it was the distinct integrated data provided through PHM that enabled North Central London to target these population groups.

Applications in real-world settings

We have drawn from the specific features of the COVID-19 vaccination programme to speculate on the conditions necessary for PHM to help ICS to reduce health inequalities. While technical conditions, that is, generating accessible shared data are necessary, they alone are not sufficient. National and local prioritisation of the inequality are also needed. In the case of the COVID-19 Vaccination Dashboard, the priority of the programme and national policy on eligible cohorts facilitated its adoption and NCL's focus on horizontal equity. Furthermore, the vaccination programme had a strong evidence base and yielded swift, measurable change, which may have made it a particularly good candidate for PHM.

This study highlights some tensions which could hamper ICS efforts to exploit the potential of PHM to help them reduce health inequalities in other circumstances. One major tension relates to how PHM is conceived, and the resulting permissions attached to its use. PHM in other contexts, mainly the US, has focused on its use for direct care of individual patients (Steenkamer et al., 2017). Indeed, the one PHM model we identified in the literature has drawn on individual behaviour change theories rather than theories of structural or social determinants of health (Scheck McAlearney, 2002). The information governance surrounding data linkage in the UK is interpreted differently across different areas, but in many cases is interpreted to cover data linkage for the purposes of informing direct health care (North Central London Integrated Care System, 2022; NHS England Transformation Directorate, 2021).

Another major tension relates to the ability of ICSs to reallocate budgets in response to information on inequalities from PHM systems (Goddard, 2023). National policy in England is now focused on health care inequalities (NHS England, 2022). This may narrow the focus of ICSs to seeking to address inequalities in access to health care, and thus limiting intersectoral contributions to reducing health inequalities through addressing wider determinants of health. The Hewitt review describes the narrow scope of health inequalities policy, and advocates for the role of ICSs as agents of improvement for wider determinants of health. In her independent review of ICSs, Hewitt also notably recommends a shift in funding towards upstream preventative services and a greater facilitation of data sharing, including ICS performance data (Hewitt, 2023).

Implications for research and practice

We conclude from the evidence obtained in this study that PHM can aid integrated health and care systems in reducing health inequalities by providing trusted data on populations that could benefit from intervention. PHM can be effective, however, only where there is organisational support for reducing inequalities The programme theory developed from this evidence suggests that the focus of PHM should go beyond the data infrastructure to facilitate actors' use of PHM tools in practice. In addition, by providing clear evidence of inequalities, PHM may be a useful tool for integrated care systems to reconcile efficiency priorities with goals for addressing inequality within local areas at intersectoral, population and individual levels.