FormalPara Key Points for Decision Makers

Public involvement (PI) in modelling is feasible and desirable.

The Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation (MEMVIE) Public Involvement Framework can enhance the quality, validity and relevance of models.

PI in modelling can contribute to the decisions policy makers have to make.

1 Introduction

Public involvement (PI) has become increasingly embedded within health research, both nationally and internationally. In the UK, PI is defined by INVOLVE as research that is undertaken ‘with’ or ‘by’ patients or members of the public, rather than ‘to’, ‘about’ or ‘for’ them [1]. This active involvement is different from patients or the public participating as passive research subjects in clinical trials or research studies with little contribution to identifying need, design, conduct or interpretation [2]. In this study, we use PI to refer to inputs from the public rather than patients, acknowledging these can be different [3]. Recently, the focus has been on the concept of ‘co-production’ of research between researchers and the public [4]. We drew on the concept of co-production, which we acknowledge is a rare approach in modelling. The notion of co-production is founded on a number of elements or principles that are relevant in research [5]. In identifying these, Heaton et al. [5] use the term ‘users’ to refer to the service context. For consistency, we use public contributors. First, in the process of co-production, public contributors are regarded as active agents and not merely passive subjects or recipients of services (or research). Second, there is greater than usual equality in the relations between public contributors and professionals, with services becoming more people driven, with their knowledge and experience being valued on a par with that of professionals. Third, public contributors and professionals recognise that they can achieve more by working together than they can apart; both also find their relationship to be reciprocal and mutually beneficial. Fourth, public contributors’ increased participation transforms the ways in which public services (or research) are designed and delivered, developing capacity for public contributors’ present and emerging needs to be met. Fifth, the participation of public contributors in the co-production of services (or research) is encouraged and facilitated by networks and organisations that support their involvement.

Ideally, the practice of PI is based on an understanding of the values that underpin it as they often reveal key motivations. These values may vary among individuals within a team, making it important to explore them before and during a study. Gradinger et al. [6] identified a range of value systems and value clusters underpinning involvement that includes normative values focused on moral, ethical and or political concerns associated with PI in research (empowerment, rights, ethics); substantive values that focus on concerns about the consequences of PI in research (quality/relevance, validity, reliability) and process values that focus on concerns about the conduct of PI in research (respect/trust, openness and honesty).

PI has become embedded in many areas of health research, where values and different approaches have been explored. However, there are areas where PI is much less common, including complex areas such as mathematical and economic modelling. A definition of a mathematical model is a “mathematical framework representing variables and their interrelationships to describe observed phenomena or predict future events”. The epidemiological component captures the health-related outcomes [7]. The health economic components of the model capture the costs and, ultimately, the cost effectiveness [8]. Modelling provides decision makers with the best available evidence to reach a decision [9]. In 2013, after analysis of the available cost-effectiveness models, the Joint Committee for Vaccination and Immunisation (JCVI), an independent expert advisory body for vaccinations in the UK, advised that a novel vaccine to prevent meningococcal group B disease would not be cost effective at any price [10]. Stakeholders, including patient groups, submitted evidence and the models were rerun to include additional disease-specific costs and health benefits [11]. This led to the introduction of the vaccine for UK babies in 2015, although a health petition called for access for older children [12]. Scrutiny of the cost-effectiveness analysis of the MenB vaccine questioned whether cost-effectiveness methods reflected the health characteristics that the public value when prioritising health spend. The adequacy of methodology is still debated and the Government has not formally adopted the recommendations from a working group that addressed this question [13]. However, involving patients and the public in all stages of health economic analysis could enhance models, improve confidence in and ultimately accelerate decision making. PI in modelling could also play a pivotal role in highlighting key areas for further research in modelling methods.

The UK spends in excess of £200 million per year on vaccines and vaccine delivery [14]. Epidemiological and economic models are important in forming judgements about the introduction of, or a change to, immunisation programmes. The MEMVIE study, commissioned by the UK Department of Health in 2015, provides second-opinion modelling. Model outputs are subsequently presented to the JCVI, the key programme decision maker. While modelling can be a very effective tool, views differ about the extent to which models truly capture the entirety of a concept or a context, as they can be limited by the availability and quality of data, and assumptions made. As George Box said, “essentially, all models are wrong, but some are useful” [15], reflecting a realistic view of the limits of modelling. Although Box did not mention PI, our extrapolation of Box’s view is that models can benefit from additional sources of input, such as PI, to help strengthen them. Different forms of experiential and community knowledge can complete the ‘world view’ that may be required to create a ‘less wrong model’. In commissioning MEMVIE, the Department of Health in the UK recognised the role the public could have in creating models that are more relevant, appropriate and acceptable from a public perspective. The investigation of the role of PI to inform such modelling was novel at the time of the commissioning of the contract by the Department of Health in 2015.

In scoping the literature, it was clear that mathematical and economic modelling has not embraced the potential of active PI, either philosophically, in terms of perceiving any value, or methodologically in practice. It became apparent that in order to embed PI within the MEMVIE study, we needed to explore where, when and how the public could contribute to modelling, understand the values that drive this involvement and develop a framework that could support MEMVIE and others in their implementation of PI in modelling.

2 Aims

  1. 1.

    To explore and capture the potential areas of contribution of the public to mathematical and economic modelling.

  2. 2.

    To identify the values that underpin PI in modelling.

  3. 3.

    To co-produce a PI framework that identifies the nature and type of PI in modelling and supports its implementation.

3 Methods

PI can be described as a form of social practice and can be informed by evidence that guides best practice. We drew on the learning from the RAPPORT study that emphasised the importance of high-quality relationships, enabled a sense of reciprocity and provided enough time for exploration and discussion [16]. As this study focused on working with public contributors as partners in research, rather than subjects of research, ethical approval was not required [17]. However, we drew on principles of good ethical practice in PI [18] and GRIPP2 to report the PI in MEMVIE [see electronic supplementary material (ESM 1)] [19]. We are one research team, each contributor with different skills and experiences. There are two groups; in one group, the skills are centred on academic profile, and, in the other group, their skills arise from practical living experiences and a wider societal view. The two groups together focussed their attention on the research question. We have called these groups public and academic contributors.

4 Key Stages of Framework Development

We identified the following key stages needed to develop the MEMVIE framework. These were iterative rather than linear, depending on what emerged from discussions.

Recruitment of public contributors We used an informal interview process, intended to identify individuals who had relevant health knowledge, experience and learning, potentially gained through different channels (including previous PI projects and their community roles). We drew on the Warwick Medical School Public Involvement UNTRAP (Universities/User Teaching and Research Action Partnership) Network, which includes patient contributors from diverse backgrounds who are not academics. The informal interview explored how potential public contributors felt about working in a complex area such as modelling. Public contributors needed to be comfortable and confident when asking questions about methods. There would be many sessions focused on learning about the methods and potentially several moments of uncertainty when people realised it may take a while to understand a method. They also needed to be comfortable with having no prescribed route map as we embarked on our exploratory journey together.

Establishing a deliberative knowledge space The development of the Public Reference Group meetings (which included both the public and academic contributors) reflected the concept of a deliberative knowledge space [20, 21]. Creating a deliberative knowledge space allowed participants to consider and discuss relevant information from multiple points of view, drawing on their own different expertise, knowledge and values as the lens through which to view a disease or a method [21]. The first few meetings were exploratory as we discussed how we would examine PI in modelling. As we progressed, the meetings started to follow a more logical path through learning (about the diseases and modelling methods) and discussion. As public contributors built up their knowledge, they gradually developed their capacity to identify potential PI contributions [20]. Discussions were facilitated (by the PI lead, SS) to ensure that a diversity of views from people with different perspectives were included and that everyone’s contribution was valued and listened to. We ensured meeting places were relaxing and conducive to discussion, moving venue during the project to ensure a higher-quality environment.

Understanding modelling methods An essential activity, which happened in almost every PI Reference Group meeting, was the explanation given by researchers about the methods. Presentation of methods ensured that public contributors felt comfortable asking questions and discussing modelling aspects, enabling them to reflect upon and explore the methods presented. Specific sessions on particular topics were often repeated as learning could be lost in between meetings, as meetings took place between four and five times per year over the 5-year period.

4.1 Emergence of the Framework

Creating a deliberative knowledge space enabled possible areas of public contribution to modelling to emerge, as public contributors were seen as active agents, with their knowledge equally valued and with power shared in how decisions were made in what to include in the framework. Both sides found their relationship to be reciprocal and mutually beneficial. We recognised that the early discussions of PI in mathematical and economic modelling represented a ‘liminal’ space, an in-between place or a place of transition, where there were no conventions of what PI in modelling might look like but one where we valued the potentially transformative contribution a co-production approach could bring [21]. This liminality initially felt uncertain and very uncomfortable for both public members and the research team, and facilitation by the PI lead (SS) was important in reassuring everyone that this feeling was normal. As meetings progressed, a dialogue emerged, within which it was possible to identify potential areas of public contribution to modelling.

We captured ideas by recording meetings rather than taking detailed notes. This provided the group with a more detailed record of their discussion if we wished to use these. However, these recordings did not provide research data in the conventional sense from which thematic analysis could be derived. Rather, from these discussions and reflections, we started to build up a list of areas of possible contribution, which gradually built up into categories that eventually formed the framework. The PI lead (SS) captured the categories identified in the discussion, which we reviewed at each meeting, with the public contributors refining and discussing new categories with the academic contributors each time. This process took 5 years, partly because of the co-production element, but was necessary to create the appropriate knowledge space for exploration in such a complex liminal space [20, 21]. As the categories in the framework became clearer, the public contributors suggested we group the categories, mapped to the key steps of modelling, to facilitate the use of the framework by the modelling community (Fig. 1).

Fig. 1
figure 1

The left-hand side of the framework represents the epidemiological components of modelling, while the middle section represents the health economic components of the modelling. Each phase is labelled Ep (epidemiological) or Ec (economic), corresponding to the associated section of the long-form framework (Appendix 2). The framework should be read across horizontally, i.e. Ep1, then Ec1, Ep2, Ec2, and so on. The legend on the right-hand side denotes the five types of PI contribution identified by the reference group. Patterned backgrounds illustrate the regions of the model pathway where the associated PI contribution type predominately features. PI public involvement, QALYs quality-adjusted life-years, GP General Practitioner

Full size image

4.2 Identification of Values

Values are important in PI as they can motivate individuals in different ways [6]. As we conceptualised PI in modelling as taking place within a liminal space, one that had not yet been explored and where things were unclear [21], we enabled the values to emerge during the discussions to ensure they were the result of a co-production process. Once we had identified the values, we then compared them with those identified by Gradinger and colleagues as a form of reflection rather than validation [6].

5 Results

We structure our study outcomes as follows. First, we present the characteristics of the Public Reference Group contributors and the academic contributors, with some quotes about process. Second, we report the values that our meetings identified as underpinning PI in modelling. Third, we consider key aspects of practical framework implementation. Fourth, we present the short form of the framework, which summarises the main components to enable an overview (Fig. 1). The final section presents the long-form framework, a detailed description of potential PI contribution (reported in full in ESM 2). It is important to emphasise that, at this stage, the MEMVIE framework has not undergone a full evaluation, although one is being conducted within our follow-up study, namely MEMVIEER (Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation and Emergency Response). Therefore, at present it is unclear which framework elements are vital to PI in modelling and which are optional. Prior to full evaluation, we would encourage teams to select and evaluate the aspects they feel are of most relevance to their specific project.

5.1 Characteristics of the Public Involvement Reference Group Contributors and the Academic Contributors

The public contributors included eight individuals at the start of the project, with a further public contributor joining in the third year. They were recruited through the Warwick University UNTRAP Network and all contributors were members of that network. They represented a range of ages, ethnicities and cultural backgrounds, different careers/professions/employment, a range of ‘family’ experiences, different lifestyles, different health issues within family/friendships, and of different ages and from different generations. Five public contributors remained at the end of the study. These individuals had a range of expertise that included a biology background, health visiting, nursing, community development, and experience of patient organisation input into the cost-effectiveness modelling of meningitis vaccines. There was also a diversity in ages and caring responsibilities. The academic contributors included two mathematical and economic modellers at any one time, an economist and a social scientist. All contributors were comfortable with complex topics, a deliberative approach and being within an often uncertain, liminal space [21]. The Principal Investigator recruited, supported, facilitated and organised PI involvement, capturing outputs and reporting impact.

5.2 Values

PI Reference Group discussions identified a range of values associated with the process of modelling and the use of model outputs (Table 1). There was recognition that the public bring forms of knowledge (such as experiences in the wider context of life) that supplement evidence from peer-reviewed papers. The expertise and broader societal perspectives that the public bring can add to the validity of a model, potentially enhancing its relevance, utility and transparency through diverse inputs. The concept of co-production to guide PI in modelling was recognised as relevant and important. The Reference Group identified the potential for PI to contribute to the credibility and consistency of the model through this scrutiny and continuous improvement through development. Additionally, the group attributed importance to contextualising the model within a wider societal view and understanding the connections to other services and contexts. The Reference Group identified tacit values that may underpin PI in modelling and are important to acknowledge, such as the belief that vaccination is a public good.

Table 1 A values framework for PI in modelling
Full size table

We compared these values with those identified by Gradinger and colleagues [6]. We identified significant alignment, particularly with substantive values focusing on the consequences of PI in research, such as effectiveness and research quality, and on process system values, those concerned with conduct of PI in research. Regarding normative values, it was recognised that the output of cost-effectiveness modelling ultimately decided whether the public would have access to sometimes lifesaving interventions and that certain methods used could create ethical dilemmas. For example, placing greater emphasis on long-term health effects by using a lower discount rate can make or break whether a vaccine is deemed a cost-effective spend of public funds. A recent example of this is extending the availability of the HPV vaccine to boys in the UK, which was only deemed cost effective when longer-term health benefits were considered. While the group are not decision makers, they wanted to ensure that a full picture of the costs and benefits of interventions are considered. These may not always be well represented under the current cost-effectiveness rules, particularly for marginalised groups.

5.3 Evaluation of Process

The use of the concept of a deliberative knowledge space and think-aloud techniques provided an open, non-prescriptive dialogue that enabled ideas and thoughts to emerge. The public contributors were able to challenge the data, the basis for the collection of data and the interpretation of that data, thinking outside of the box in a safe space where modellers could rework their thinking.

One of our public contributors stated that the process “was more akin to embarking on a long journey on a complex ship, with no captain, with an unknown destination with no operating manual and a very mixed and unfamiliar crew who were trying to figure out how this ship works, how each crew member fits in whilst trying to write the outline of an operating manual for future passengers on a similar journey”.

The meetings enabled thematic development over time as the Reference Group contributors worked with the academic contributors on continuous iterations of the emerging framework. We held 21 MEMVIE meetings over 5 years, each lasting between 2–3 h, with email contact in-between with the group commenting on documents.

Two quotes illustrating the public contributors’ journey and a third view from a researcher are presented in Table 2.

Table 2 Public contributor and academic contributor views
Full size table

5.4 Implementing the Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation (MEMVIE) Framework

As PI in modelling for developing policy vaccination recommendations is in its early stages, the modelling community may need support to implement the MEMVIE Framework. The Public Reference Group suggested adopting a ‘What, How, When, Where, with Whom’ approach that a modelling team could use to plan for PI in their study (see ESM 3). A brief summary is provided below.

We encourage teams to define PI before they consider how to undertake it. A suggestion of co-production as an approach to PI is made [4, 5]. However, as co-production is difficult with some aspects of modelling, it may be that teams need to recognise the current limits to co-production and perhaps draw on the ACTIVE Framework for patient and public involvement (PPI) in systematic reviews [22], which recognises a range of levels of public input in different scenarios.

We encourage teams to use the concept of a deliberative knowledge space in which to hold discussion, to ‘think aloud’ and to draw on standards [20, 21] (such as UK INVOLVE standards) [23] and the developing PI evidence base to guide practice. The need for teams (particularly researchers) to undergo training in PI was seen as vital to developing appropriate skills that support the process. Facilitation of PI is key and a PI lead role is considered essential. Similarly, the need for the public contributors to undergo training in the basic methods of health economic evaluation was also seen as crucial. Furthermore, the values that are relevant to a particular project need to be identified (Table 1) as they will be embedded through discussion and can help understand differences in view and perspective.

We encourage teams to be flexible according to the needs of the project, but recognise that by definition the ‘complete modelling team’ should include both researchers and public contributors. Evaluating process and outcome is crucial. One method proposed by our Reference Group was an impact log to capture key contributions after each meeting or key interaction. The log then forms the basis of a narrative model, which can provide a qualitative ‘story of model development’ in a way that replicates the quantitative elements with narrative, capturing key decisions, key assumptions, values and other aspects of discussion that public contributors feel are important. This qualitative, narrative model could be reported alongside the quantitative model.

5.5 The Short-Form MEMVIE Framework

Due to the complexity and length of the MEMVIE framework, the Reference Group identified the need for a short form, or summary, of the framework (see Fig. 1).

The framework breaks down the epidemiological and economic components of the modelling. Each phase is labelled Ep (epidemiological) or Ec (economic), corresponding to the associated section of the long-form framework shown in ESM 2, and should be read across horizontally. It is important to emphasise that this figure reflects one representation of modelling in order to demonstrate the potential for PI. Modelling can be flexible and iterative as a method. Through Reference Group discussion, we identified a range of potential macro areas of contribution for PI in modelling. The five overarching types of PI contribution (right-hand column, Fig. 1) are Reviewing Context and Relevance, Assessing Data and Justifying Model Choice, Troubleshooting, Interpreting and Reviewing Outcomes and Decision Making. Each of the components of these overarching areas is described in detail in the long-form framework.

5.6 The Long-Form MEMVIE Framework

Each of these model steps described in the short-form framework have a more detailed set of potential PI contributions presented in the long form, explained in ESM 2. The public Reference Group felt it was important to differentiate between the academic modelling team activity, the nature of the PI contribution and the components that support PI input. As with the short form, the framework should be read horizontally across, i.e. Ep1, then Ec1, Ep2, Ec2, and so on. We provide here an overview of the five overarching types of PI contribution.

The initial stage is concerned with reviewing the context and relevance of the model and determining the state structure; that is, determining what should go into the model, with inputs from disease-specific experts and through reference to relevant literature. A contribution that the MEMVIE Public Reference Group made at this stage, when considering the transmission of pneumococcal bacteria among the population, was identifying the potential for uncertainty because the study used to identify mixing patterns in the population was conducted 10 years previously and mixing patterns may have changed due to broader societal change [24]. The modelling team had not identified this aspect at that point.

Within assessing data and justifying model choice activity (PI contribution), a farther-reaching benefit of PI is the Reference Group querying the robustness of a tool or questionnaires that have been used to collect data. For example, our Reference Group requested to see the NATSAL survey questionnaire [25] to judge the quality of data that MEMVIE was drawing on. The troubleshooting activity provided an opportunity for the modellers to host a series of thinking-aloud sessions while the models were under construction. Any preliminary findings from the models were presented to the group and the influence of some of the factors thought to be contributing to uncertainty were discussed in more detail.

The interpreting outcomes stage allowed the modellers to present their models and associated outcomes in their entirety. Both the economic and epidemiological models were merged together and it was an opportunity for the group to review the models and consider the sensitivity of the results in relation to the parameters used, and to critique the model assumptions. Another step for the PI group to consider was the impact of ‘rules’, such as the time horizon considered for the evaluation and the discount rate used, and how these rules affect the cost effectiveness of the intervention under consideration.

The final stage was decision making. While this phase would most likely be enacted by an independent committee of experts, PI contribution at this stage could be in the form of a report detailing the findings of the group, which could be considered alongside the model or presented by public representation on vaccination committees themselves.

6 Discussion and Conclusion

The MEMVIE study has been a key source of policy advice to the Department of Health in England, creating mathematical and economic models in a number of clinical areas, including human papillomavirus (HPV), pneumococcal and influenza. We used each of these projects as opportunities to build PI into modelling, a highly technical process often hidden from public or patient scrutiny, and yet with the potential to impact their health provision and health status in significant ways. We were not able to identify an existing framework for collaborative PI in modelling, and therefore we believe MEMVIE is the first international framework for PI in vaccination modelling with the intention of informing policy. The development of the MEMVIE framework demonstrates the feasibility of involving the public in the deliberation and decisions that inform models, providing that the context for implementation is appropriate.

Through the MEMVIE study, we were also able to identify the enablers that can support PI in modelling; for example, the creation of a deliberative knowledge space. Understanding each other’s values was vital to recognising the different perspectives public contributors can bring, which need to be explored at the start of and during the process [6]. Planning for the implementation of PI in a study was seen as important and emerged from recognising that PI in modelling is novel and requires careful facilitation from a PI lead to support the entire team of academic and public contributors working together.

The idea of a deliberative knowledge space enabled the technical and practical–experiential forms of knowledge to come together in a context where effective contributions could be made, informed by many deliberative sessions learning about methods. Through the deliberations, we identified the need to develop a narrative model that could complement the quantitative model. This provided the ‘story’ of model development, making accessible the often inaccessible equations and graphs. We will refine the idea of a narrative model in our next study, MEMVIEER, using a reflection and impact log to build up our understanding of contribution, guided by the MEMVIE framework.

As the primary beneficiaries of healthcare modelling, the public have a moral right to be involved as key valued and impacted stakeholders. Public contribution has the potential to ‘ground’ models in the reality of lived experience and behaviour and add to its interpretation and understanding. As Williams and Popay [26] say of the public “they are themselves bearers of considerable lay knowledge of their own life worlds and through their own ‘stocks of knowledge at hand’, were skilled interpreters and translators of the ‘external’ evidence provided by professional experts in dialogue with whom they could share in the decision making about what was to be done”.

Through deliberative discussion, challenging assumptions and enhancing the richness of the dialogue, model validity is enhanced, with models created that sensibly account for the complexity of a disease and of societal response. Extending the approach by Box and Draper [15], in the context of PI in mathematical and economic modelling, PI could ‘help models be less wrong and more useful’.

While the MEMVIE modelling study provides useful policy advice, the MEMVIE PPI framework has yet to be evaluated or used in practice. This is the intention in our next study, MEMVIEER, but other modelling teams may wish to use all or part of the framework to guide their work. We recognise that PI in modelling is not a linear process, with some phases undertaken in a different order to that illustrated by Fig. 1. We fully expect the framework to evolve over time as teams implement it and report their framework refinements. We would encourage them to publish their outcomes to strengthen the PI evidence base in modelling, building high-quality practice that enhances the validity and utility of models, making models more accessible and accountable to the public, and creating significant patient and public health benefits in addition to furthering public trust in recommendations.