practice Evidence to support delivery of effective health services: a responsive programme of rapid evidence synthesis

Background: Two UK academic centres were commissioned to provide a responsive rapid evidence synthesis service. The service covered topics identified by the National Institute for Health Research Health Services & Delivery Research (NIHR HSDR) programme as priorities for the National Health Service or to inform research commissioning. Aims and objectives: To describe and evaluate the review teams’ interactions with the y Methods: Narrative review of stakeholder interactions at different stages of the review process: prioritisation and defining scope; dealing with unexpected results; dissemination of findings; and measuring impact, illustrated by examples from the first three years of the service (2014–17). Conclusions: Timely production of high-quality outputs was facilitated by: initial mapping and scoping of the available published evidence; early engagement with stakeholders to optimise their involvement within limited time and resources; and willingness to consider creative solutions and different ways of working to overcome problems encountered in specific projects.


Background
The National Institute for Health Research (NIHR) Health Services and Delivery Research (HSDR) programme aims to commission research that will result in 'rigorous and relevant evidence to improve the quality, accessibility and organisation of health services'. The programme covers both primary research and evidence synthesis, and topics originate from the programme itself and from researcher-led proposals.
In early 2013, the NIHR HSDR programme invited expressions of interest for production of rapid evidence syntheses for the programme. The call aimed to identify suitable teams or review units to undertake up to five projects per year. Outputs were expected to be of immediate value to the National Health Service (NHS). Evidence synthesis was defined as 'a comprehensive review of published literature with an explicit search strategy, using an appropriate range of sources and including critical assessment of quality of evidence and strength of findings'. The scope of the programme was to extend beyond systematic reviews of clinical trials, and the funder did not specify the methodology to be used. A key rationale for commissioning the evidence synthesis teams was to offer an alternative more rapid method of evidence synthesis to the standard research commissioning cycle.
Teams from the Universities of York and Sheffield submitted successful bids in response to the call, and three-year contracts ran from early 2014 to 2017. Table 1 details the range of topics covered. With the exception of one rapid scoping exercise, reports followed the usual HSDR programme format and were published online after peer review. Most were subsequently published as issues of the Health Services and Delivery Research journal. The two teams also collaborated to produce a report outlining the lessons learnt from the programme, covering methods, stakeholder involvement and dissemination of findings .
As noted above, review teams were expected to undertake up to five projects per year. The funding provided by the HSDR programme reflected the expected workload, but the exact nature of the outputs was not specified in advance. Methods to be used were decided by the review team for each project in consultation with the HSDR programme team and other relevant stakeholders, including topic experts. The range of stakeholders involved in the review projects varied according to the source of the request, but frequently included representatives of NHS England or Public Health England as well as the HSDR programme itself. Project timetables were agreed as part of the process of protocol development, with a degree of flexibility to allow for unforeseen developments such as urgent requests for a new piece of work to be undertaken.
Both core review teams comprised experienced systematic reviewers with an interest in topics related to organisation and delivery of health services. We augmented our in-house expertise by drawing on other sources of clinical and policy expertise where necessary.
The literature on dissemination of research distinguishes between push (efforts by researchers to disseminate their findings), pull (the needs and requests of evidence users) and linkage and exchange (evidence users and researchers forming links and working together to address users' needs) (Caplan, 1979;Gagnon, 2011;Landry et al, 2001;Lavis et al, 2003;Stone, 2002). The evidence syntheses undertaken by the two teams typified the first two approaches, with topics originating from NHS sources or the programme itself, and with review teams taking responsibility for dissemination of the findings. An element of linkage and exchange was seen in some projects, but the ability of stakeholders to devote time to close involvement was a limiting factor.
Following a review of the first three-year programme, and after open competition, both evidence synthesis centres were re-commissioned to provide a similar responsive review facility for a further three years. A third centre was added.
The objective of this paper is to describe and assess the review teams' interactions with the evidence users targeted by the programme, primarily NHS clinicians, commissioners and managers. We particularly aim to highlight transferable barriers and facilitators that impact on the uptake and use of research evidence by decision makers. For a discussion of interactions with the funding programme and with patients and the public, see the associated report . Ethical review was not required for the work reported in this paper. Models of urgent care Rapid review(s) 8  Community diagnostic services Evidence mapping exercise and focused rapid reviews 9 (+2)  TB contact tracing Mapping review and focused rapid review 7 (+2)  Frail elderly in the ED Mapping review 7 (Preston et al, 2018) SMI, serious mental illness; PTSD, post-traumatic stress disorder; ED, emergency department Rapid review: Review using systematic methods but with some elements of abbreviation and/or acceleration Scoping review: Initial exploration of evidence in a topic area primarily to identify gaps/scope for future research Mapping review: Exploration of evidence in a topic area as a project in its own right without detailed synthesis or quality assessment  Table 2 summarises key barriers and facilitators to working with evidence users at different stages of a review project. Exemplar projects undertaken by both teams are also shown.

Prioritising topics
Topics to be addressed by the evidence synthesis centre teams were prioritised by the HSDR programme team, who supplied a topic specification and potential contacts. These contacts were generally topic experts in the NHS or academia who sometimes also had roles with NHS England or other evidence users. The originating organisation or individual first suggesting a topic was not always provided to the HSDR programme team. When the provenance of a topic was clear, the review teams were better able to locate the proposed work within the wider context of NHS policies and priorities. Examples of this were the projects on congenital heart disease services (Turner et al, 2014) and diagnostic testing services in community settings . Working with the originator of a topic request was helpful to the review teams in defining an appropriate scope for the project and developing the review protocol, although the teams had to bear in mind the possibility of being unduly influenced by the perspective of a single influential stakeholder.

Setting the scope
Defining the scope, including the question(s) to be addressed and the proposed methods, is a key part of any systematic evidence synthesis project . The scope should also take into account the staff resources and the time available to complete the work. As a responsive service the evidence synthesis centre teams were often asked to address topics for which we had little or no prior knowledge of the nature or volume of available evidence. This meant that scoping work at an early stage (before finalising the project protocol) was critical in forming an idea of the quantity and quality of the available research evidence. This process also reassured both the review commissioners and the review team that there was no existing or ongoing evidence synthesis covering the topic. Stakeholder involvement at this stage was particularly important when the exploratory work suggested that modification of the original scope was required (Example 1) or when the original scope was prohibitively broad (Example 2). Project protocols were approved by the HSDR programme team and published on the review teams' and HSDR programme's websites. When appropriate, protocols were registered with PROSPERO.

Example 1: refining scope in light of initial exploration of the topic
A review on physical healthcare for people with serious mental illness (SMI) provides an example of modifying the scope of a review in the light of initial scoping work (Rodgers et al, 2016a) The original research questions were: • What models exist for the provision of integrated care for people with mental health problems? • What evidence exists for the effectiveness of these models?
• Are there evidence gaps that require either further primary research or a full evidence synthesis?
The review team performed a rapid scoping of the existing literature and made contact with practitioners, researchers, local mental health service leaders and relevant organisations, including Collaborations for Leadership in Applied Health Research and Care (CLAHRC) and NHS England. A relevant development was that NHS England had recently set up a taskforce to develop a five-year strategy for mental health across England.
Initial literature searches identified a recent systematic review from the US that directly addressed the broad questions about existing integrated care models and their evaluation that had been suggested by the funding programme team. Around this time, NHS England also announced both demonstrator sites for integrated and personalised commissioning (IPC) for people with complex needs, and the first group of 'vanguard' sites to inform the development of new care models for the NHS. Given these developments and the limited resources of a rapid review, the team worked with expert advisors, including topic experts identified by the team and local service users, to refine the scope and produce a revised set of review questions: • What types of models exist for the provision of integrated care specifically to address the physical health needs of people with severe mental illness when accessing mental health care services? • What are the perceived barriers and facilitators to implementation of these models? • How do models implemented in practice compare and contrast with those described in the literature? • What are the high priority areas for either further primary research or a full evidence synthesis?
The revised scope thus looked beyond purely evaluative evidence to consider implementation issues. In addition, while all evidence would be interpreted in the context of the NHS, included evidence was not limited by country of origin.

Example 2: modifying a broad review question to produce a manageable scope
The objective was to assess the evidence base for diagnostic testing services provided outside hospital settings . The question was broad, potentially covering all types of diagnostic test for any condition. Following preliminary discussion with the HSDR programme team, it was agreed to conduct the review in two stages. The team performed an initial evidence mapping exercise, and discussed the findings in a teleconference with NHS England's National Clinical Director for Diagnostics and the funding programme team. This identified topics for three focused reviews (diagnostic ultrasound, logistics of community diagnostic services and diagnostic pathways for people with breathlessness) which were conducted separately but brought together in the overall discussion in the final report. In contrast to the previous example, this topic involved working with a single expert stakeholder as well as the research funders. Overall, the combination of research and methodological expertise with the clinical and policy insight provided by the National Lead was effective in focusing resources where they could achieve the most useful impact.

Staying on track
The rapid review model employed by both review teams was predicated on them requiring minimal stakeholder input, once the scope was agreed, until preliminary findings were available. Interaction with busy clinicians was reserved for circumstances where their input was deemed critical to the conduct, analysis or interpretation of a specific review. For this reason, neither evidence synthesis centre maintained a standing advisory board. One of the teams specifically allocated resources to bring in additional topic expertise from its own university or elsewhere in the UK university sector or NHS where required. Example 3 illustrates the value of expert stakeholder consultation to resolve a specific question at a key decision-making point.

Example 3: choosing between different areas of focus
The focus of a review on tuberculosis (TB) contact tracing  was targeted on contact tracing within specific population groups ('hard to reach' populations). However, initial literature searches revealed that the available evidence related to these groups comprised a small number of generally low-quality studies. The team therefore consulted stakeholders selected by the research team in conjunction with the HSDR programme team before undertaking further work. Stakeholders included national and local policymakers, infectious disease and public health practitioners, and the review commissioners. Three options were presented for discussion: widen the review to TB contact tracing in any population; examine contact tracing in specific populations for other conditions; and focus on two specific interventions (social network approaches and community workers).
Consultation revealed a consensus that the first option was most promising. The review team therefore broadened the scope to include TB contact tracing in any population while retaining a focus on applicability of evidence to the specific population groups of interest.
Example 3 differs from Example 1 in several respects. The modification to the original scope was somewhat more radical, with stakeholders choosing between options with different implications in terms of the focus and outputs of the review. The resources required to carry out the work varied according to the option chosen. In practice, the reviews covering any population and specific population groups were conducted separately before being brought together in an overall narrative synthesis.

Getting the word out
The main audience for the evidence synthesis teams' outputs was envisaged as NHS decision makers (for example commissioners and managers) needing to use and make sense of research evidence to help them in their work. Findings could also be relevant to researchers and, in some cases, directly for patients and the public. Some projects were targeted at the needs of the funding programme itself by scoping areas of research to inform potential calls for new commissioned research. It was therefore important to ensure that the results were disseminated actively and appropriately, and to identify evidence of the impact of reports and other outputs. Impact is discussed in the following section. Dissemination was the responsibility of the evidence synthesis centre teams. The funders organised peer review of the main reports and required to be kept informed of other outputs.

Example 4: dissemination to specific stakeholders
A review of congenital heart disease services was commissioned to support a consultation process by NHS England. The high-profile nature of the issue required that the review be supported by a comprehensive programme of dissemination. In particular, the full review was included in the consultation pack and distributed widely to people with a specific interest in the consultation. The report was also disseminated through published journal articles .

Example 5: making the evidence more accessible
A favoured method of dissemination for the York centre involved the production of stand-alone evidence summaries tailored to the needs of NHS decision makers. These summaries, typically of four printed pages, followed a long-established format with key points on the front page followed by a more detailed summary, and were written to be as accessible as possible to non-specialists. The team developed an evidence summary to accompany their report on service user engagement in health service reconfiguration (Dalton et al, 2015). This broad topic area lacked a single clearly-defined target audience. The team therefore took a pragmatic approach, basing the evidence summary around six 'exemplars' of good practice. Exemplars covered service user engagement in urgent and emergency care settings; maternity, mental health, and eating disorder services. The evidence summary sought to deliver two main messages: what works when engaging service users and what is most important for future evaluation and reporting (Centre for Reviews and Dissemination, 2015). The summary was commended by both service providers and academics and later formed the basis of a conference poster presentation.
The evidence synthesis teams used a variety of dissemination channels, including journal articles, conference presentations and, to a lesser extent, social media. With the exception of one brief scoping output , all reports were peer-reviewed and published online, with the majority published in the Health Services and Delivery Research journal series. This approach was beneficial for dissemination of the teams' findings. Our experience has been that journal editors sometimes see potential duplication with the main report as a barrier. This may be overcome if the relationship between the two publications is clearly explained and a rationale is supplied for publication of an additional paper, although automated journal submission processes can be problematic. A number of articles from the programme have already been published with others under review or awaiting publication. Conference presentations are another traditional method of disseminating research findings and interacting with other researchers and research users. Because of the programme brief to address the needs of the UK NHS, the main focus of conference presentations has been the annual Health Services Research UK (HSR UK, formerly Health Services Research Network) conference. Team members have also presented at the Society for Social Medicine (SSM) annual scientific meeting.
Other approaches to making research more accessible include social media activity such as project blogs and the use of Twitter and Facebook to disseminate information and engage with the research community and the public. These activities are timeconsuming and may be of limited value unless carefully targeted. For this reason, the teams have made limited use of social media to date. An exception was an article on the widely read Mental Elf blog about the integrated care for people with serious mental illness project. Outputs from the evidence synthesis teams have also been disseminated via the national Dissemination Centre (https://www.dc.nihr.ac.uk/), which has a brief to serve a wide range of audiences.
The two evidence synthesis centre teams have used a variety of primarily traditional channels to disseminate the results of their projects. The teams have made slightly different choices about where to focus their efforts, and both have achieved some successes. The reviews highlighted in Examples 4 and 5 were relevant to areas of active decision making around service change, creating a favourable context for dissemination of the findings.

Enabling and documenting impact
A key objective of the evidence synthesis centres is to produce outputs of immediate use to the NHS. This not only involves addressing relevant topics and disseminating findings as already described, but also monitoring the uptake and use of outputs as far as possible. Impact may be demonstrated in diverse ways, most obviously when research is cited in support of a decision to change policy or practice. The review of congenital heart disease services was clearly used by stakeholders in the consultation process, and this was formally acknowledged in published meeting minutes (Example 6).

Example 6: documented impact on decision making
The congenital heart disease services review (Turner et al, 2014) was commissioned to support a consultation on 'Proposed congenital heart disease standards and service specifications: a consultation 15 September 2014 to 8 December 2014'. The formal minutes of the New Congenital Heart Disease Clinical Advisory Panel held on 18 June 2014 record that the panel directly discussed the findings of the rapid review. The Chair then asked members whether the findings would mean a change to the draft standards, and this was discussed as further documented in the published minutes. It was therefore possible for the review team and the funders to produce evidence of an immediate and unusually direct impact of the research on the service review.
The evidence synthesis teams also undertook projects to inform future research commissioning. This required an emphasis on identifying evidence gaps alongside summarising existing evidence. Reviews with an impact on research commissioning included those on group clinics  and urgent and emergency care systems .
A project on reporting standards for organisational case studies (Rodgers et al, 2016b) was requested by the HSDR programme to address concerns about the quality of case study research proposals submitted to the programme. The project was unusual in that it involved a Delphi consultation exercise as well as a review of relevant literature. The team successfully engaged with the research community to achieve a broad consensus on the proposed standards as described in Example 7.

Example 7: impact through engagement with a research community
The objective of the project was to develop reporting standards for organisational case study research, with particular application to the NHS (Rodgers et al, 2016b). The research team was concerned that standards derived purely from a review of the literature might not be accepted by those working in the field. They therefore developed a hybrid methodology combining a rapid review with a modified Delphi consensus process. An initial pool of possible items for inclusion in the standard was derived from a review of the methodological literature. These items were then rated in two rounds by a Delphi panel of experts, all of whom had direct involvement with case study research. Participants were identified by the review team from the rapid review, personal contacts and by contacting relevant organisations such as the Health Services Research Network and the Social Research Association. Data on personal characteristics were collected to assess representation of different stakeholder groups and identify any important differences in their responses.
The final checklist consisted of 13 items for which there was a high level of consensus. The standard has been accepted and made available by the EQUATOR (Enhancing the Quality and Transparency of Health Research) reporting guidelines network. The hybrid approach was key to achieving this degree of impact in the field as it enhanced the credibility of the process by engaging researchers, and allowed those with concerns to participate and have their views incorporated into the process.
The above examples illustrate impacts achieved by two evidence synthesis centre projects. In both cases, early and appropriate engagement with stakeholders was critical to the outcomes achieved. Traditional metrics of academic impact, such as citations and the impact factor of journals in which papers are published, take time to accumulate and are less relevant where the objective is to be useful to policymakers and practitioners. The use of research evidence by NHS decision makers is difficult to document, as illustrated by a recent study of a responsive evidence service for clinical commissioning groups (Wilson et al, 2017).

Discussion and conclusions
This paper describes and illustrates key issues involved in rapid evidence synthesis programmes aiming to provide useful evidence for health system decision makers and research commissioners. Both core review teams comprised systematic reviewers rather than clinicians or topic experts. Both teams therefore faced potential barriers related to limited topic expertise in some areas and lack of knowledge of the underlying policy context behind some of the topics we were asked to address. We therefore augmented our in-house knowledge by calling on other sources of clinical and policy expertise where necessary. Production of high-quality outputs in a timely fashion was facilitated by such key processes as initial mapping and scoping of the available published evidence; early engagement with stakeholders to optimise their involvement with limited time and resources (Example 3); and willingness to consider creative solutions and different ways of working to overcome problems encountered in specific projects (Example 7).
The findings presented represent diverse evidence review topics across the broad area of health services research covered by the funding programme. Almost all of the outputs from both evidence synthesis centres have undergone editorial and peer review in line with standard processes, and the projects have generated numerous additional publications and conference presentations. Although both teams worked independently and developed different ways of working, this paper represents a joint reflection on the experience of running the programmes over the first three-year contract. One limitation is that this paper presents our perspective as researchers; evidence users, topic experts and other stakeholders may have different views.
The decision to commission a second three-year programme with a third review team added (see Background above) reflects both the value of the two evidence synthesis centres to decision makers and the need to increase capacity to deliver timely evidence synthesis. The new programme will provide opportunities to further develop the services and to take forward a research agenda around rapid evidence synthesis methods. Expert stakeholder involvement is an area to which those setting up similar programmes should pay particular attention. This includes both interaction with topic experts to guide the scoping and conduct of reviews and the effective dissemination of findings to health professional audiences. Other areas identified for attention include optimising the use of software to support the review process (including text mining); standardisation of quality of reporting to reduce the time spent on report editing and hence speed up publication; and improved public involvement, both for the programmes as a whole and for specific projects.Compared with other models for commissioning rapid evidence synthesis projects (that is, commissioning projects individually and evaluating proposals submitted by researchers (researcher-led)), the evidence synthesis centre model offers the potential advantages of improved timeliness, flexibility and the potential for improved working relationships and collaboration. One possible disadvantage is greater reliance on experts located outside the review team who may be less committed to offering advice and comments when requested. This factor is partially offset when experts are from the review team's own institution. For example, the Sheffield team had access to several experts in urgent and emergency care. However, in general the review teams were able to obtain only limited comments from topic experts on most draft reports before peer review.
This account has highlighted some examples of impact from our work to date. However, evidence use and its integration with other considerations to aid decision making at the local level is challenging to document for a programme with a national focus and limited resources. This may be one reason for the increasing interest in researcher-in-residence models to promote links between evidence producers and users (Marshall et al, 2014). Research is ongoing to understand and optimise this type of intervention.
There is a substantial body of research on health system decision makers' use of evidence in general (Oliver et al, 2014;Whitty, 2015), and of systematic reviews and other types of evidence synthesis in particular (Murthy et al, 2012;Tricco et al, 2016). Our experience supports the value of working with decision makers in planning and conducting evidence syntheses and of summarising the findings in an accessible form. However, the barriers identified in earlier research, such as timeliness and documenting the actual use of evidence to support decisions, remain relevant. Encouragingly, a recent study based on interviews with Australian policymakers found that 89% of commissioned rapid reviews were used by those who commissioned them, with 338 instances of use being identified for 139 reviews (Moore et al, 2018).
Responsive evidence briefing services for decision makers based on existing systematic reviews have existed for some time (Chambers et al, 2011). One such service was recently evaluated with clinical commissioning groups in northern England, but did not improve uptake and use of research evidence compared with less intensive alternatives (Wilson et al, 2017). The model we have described in this paper, namely centrally commissioned de novo rapid evidence synthesis to respond to national priorities, appears promising but further development and evaluation is needed.
In summary, this three-year programme has covered a wide range of topics prioritised for evidence synthesis by the funding programme team and/or NHS stakeholders. The review teams have developed ways of working that have enabled delivery of high-quality outputs to an agreed timetable. The rapid timescales have required a particular emphasis on clarifying the scope of each project (often by an iterative process) and understanding the intended purpose(s) of the project outputs. We have confirmed the experience, reported by others, of the value of using scoping and mapping activities as staple processes in advance of a formal evidence synthesis (Rebelo Da Silva et al, 2017). The continuation of the programme for a further three years offers an opportunity to build on the review teams' experience to date and further improve the service we offer to the funding programme and the NHS.