Collaborating to plan efficient public health guideline evidence reviews : understanding which sources retrieve evidence

Journal of EAHIL 2021; Vol. 17 (1): 9-13 doi: 10.32384/jeahil17449 Abstract The National Institute for Health and Care Excellence (NICE) produces public health guidelines. Information specialists collaborate with technical analysts to perform systematic searches for evidence reviews (ERs). Public health ERs require searches from multiple disciplines across a range of sources leading to high volumes of results. The purpose of the project was to provide evidence to support the choice of sources for new topics. It aimed to retrospectively analyse a sample of NICE public health ERs by examining which sources retrieved publications. Medical databases found the highest proportion of publications, but smaller subject focussed databases and search techniques also contributed. These findings justify use of a range of sources for public health reviews and help the planning of ERs.


Introduction
The National Institute for Health and Care Excellence (NICE) is an independent organisation based in England that produces a range of guidelines. NICE public health guidelines aim to promote healthy living and prevent ill health. Information specialists (IS) have a key role in both the scoping and development of all new guidance at NICE. They collaborate closely with technical analysts and advisers to perform systematic searches for the evidence reviews that support the recommendations for NICE guidance. The search process for any topic is lengthy. It starts with searches for guidance (from NICE and other organisations) and systematic reviews on the topic in question. This helps the analysts to gain an insight into what is already published on the topic and to discover what the gaps in evidence are. During the guideline development phase the IS develops a strategy in MEDLINE which is run and translated into other databases e.g. Embase and Cochrane Library. Additional search techniques may be used to find further results. The process includes writing the search protocol and keeping records of methods and search dates to aid the transparency.

Public Health evidence reviews
Public health topics can be more challenging to search than clinical topics. There are a number of reasons for this but key factors are the type of evidence needed to answer these questions and the volume of results that can be found. The evidence for public health guidelines may be found in medical databases like MEDLINE and Embase. However, searches may also need to be undertaken in databases with a different disciplinary focus. For example, a recent NICE guideline "Alcohol intervention in secondary and further education NG135" required searches in education focussed databases as well as medical. There may also be useful evidence in websites or unpublished reports. Locating this evidence requires additional searching. Searching itself can contribute to the large volume of results because the terminology used to describe public health interventions is complex. The concepts can be difficult to define. For example, a search on a behaviour change technique would require search terms covering behaviour change as a term with various iterations as well as terms covering the behaviour that was being changed, for example stopping smoking. For clinical topics, it is often appropriate to limit Andrea Heath, Paul Levay and Daniel Tuvey searches by study type to reduce the number of results. For example, if the research question is answerable by randomised controlled trials (RCTs) this would be a suitable study limit. However, for public health guidelines, it may not be possible to limit results in this way. RCTs may not be the most or only appropriate study type to find the evidence, meaning that other study types should be included. Consequently, the volume of results for public health topics is usually high. This results in more time needed for searching additional sources, uploading results and de-duplicating results. With this challenge in mind, NICE began an investigation into where the evidence came from for a sample of NICE public health guidelines. The aim of the project was to discover if databases and search techniques used in public health topics were essential for identifying all the included publications in the evidence reviews.

Methods
The project was initiated in 2016 and covered ten guidelines in total ( Figure 1). Most of the guidelines had more than one evidence review which resulted in analyses of 29 evidence reviews. Public health evidence reviews illustrates an example of the search summary table used to collect data from each search. For each search, an un-deduplicated Endnote Library was retained. The publications included in the evidence reviews were found and inserted into the table. The un-deduplicated Endnote library was searched to check which database or source had found each included publication. The unique sources for each review were established by checking which included publications had been retrieved by a single source.

Which sources retrieved publications included in the evidence review?
Which databases did we need? Across the reviews, 28 databases were used ( Figure 2). Seven of the 28 databases are routinely searched for all public health reviews. These are CDSR, CENTRAL, DARE, Embase, MEDLINE, MiP and SPP.
MEDLINE, Embase, CDSR and PsycINFO (when searched) found over two thirds of the included publications in many reviews. In some cases the figure was much higher. For example, Embase found 86% of included publications for SH. However, for some reviews it was the smaller, subject focussed databases that found a large percentage of the included publications. For example, for TRAP (Evidence review 1) Greenfile found 52% of included publications. Five databases did not retrieve any included publications at all. These were Cochrane HTA, Community Pharmacy Future, Health Evidence, Social Welfare and Sociological Abstracts.
What other search techniques did we need? Across the reviews, eight techniques were applied (Figure 3). In-cluded publications were found this way for some reviews. SP (Evidence Review 1) found 91% of included publications via citation searching. DMP (Evidence Review 1) found 31% of included publications from citation searching and 60% from reference harvesting. However, for some reviews, no included publications or very few, were found by using additional techniques. For example, handsearching did not produce any included publications for any review.

Which included publications were retrieved by a single source?
Databases. Sixteen of the 28 databases found at least one included publication that was not found anywhere else. Medical databases (CDSR, MEDLINE, Embase) accounted for high proportions of unique included publications for some reviews. For example, CDSR found 81% of unique included publications for SSIS and MEDLINE found 67% of unique included publications for SP (Evidence Review 8). Smaller databases also found included publications that were not found anywhere else. SPP found the only unique included publication for SP (Evidence Review 2). Search techniques. Five of the eight techniques used retrieved unique included publications. The highest proportion was in DMP (Evidence Review 1), where 83% of the unique included publications were due to one of these techniques. This included 50% from reference harvesting and 33% from citation searching. This was followed by SP (Evidence Review 7) where 25% were found from citation searching. Website searching led to a unique included publication seven of the 28 times that it was used.

Reflections
The results illustrate that both commonly searched medical databases and smaller subject-focused databases have a place in public health search strategies. Additional search techniques can also provide included publications for some topics. Five databases did not contribute any included publications and so it

Fig. 3. Techniques used in the 10 evidence reviews
Andrea Heath, Paul Levay and Daniel Tuvey would be possible to reduce the sources for individual topics without losing relevant publications. However, the knowledge of which sources find relevant results and which don't only comes after searching and screening are complete. Therefore, given the range of topic areas within the public health discipline, dropping any one source globally could compromise future guidelines. The results suggest that NICE is justified in using a wide range of sources to support public health guidelines. A benefit of the work was the development of a robust methodology that can be applied in other settings. For example, the methods used in the project were applied to recent work undertaken for COVID-19 guidance and used to pinpoint which sources were finding useful evidence.

Developments: database selection
The project results have helped NICE to make decisions about which sources to search for future topics and to supply data for scoping searches. For example, the project findings from HIV and SH were used by an IS for a new topic "Reducing sexually transmitted infections" to define which sources to search. The IS looked at which sources had returned included publications for HIV and SH and, after performing some tests, was able to make the decision not to search the databases that had not previously been productive. There was a large body of evidence for this topic so being able to use the project findings to pinpoint which resources were most useful made it possible to focus the search.   Public health evidence reviews The project methodology has also been useful for searches for publications to inform the decision to update a guideline and the evidence reviews that inform the update of guidelines. This is because it helps to target which sources should be searched. For example, sources could be dropped if they have been unproductive in the original search.

Collaboration
Information specialists collaborated with two internal teams during the project. Public health analysts had been aware of the project since its inception as it would directly affect their work. The protocols for new or update topics can be evidence based, as the IS can review which sources were useful for similar topics and use this information in discussions with analysts about choosing the right sources to search. For example, website searching can be time consuming but ISs have been able to demonstrate when it has been able to provide evidence that would not have been found anywhere else. This information also helps to plan resources for the development of a new or updated guideline. Knowing which sources are likely to yield useful results and factoring in the time needed to develop a strategy, search and then sift helps to plan a realistic timescale. The project illustrated the benefits of continually evaluating which sources are most useful. However, the original data collection method via Excel spreadsheet could be time consuming. As a manual process, mistakes were more likely to occur due to spelling discrepancies or other manual errors. Consequently, ISs collaborated with the Digital, Information and Technology (DIT) team at NICE to develop the functionality to support the continued surveillance of sources by automating the process. Since the project was initiated, NICE has moved from using Endnote to EPPI Reviewer for evidence management. The DIT team has made it possible to view an automatically generated report that includes some of the features of the search summary table (Figure 4).

Conclusions
Research into which sources found included results in a list of ten public health guidelines found that most sources are useful for finding included publications. For some topics, smaller, more focussed databases and search techniques retrieved included publications that were not found elsewhere. The results indicate that NICE information specialists have been making the right decisions about which sources to search. Additionally, the results have been useful in planning searches for new and update topics. They also aid collaboration with the Public Health team by offering evidence to support decisions. Collaboration with the DIT team resulted in an automated method to collect data for this beneficial methodology. The next stage in this work is to devise and report on scenarios that describe in more detail the implications of this research.