Searching for evidence in public health emergencies: a white paper of best practices

Objectives: Information professionals have supported medical providers, administrators and decision-makers, and guideline creators in the COVID-19 response. Searching COVID-19 literature presented new challenges, including the volume and heterogeneity of literature and the proliferation of new information sources, and exposed existing issues in metadata and publishing. An expert panel developed best practices, including recommendations, elaborations, and examples, for searching during public health emergencies. Methods: Project directors and advisors developed core elements from experience and literature. Experts, identified by affiliation with evidence synthesis groups, COVID-19 search experience, and nomination, responded to an online survey to reach consensus on core elements. Expert participants provided written responses to guiding questions. A synthesis of responses provided the foundation for focus group discussions. A writing group then drafted the best practices into a statement. Experts reviewed the statement prior to dissemination. Results: Twelve information professionals contributed to best practice recommendations on six elements: core resources, search strategies, publication types, transparency and reproducibility, collaboration, and conducting research. Underlying principles across recommendations include timeliness, openness, balance, preparedness, and responsiveness. Conclusions: The authors and experts anticipate the recommendations for searching for evidence during public health emergencies will help information specialists, librarians, evidence synthesis groups, researchers, and decision-makers respond to future public health emergencies, including but not limited to disease outbreaks. The recommendations complement existing guidance by addressing concerns specific to emergency response. The statement is intended as a living document. Future revisions should solicit input from a broader community and reflect conclusions of meta-research on COVID-19 and health emergencies.


1A. Search both traditional bibliographic databases and emerging resources of evidence
The number and types of resources searched depends on the informational needs; question(s) being asked; urgency; and access to resources. Searchers should understand research needs and the information landscape [37] and be flexible in their approach.
Essential elements: • Most requests could be fulfilled by searching two to three traditional databases.
• Subject-specific databases should be used for targeted questions.
• Traditional databases offer a breadth of content, usability, and documentation and should be searched for evidence directly and indirectly relevant to the emergency.
Preprint servers and open access collections should also be searched. • Searchers should address overlapping coverage when searching multiple databases, for instance by using reference managers to compare results and remove duplicates.
Additional elements: • Grey literature should be considered as it contains relevant information that may not make it to mainstream journals. • Databases specifically cataloging retractions [38] may be pertinent, particularly for rapidly-evolving topics and longer-term evidence syntheses, i.e. living reviews [39].
• Monitor public health emergency research developments across all languages.
Example: The National Collaborating Centre for Method and Tools' (NCCMT) Rapid Evidence Service (RES) searches specialized evidence sources, including preprint servers and COVID-19specific collections, as well as traditional databases, for each COVID-19-related rapid review [40]. The RES monitors changes in the search strategies employed by each specialization COVID-19 collection and updates their search strategy.

1B. Consult documentation to critically assess new and emerging resources
When considering new and emerging resources, review their creation and curation documentation, generally found within the resource or as a research article, to assess the following essential elements [31,41]: Essential elements: • Information about the resource creators and their credibility; rationale for resource's creation; resource aim and scope • Creation and curation methodologies; types of resources included and their uniqueness • Search strategies, screening processes, other workflows, if the content is sourced from other databases • Maintenance methodologies such as updates/search frequencies; records ingestion; subject indexing; or record enhancements Additional elements: • Consider how artificial intelligence (AI) and machine learning were used in curation to intelligently analyse data on a large scale [42] • Seek studies evaluating performance of novel resources • Seek documentation that details linkages, such as between preprints, published articles and errata or retractions.

2A. Broadly, follow Cochrane Rapid Review methodology for developing a search strategy
Although public health emergencies can impose resource restrictions, adhering to best-practice guidelines for review methodology [49] improves the quality of the resulting evidence synthesis. Specific deviations for searching methods in an emergency context are outlined in the following recommendations. Balance between sensitivity/recall and precision will depend on the context and question.
Essential elements: • Work with stakeholders to clearly define search concepts, develop a protocol, and identify key papers. • Engage an information specialist to design and conduct the search.
• Include date limits where appropriate for a time-bound emergency to reduce irrelevant retrievals.
• Provide justification for other limits, recognizing any resource limitations (50

2B. For emerging concepts, use a variety of sources to capture the latest terminology and reevaluate terms regularly
Emerging concepts may not be well represented by index terms, and inconsistent terminology can impact retrieval [54]. In addition to traditional terminology sources, such as subject headings, alternative sources -including professional listservs, social media, government bulletins, and news -can identify evolving terms.
Essential elements: • When a concept is emerging, re-evaluate search strings frequently -at minimum monthly. For example, COVID-19 terminology evolved rapidly in the emerging phases. Later, terminology had standardized, but additional terms were needed to capture variants.

2C. Particularly in the early phases of an emergency, do not limit by publication type
The COVID-19 pandemic saw much early literature being published as letters, comments, or preprints. These publications, often filtered out in evidence syntheses, were vital sources of information early in the emergency.
Essential elements: • Early in an emergency, avoid publication type filters unless justified by the specific question and context. • Be aware of potential discrepancies between non-peer-reviewed literature and later peer-reviewed results. • Monitor retractions [61] and adjust reviews accordingly.
• Use new and existing tools and guidance from trusted organizations for locating and sharing evidence syntheses.
Examples: Metzendorf and Featherstone's evaluation of the Cochrane COVID-19 Study Register [62[ found many studies published as letters early in the pandemic. This observation led to a revision of the Register's search strategy: letters were re-evaluated to ensure adequate coverage of these publication types.

2D. Carefully consider the use of language filters
Limiting searches to English-language publications expedites the completion of rapid reviews [49]. This approach can miss non-English language research local to the impacted area(s). Use of a language filter due to capacity restrictions should be acknowledged as a limitation.
Essential elements: • Include publications in languages common to areas impacted by the emergency.
Additional elements: • Where feasible, incorporate local-language terms into the search strategy. Consult regional experts or local-language publications, including news media.
Examples: Tricco, et al. [29] advocate including studies in all languages for COVID-19-related rapid reviews. Levay and Finnegan note that "some references relied on country-specific phrases", suggesting "localised terms in conjunction with a geographical search filter to capture relevant additional literature" [59].

2E. Consider a "universalized" approach to searching to simplify translation between databases
Many traditional databases have advanced features, including subject headings. Databases established during a public health emergency are often more limited in their functionality. A search that can be readily translated between databases streamlines searching.
Essential elements: • Be aware of limitations of the databases being searched and adapt the search.
• When searching a database with hierarchical index terms, spell out narrower terms rather than relying on term explosion [59]. • Ensure the keyword list is comprehensive and, where possible, consider phrase searching over proximity operators since more databases include the former.
Additional elements: • Evaluate the contribution of different databases to the results of evidence syntheses. Consider the use of search summary tables if time allows [63]. • Share information about database coverage and limitations.
• Work with providers to validate novel databases.
Examples: L*OVE [44] uses a question builder design that may be precise for some topics and inappropriate for others. Carefully consider the process to translate precise strategies from established databases into emerging resources as direct translation may not be possible.

Publication Types
During public health emergencies, publication trends may change quickly. Existing guidance on finding and managing non-peer-reviewed publications is not always sufficient for emergency settings. Non-peer-reviewed publication types include research outputs like preprints, clinical trial registration records, and datasets; and media publications like news articles and press releases. Recommendations in this section address how to utilize, monitor/track, and contextualize non-peer reviewed literature.

3A. Evaluate current trends and incorporate searching for critical publication types
Timely information is crucial; it is important to consider the channels used to quickly disseminate emergency-specific research.
Essential elements: • Publication types that can be disseminated faster are favored, including preprints, letters, reports, and rapid reviews.

3B. Develop a strategy and identify tools to monitor and track publications
Non-peer-reviewed publications are more dynamic, and publishing norms, platforms, and tools evolve during emergencies. A robust strategy is needed to monitor and update publications. Tools and strategies should be evaluated on ease of implementation, team capacity, automation features, etc.
Essential elements: • Follow clinical trials updates using a sensitive search string in a central database.
• Set up search alerts for trial registry numbers.
• Build communities of practice to share new evidence; see Recommendation 5D.
Additional elements: • Additional tools may be needed to locate multiple publications on the same study.

3C. Clearly identify references by publication type for review teams and end users
When incorporating non-peer-reviewed publications into search, transparency is crucial. Searchers should identify non-peer-reviewed publications for users and provide context if needed.
Essential elements: • Document peer review status and tag references by publication type to avoid user confusion and enable quick appraisals. Clear visual indications reduce confusion and allow the user to make quick appraisals. • Provide notes about versioning and edits across publication versions.
Additional elements: • Use systems such as metadata, signposting, and/or formatting (e.g. title [preprint] Essential elements: • Consider PRISMA-S requirements before beginning.
• Document as the search is being conducted.
• If search strings from other sources were incorporated, provide credit.
• Use the PRISMA-S checklist [19] to confirm all requirements have been met.
Additional elements: • Note limitations or uncertainties with new sources. Advise whether the source is likely to produce reproducible results, based on current performance. • Acknowledge any limits imposed due to resource constraints.

Examples:
For examples of transparently reported reproducible search strategies created by information specialists see et. al. [87] and Okoli,et. al. [88].

4B. Practice open, transparent, and reproducible data management whilst working on metaresearch related to public health emergencies
In an emergency, findings should be rapidly and freely availably to a wide audience.
Essential elements: • Research related to public health emergencies should be published open access.
• Data should be shared using FAIR principles [89].
Additional elements: • Researchers should write data management plans (DMPs) describing types of data to be collected and conditions for data storage, access, and preservation.

Collaboration
Given the resource limitations associated with public health emergencies, openness and collaboration are key for improving evidence synthesis and reducing duplication of effort.
Recommendations in this section address the need for collaboration among information professionals as well as with other stakeholders.

5A. Develop a centralized repository for sharing search strings and strategies
Sharing search strings supports rapid responses and reduces duplication of effort. A centralized repository streamlines the process of locating them and allows for inclusion of associated metadata. This repository could be hosted by a collaborative organization (see 5D).
Essential elements: • Information professionals should share search strings or strategies with associated metadata, including creation/update date, inclusion/exclusion criteria, justification for limits, and name and credentials of the searcher. • Cite sources when reusing search strings for publication. Use a recommended citation where provided.
Additional elements: • Metadata could describe efforts to validate the search, such as having it peer reviewed or testing retrieval of known items. • Search strings and strategies should be licensed for broad reuse, for instance Creative Commons Attribution (CC BY), or shared without any attached licensing requirements. • The centralized repository could include other search-related products such as database evaluations.

5B. Share full protocols for evidence synthesis projects that are planned or in progress
Protocol sharing reduces duplication of effort by making researchers aware of work underway.
Essential elements: • Before beginning a new project, search for protocols to identify similar projects in progress.
• Include the research question and inclusion/exclusion criteria. Additional elements: • Update protocols to reflect changes in timeline or scope, or to note discontinuation.
• Where researchers discover a systematic review protocol that is similar to a review they plan to conduct themselves, there may be an opportunity to collaborate. , and specialist journals such as Systematic Reviews [26]. As with recommendation 5A, a broad centralized repository would improve protocol findability and reduce duplication of effort.

5C. Information professionals should conduct and evaluate searches in evidence syntheses and their roles should be acknowledged according to the CRedIT framework
Involving information professionals in the development of search strategies significantly improves their quality [86,98,99]. Content specialists can effectively critique research in their subject area but may be unfamiliar with search methodologies. Research leads should ensure the involvement of information professionals in evidence synthesis projects.
Essential elements: • Information professionals serve as essential collaborators in scoping reviews, validating search terms, developing search strategies, and managing citation data. They should have authorship for their intellectual research contribution in accordance with International Committee of Medical Journal Editors (ICMJE) criteria [100].
Additional elements: • Publishers of evidence syntheses should involve and acknowledge information professionals as peer [52,101]. • If resource limitations preclude the involvement of an information professional, this should be stated as a limitation. • Depending on capacity, information professionals may be involved in other evidence synthesis tasks, for example, project or data management. • Information professionals lead in meta-research projects by identifying gaps in the literature, applying for grants, and proactively looking for collaborators.
• Information professionals should elevate related advocacy efforts, e.g., those of Rethlefsen,et. al. [99], who propose that librarians play a central role in the development of all reviews.
Examples: Information specialist participation in evidence synthesis significantly improved reproducibility and adherence to search standards [102]. " [T]o minimize bias in SRs [systematic reviews], authors and editors could encourage librarian engagement in SRs including authorship" [102].

5D. Collaborative organizations of library professionals should support work related to public health emergencies
Formal networks improve collaboration among information professionals across geographic and organizational contexts. These could include volunteer task groups, professional organizations, communities of practice, or a central registry of librarians available to review search strategies.
Essential elements: • Information professionals and researchers should be made aware of these networks and their value.
Additional elements: • These networks can provide training/mentoring to support capacity building. •

5E. Information professionals and professional organizations should be active in advocating for improved access to research
Paywalls challenge the ability of researchers to access and respond to key research findings, particularly in the context of public health emergencies in which rapid access is essential. Lack of database access complicates comprehensive searching. Information professionals should collaborate with publishers and vendors to support mechanisms by which access can be made more streamlined and less expensive, particularly through open access models.
Essential elements: • Support the suspension of paywalls and the availability of relevant research, regardless of subscription or payment.
• Communication across disciplines must increase to improve coordination and support.
Additional elements: • Information professional volunteer organizations should include a variety of domains.
• Administrators should support cross-organizational collaboration and ensure libraries are not siloed. • There is a strong need to map vocabularies and perspectives across disciplines to improve cross-domain communication.
Example: Groot,et. al. [111] bring together a COVID-19 Evidence Support Team of librarians and clinical experts to respond to research queries. They describe this multidisciplinary team as "a catalyst to begin to eliminate… silos and move towards collaborative rapid learning" [111].

Conducting Information Science Research
During the COVID-19 pandemic, information professionals raced to develop systematic search strategies and artificial intelligence algorithms to identify relevant research. Quick and collaborative validation of these methods has been essential to ensure confidence in their comprehensiveness and utility. This section recommends best practices for the conduct of meta-research to support evidence-based information responses to public health emergencies.

6A. Review the evolution of information needs and evidence and evaluate actions taken during the COVID-19 pandemic as well as previous public health emergencies
Preparedness and planning are key to efficient response. The evolution of information needs in the COVID-19 pandemic and in previous emergencies can be reviewed to anticipate future information needs. Reviews of such practices, information needs, and how each should be prioritised will inform what future research may be required.

Essential elements:
• Reviews of the information response to public health emergencies should focus on identifying and prioritising research questions [1] across biomedical, social, or economic domains. • Reviews should compare the response with national emergencies which did not reach international concern.
Additional elements: • A cross-disciplinary, collaborative review on the evolution of information needs across domains (biomedical, social, and economic) would help identify wider research questions. Such a review should cover the nature, scope, and quality of evidence as the emergency developed. • During the recovery phase, focus groups and surveys should be conducted to identify research needs beyond those reflected in the publication record. Time and resource constraints during emergency situations often prevent these meetings, so preparation, prioritisation, and anticipation of future emergencies is key.
Examples: Several reviews characterize the evidence that emerged early in the COVID-19 pandemic [17,68,112], the quality of research [4], and the evolution of research topics [113]. Tools such as the taxonomy created by COVID-END could help information professionals anticipate and describe future research questions [114].

6B. Common infrastructures and processes required to respond to information needs and conduct meta-research should be identified, anticipated, put into place, and appropriately funded ahead of future public health emergencies
Although different emergencies will have varying information needs, the infrastructure and workflows required to meet these needs will remain relatively unchanged. Information professionals can prepare for future emergencies by considering actions they might take to respond. This work needs to be valued and appropriately funded.
Essential elements: • Common infrastructures, including systematic review protocol repositories, bibliographic databases, preprint servers, and search string repositories (see 5A) should be identified or developed. • Information professionals should develop workflows that enable searching, deduplication, and annotation of relevant publications. • Information professionals should prepare for similar or greater quantities of research in future emergencies. Preparation may include ensuring computing capacity and developing trust and expertise in the uses of artificial intelligence to accelerate analysis.
Examples: The pre-existing systematic review citizen science platform, Cochrane Crowd, was successfully used to coordinate Cochrane's response to the COVID-19 pandemic [115]. Hair,et. al. [116] developed an adaptable workflow to allow searching, deduplication, screening, and annotation of primary COVID-19 research publication records using R programming language.
6C. The performance of artificial intelligence used to accelerate or aid meta-research efforts in public health emergencies should be rigorously validated before use