A rapid response to the COVID-19 outbreak : the meta-evidence project

Early in the pandemic, as scientific reports and preliminary research on both clinical and public health aspects of COVID-19 were rapidly generated, we recognised the need for a dynamic, interactive tool that could capture and collate emerging evidence sources to inform research and decision-making efforts. In particular, we observed that numerous similar research efforts across the globe were happening in parallel prompting an urgent need to connect research teams with each other and maximize research efficiency. Our colleagues in China provided daily translations of emerging evidence to aid networking between research groups working across the world. Here we describe how the meta-evidence project met daily and ongoing challenges and what was learned as a result. We describe the benefit of finding ways to instead work with better resourced teams and promote collective and open efforts to synthesise the evidence, which in the end, outweighed the considerable costs.


Background
On 11 March 2020, the WHO declared the SARS-CoV-2 outbreak a global pandemic and as of 29 April 2021, there have been 147,443,848 confirmed cases and 3,117,542 deaths (European Centre for Disease Prevention and Control figures). As the global spread of COVID-19 continues to grow, disease control and prevention will be challenging, and this requires collaborative solutions and cooperative spirit from all groups. There has been a tremendous response from the scientific community to generate timely and responsive research, which has translated to an exponential growth in COVID-19 related research literature ( Figure 1). It is estimated that there are at least 129,570 COVID-19-related publications to date (1). While this wealth of research is a potential boon for addressing both multi-dimensional aspects of the pandemic, from clinic to social, the reliability and rigor of these papers is quite variable. In fact, a study conducted two months after the pandemic was declared found that most of the papers being published had a shorter time to publication and were of lower methodological quality than matched control studies on other clinical studies from the same journals (2). Moreover, many papers are quite similar in topic -for example, as of the 30 April 2021, there are approximately 5,590 health systematic reviews on COVID-19 (L.OVE platform, Epistemonikos foundation) with many on the same topic. One research team identified 25 systematic reviews reporting on 17 primary studies, all answering the same question of interest (3). This overlap and duplications signal that much of the race to research COVID-19 has resulted in "research waste" (4) and that research efforts could be combined to produce more rigorous and/or comprehensive insights.

The role of evidence synthesis and semi-automated text-mining bots
The rapidly evolving landscape of knowledge with respect to viral biology, disease presentation, clinical outcomes, social and economic impacts, and potential treatments and prevention required and continues to require a rapid, dynamic approach to synthesize emerging information to inform on-the-ground decisions. Evidence syntheses (including systematic reviews, evidence and gap maps, scoping reviews, and A rapid response to the COVID-19 outbreak meta-analysis) are critical tools for collating and synthesising insights from the broad evidence base using transparent and reproducible methods. The information-poor and high-risk context of the COVID-19 pandemic requires rapid approaches for evidence synthesis (5) to provide and communicate reliable summaries of emerging evidence and inform and update clinical guidelines and public health recommendations. Over this past year, while many syntheses (including systematic reviews) have been conducted, the pace of knowledge production means that these reviews become rapidly outdated and lose both relevancy and accuracy (5).
Dr Gabriel Rada, co-founder of Epistemonikos, describes reviewers missing relevant studies and outlines an unpublished analysis which found 95% of reviews about drug treatments in COVID-19 were out of date due to rapid publication of new clinical trial data (6). In addition, the rapid production of reviews to keep up with the pace of publications has resulted in the production of reviews with lower methodological quality ˗ thus reliability ˗ which is further exacerbated by the acceleration of the publication process, often skipping or rushing the peer review process (7). We developed the COVID-19 twitter bot (www.twitter.com/@COVID_Evidence) to harness, in real-time, the emerge of COVID-19 research. The bot, which has now been active for over a year, produces a diverse range of real-time research and commissioned reports directly onto a twitter feed using the RSS sources from a range of science and medical databases. The COVID-19 Twitter Bot was one of the first sources to emerge with a focus on real-time acquisition and collation of research findings about COVID-19. This bot created by an evidence synthesis expert and a consultant kidney physician was capable of persistently posting relevant content without requiring sustained human involvement past its creation.

Building a collaborative "living" atlas of COVID-19 research
Building and launching the COVID-19 Twitter Bot allowed our group to collate significant bibliographies of emerging research on COVID-19. Given growing limitations on resources under economic impacts from the pandemic, we saw a need for a living atlas that could aid networking between research groups working across the world. Having this knowledge could help future research efforts be more targeted towards existing gaps, reduce inefficiency and duplication, and foster collaboration. We built an interactive, visual database of COVID-19 research that featured an interactive geographical map that reflected emerging evidence sources (e.g. articles and resources) collated from the automated aggregating Twitter feed, and supplemented by Ciara Keenan, Chris Noone, Karen McConnell and Samantha H Cheng sources such as the WHO database ( Figure 2). We named this endeavor the meta-evidence project. The interactive geographic map was powered by EviAtlas, an open access platform for visualizing synthesis data (8).
In particular, at the time we were building the map, the literature being captured by the bot was primarily in English and limited to geographies with users present on Twitter. Thus, there were significant gaps in coverage, particularly research emerging from China, who at that stage, had more lived experience of the virus and disease than the rest of the world. Dr Howard White, CEO of the Campbell Collaboration collaborated with Professor Kehu Yang, Director of Lanzhou University's Evidence Based Medicine Centre in China to compile and translate evidence from sources in Chinese to expand the map's coverage. This work was made possible given the generous pro bono effort from Professor Yang and a team of seven researchers.

Challenges
The meta-evidence project responded rapidly to the global spread of COVID-19 at a time when rumours and conjecture were also spreading through social and mainstream media, this pace was met with various and significant challenges. First, the "living" atlas was dependent on the resources of volunteers who were building the atlas in their spare time. This effort was borne from a tweet asking for help and pulled in medical professionals, data scientists and evidence synthesis experts from all disciplines, globally, and highlighted a true sense of community contribution. However, this meant that volunteers were working on the map in their "spare" time, typically after long workdays and during a time when spare time was already being eaten due to closures of schools and daycare facilities. Also, we learned of much-better resourced groups like the EPPI-Centre, COVID-NMA and the Norwegian Institute of Public Health who were, and still are working hard to avoid out-of-date research by producing living maps of Evidence. Second, we noted that publishers had relaxed some of their strict guidelines. This meant that many studies had been widely published without sufficient peer review. We felt that it is important to incorporate an assessment of quality so that readers can understand which studies are likely to be reliable. However, this was something we simply did not have the people power to do. Third, pandemics such as COVID-19 require expedited data and research findings to help understand the situation and potential treatment and vaccines. Rarely though does a pandemic affect all sectors of society and the research community on such a global scale. Due to the impact of COVID-19 across all sectors of society, the research was and still is being produced in large volumes across all disciplines which made it difficult to categorise. This difficultly meant that it was difficult to continue map- (8).

Fig. 2. A screenshot of the interactive geographical map built using EviAtlas
ping the research on the atlas in a meaningful and useful way for individual disciplines. Finally, after conversations with key people in leading organisations including Cochrane, Campbell, EPPI-Centre, Evidence Aid, Evidence synthesis Ireland, it became clear that this work had been overtaken by other teams with dedicated resources. One of the most influential conversations had in this time was with Professor Mike Clarke, founder of Evidence Aid and expert in the human response to humanitarian disasters. After speaking with Prof Clarke, we realised that our effort, although commendable and extremely useful at the early stage, was also adding to research waste and our time might be better spent supporting others who could sustain the effort required. One of the key lessons we draw from the meta-evidence project (and hindsight afforded to us over a year after the pandemic started) is the importance of early and meaningful stakeholder engagement when creating research priorities. Stakeholder engagement was simply something we did not consider, and possibly may have ignored due to limited time and resources and the increased pressure to provide information, fast. However, Cochrane's Question prioritization process and the Core Outcome Measures in Effectiveness Trials (COMET) initiative (9) demonstrates that engaging with multiple stakeholders including researchers, clinicians, patients, funders and policy makers is possible and will likely provide useful and meaningful research findings.

Conclusions
Combinations of evidence synthesis, information retrieval, and medical expertise allowed the team to carefully curate specific and useful RSS feeds which directly fed to an automated twitter bot. This was a good way of finding and presenting much needed evidence quickly and early in the pandemic at a time where rumours and conjecture were spreading throughout social media and causing panic. We were able to recognise (through the bot and early mapping exercise) that there was important and potentially life-saving research being produced from all corners of the world, particularly China, and we recognise the need to engage with researchers in China to allow us to effectively map and better represent the global evidence.
In conclusion, the meta-evidence project was extremely useful in the early stages of the project as a place to produce, in real time, the emerging global evidence, and also to visually present it. The decision to halt some parts of the project was a pragmatic one. A decision that allowed us to collaborate and support those groups who are still working tirelessly today. of this research.