Covid-19: Should doctors recommend treatments and vaccines when full data are not publicly available?
BMJ 2020; 370 doi: https://doi.org/10.1136/bmj.m3260 (Published 24 August 2020) Cite this as: BMJ 2020;370:m3260Read our latest coverage of the coronavirus outbreak
- Raymond M Johnson, associate professor of medicine (infectious diseases)1,
- Peter Doshi, associate professor2,
- David Healy, professor3
- 1Yale School of Medicine, New Haven, Connecticut, USA
- 2Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, Maryland, USA
- 3Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada
- Correspondence to: R M Johnson Raymond.Johnson{at}yale.edu, P Doshi pdoshi{at}rx.umaryland.edu
Yes—Raymond M Johnson
In normal circumstances, insisting on full data transparency and limiting decision making to published data alone is rightly paramount. But a pandemic is far from normal, and to insist on normal practice adds delay to interventions that could cost lives. A pandemic gives us little choice other than unpublished manuscripts (preprints) to guide therapeutic decision making. They should be used, thoughtfully.
Historians are likely to conclude that academic medicine was as unprepared for covid-19 as faltering government agencies such as the US Centers for Disease Control and Prevention, the Food and Drug Administration, and the Department of Health and Human Services. Worldwide, many hospital systems have surge capacity plans based on concerns about pandemic avian flu1—but few had strategies for rapidly evaluating therapeutics when SARS-CoV-2 arrived.
Of course, physicians would prefer to prescribe treatments and vaccines that have been thoroughly tested and scrutinized in peer reviewed, randomly controlled trials with full data transparency. But when push comes to shove this isn’t always possible, when we don’t know how much data collection is enough and we lack the understanding of a disease to know how to interpret the findings.
The usual “essential” of full data transparency before prescription should become a “nice to have” or an “as much as possible” in the urgent, fraught emergency circumstances we find ourselves in. We’re in a fast moving pandemic, and we should be ready to endorse products without full, exhaustive data, without peer reviewed publications, and without regulatory approval.
Preprint data and adaptive trials
Beyond surge capacity, our medical systems need prepositioned, randomized, adaptive cascading trials to evaluate treatments. Quality preprints can identify therapeutics, or inform study arms, during adaptive clinical trials. This applies especially to repurposed or off-label drug use where prescribing them on the basis of unpublished data has a lower threshold than adopting newer treatments or vaccines, because the treatment and its effects and side effects are to some extent known, and the infected patients who would receive them are acutely and specifically at risk for harm.
Hypothetically, back in March 2020, the first iteration of a covid-19 randomized trial could have included standard of care versus hydroxychloroquine (HQ)2 versus lopinavir/ritonavir,3 on the basis of published SARS-CoV-2 in vitro and MERS case-control data. In April, when a 150 subject randomized control trial preprint was released showing no HQ virologic or clinical benefit,4 hypothetical investigators could have closed the hydroxychloroquine arm and substituted steroids on the basis of the clinical experience in China.5 When lopinavir/ritonavir was subsequently found ineffective,6 the lopinavir/ritonavir arm could have closed and an alternative arm opened, such as convalescent plasma.
Unlike conventional clinical trials, adaptive cascading trials are not intended to answer a prespecified question to garner FDA approval and publication; they are intended to rapidly optimize medical treatment during a pandemic.
Adaptive clinical trials are not a new idea,7 but, formulated through existing health bureaucracies reliant on publication and peer review, they are likely ineffective. Public health agencies are the antithesis of nimble and adaptable. This is exemplified by the World Health Organization becoming bogged down in a pedantic debate about aerosolized particle size and masks six months into the covid-19 pandemic, as well as the imprecise summaries of primary clinical data from SARS-CoV-18 and MERS9 that led to recommendations from the National Institutes of Health, CDC, and WHO against steroids for SARS-CoV-2 (before release of the RECOVERY trial data).
Academic medical centers can contribute to pandemic responses by scrutinizing primary published data and preprints to design and perform adaptive clinical trials. Therapeutics can evolve during pandemics, or we can use what we have and hope to do better next time.
RECOVERY trial
We have one real life example already. The RECOVERY trial is an adaptive clinical trial that argues for using unpublished data to shape medical practice during a pandemic.10 The trial structure incorporates several critical features for research in such a situation.
First, it identified a limited number of treatment arms to allow definitive comparisons between them (standard of care; hydroxychloroquine (now ended); dexamethasone (also ended); lopinavir-ritonavir; azithromycin; convalescent plasma; and, in a second randomization for patients who deteriorate, the anti-inflammatory drug tocilizumab).
Second, it intentionally focused on clinical outcomes, rather than mechanistic investigations, to reach clinical efficacy endpoints. An independent data monitoring committee analyzes the interim data to identify benefit and harm early, to “adapt” the trial as it moves forward.
RECOVERY was designed to close and add treatment arms over time. Unpublished RECOVERY data were released as a press release (though this is not without its problems), and then as a preprint,11 showing a mortality benefit for steroid treatment in covid-19 patients developing hypoxemia.
Because RECOVERY is a high quality trial its unpublished data should be guiding treatment decisions now. Because pandemics are temporally and geographically dynamic, limiting decision making to full published data adds a delay between submission and publication that adversely affects the design of adaptive trials and, potentially, pandemic outcomes.
The critical part in using unpublished data in medical practice is content review and ensuring as much as possible that preprint data contain key information—the study protocol, summary data tables, and the statistical analysis used. Qualified parties should critically review preprints, pharmacodynamics, and toxicities to assess biologic plausibility and risk before incorporating therapeutics into adaptive trials or practice. Thus, unpublished trial results, even with limited data transparency and peer review, can deliver crucial interventions without sacrificing integrity.
No—Peter Doshi and David Healy
The trust we place in licensed medicines is a strong reason for insisting on full data transparency and reporting, even in the face of a pandemic. Few would disagree with the importance of data transparency, but even during normal times it remains a challenge—so, why demand it during a pandemic? The reason is that data transparency builds the foundation for information we can trust. Data secrecy, by contrast, creates risks too large to take.
The first critical risk is that of an exaggerated estimate of a product’s benefits when relying on scientific publications alone, not the underlying data. When the underlying clinical study reports for oseltamivir were finally made public they revealed that the data collection on lower respiratory tract complications relied on patients’ self-reporting, which makes sense for some outcome measures, such as pain, but not pneumonia. The result was a complete loss of confidence in the quality of data collected for the key performance assumption underpinning global stockpiling.12
The second critical risk is underestimating a product’s side effects. A year after novel vaccines were manufactured and rolled out on expedited timelines to tackle the threat of 2009 H1N1 swine flu, post-marketing reports of narcolepsy emerged in some Pandemrix vaccine recipients. But it would take a further seven years—and a lawsuit—to unearth internal pharmacovigilance reports by the manufacturer, which had suggested that problems with the vaccine’s safety had actually been produced in real time during the pandemic.13
Copious evidence already shows that adverse event data collected in trials are under-reported in journal publications.14 Moreover, serious adverse events may disappear if classified under rubrics such as “intercurrent illness” or “new medical histories,” which do not require serious adverse event reports—as has happened in vaccine and treatment trials.1516
Only publicly available full datasets will allow for a thorough assessment of side effects.
Placing our trust in data secrecy
But the benefits of transparency go beyond a truer understanding of product safety and efficacy: earning public trust, for a start. Jobbing doctors and patients alike reasonably expect any licensed covid-19 treatment or vaccine to work as advertised. This is about a chain of trust: only open data can allow other researchers with the ability to analyze it to do so, generating the trust that stems from knowing that judgments have been scrutinized and challenged.
Data transparency also creates the optimal environment for products—and there will be many covid-19 products, to be sure—to compete on the strength of their evidence base, not on the strength of promotion and buzz.
Finally, it must be recognized that there are no legitimate barriers to data transparency during the covid-19 pandemic. Companies can have little basis for claiming commercial confidentiality, as most products with any prospect of market entry have already been guaranteed massive profits through advance government purchases. There should also be no concern about patient privacy: guarantees to patients and trial participants regarding the privacy of their data should be honored, and such patient level data can and should be duly de-identified.
Nor should data release cost us valuable time. While it does take time to prepare data for sharing, the core work involves de-identification, and the trial specific methods can be determined in advance while trials are ongoing, for easy release when data collection is complete.
Before any covid-19 treatment or vaccine is made widely available, study protocols should be in the public domain, along with statistical analysis plans, clinical study reports, patient level data, and copies of the correspondence with regulators and other key stakeholders.
Data transparency is not a “nice to have.” Claims made without access to the data—whether appearing in peer reviewed publications or in preprints without peer review—are not scientific claims. Products can be marketed without access to the data, but doctors and professional societies should publicly state that, without complete data transparency, they will refuse to endorse covid-19 products as being based on science.
Footnotes
Competing interests: RMJ has no competing interests.
PD has received travel funds from the European Respiratory Society (2012) and Uppsala Monitoring Center (2018); he has received grants from the Laura and John Arnold Foundation (2017-21), American Association of Colleges of Pharmacy (2015), Patient-Centered Outcomes Research Institute (2014-16), Cochrane Methods Innovations Fund (2016-18), and UK National Institute for Health Research (2011-14); and he is an editor at The BMJ and an unpaid member of the Reagan-Udall Foundation for the FDA.
DH is a cofounder of RxISK.org, an adverse event reporting website. He has been one of a team involved in a notable data transparency exercise, the Restoration of Study 329.
DH has a further conflict: while a supporter of vaccines, he is not minded to take a covid-19 vaccine whose data are not fully available and will review the options for any legal challenge to force him to take a vaccine whose data are not fully available.
Provenance and peer review: Commissioned, not peer reviewed.
This article is made freely available for use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.
https://bmj.com/coronavirus/usage