• epidemiology
• information management

This is the second in a series of notes addressing the need to change our evidence sources for assessing the effects of pharmaceuticals and biologics because of the danger of unrecognised reporting bias. In this note I shall discuss the possibility of using a shorter way of redefining the E than using clinical study reports.

In the first note—‘Redefining the ‘E’ in EBM’— I made the point that trial publications (if they exist) are the tip of the data iceberg.1 The ‘iceberg’ is made up of a complex and massive reality of thousands of pages of clinical study reports, manuals, forms and slides of meetings invisible below the waterline. The submerged part is likely to represent anything between 1 and 8000 pages of published material.2 Even more importantly, the submerged material has the potential to change the conclusions of systematic reviews. To prove this point we have presented a list of references in our first editorial1 and at the Restoring Invisible and Abandoned Trials (RIAT) (https://restoringtrials.org/whatisriat/) initiative.

I have another reason to be wary of the published written word, especially in prestigious journals. The reason is not new, but is not widely known. It is the refusal of some authors and editors to correct misreporting in their trials, even when presented with the evidence of misreporting. BMJ-EBM recently published the evidence of misreporting of an active ingredient as an ‘inert placebo’ in pivotal trials of human papillomavirus vaccines.3 Long before we embarked in the restoration, following RIAT rules, we contacted sponsors and authors of the misreported articles publicly inviting them to take action.4 We got no response. Once the restoration was complete we sent our manuscript to the same journal who had presided over the publication of three of the misreported trials. No action was taken. We encountered the same problem with the journals which published ghost-written pivotal trials of the antiviral Tamiflu.5

In our initial note,1 we suggested addressing reporting bias by including regulatory data in our reviews and briefly mentioned “different types of regulatory data now on offer”,1 such as regulators’ reports.

We first came across regulators’ reports in the preparation of our Cochrane review on neuraminidase inhibitors, but we were not the first to use them.6 Because clinical study reports had not yet been released to us,6 we accessed 2000 pages of letters, medical and statistical officer reports and minutes of meetings preparatory to the granting of the licence by the US Food and Drugs Administration (FDA). The documents (which are cited by file number and content in our review) provided a unique insight into the workings of regulators and their reviews of what we could not (yet) see: the thousands of pages of clinical study reports on both neuraminidase inhibitors, Relenza and Tamiflu.6 The FDA documents were extremely detailed and impressive king-sized peer-review reports. Unlike journal peer-review reports, they addressed several trials in the same development programme, and provided tables listing all the trials in the programme of interest to the regulator. The listing sometimes eliminated the problem of identification and (at times) publication bias altogether. These reports (all with different names) are now available from several sources and easily downloadable. Some are very difficult to follow but others are eminently readable and authoritative.7 Over time, FDA reports have become more readable, although there is a worry that recent proposal to conflate various reports might affect detailing.8

Table 1 summarises the main regulators’ reports on offer and their sources.

The average length and complexity of these reports is way below that of an evidence development programme of clinical study reports, typically being in the low hundreds of pages. However, unlike most clinical study reports, regulator’s comments reports are easily accessible without registration and can be very useful as a halfway house between publications and clinical study reports. Their biggest drawback is that they represent a third party’s view of what you cannot see: the clinical study reports.

Some National Competent Authorities’ websites provide further information such as assessment history, labels patient information leaflets, postmarketing activities which could be invaluable in assessing the long-term effects of an intervention (table 1).

Regulators’ reports should be routinely introduced as a source of data and information in evidence synthesis of pharmaceuticals when time is short and clinical study reports are not available. The halfway house represents a practical refining of the Evidence in EBM in the current scenario of universal reliance on inscrutable dubious publications and unaccountable authors and editors.