Adherence of trials of operative intervention to the CONSORT statement

This paper by Gray et al highlighted the inadequacy in the reporting of non-pharmacological randomised controlled trials. This is consistent with previous work by our group,1,2 which has demonstrated the huge need for improvement with regard to the methodological reporting of surgical trials.


SURGICAL RESEARCH
Adherence of trials of operative intervention to the CONSORT statement extension for nonpharmacological treatments: a comparative before and after study The randomised controlled trial (RCT) represents the gold standard method for determining an association between treatment and outcome. 1 As important as the quality of the trial is the quality of its reporting; without transparent reporting, adequate appraisal of a trial's methodological quality and external validity is not possible. The Consolidated Standards of Reporting Trials (CONSORT) statement provides a minimum set of recommendations for the reporting of RCTs and is endorsed by many peer reviewed journals. 2 Its endorsement is widely recognised to have improved the reporting quality of RCTs. 3 However, since its introduction and update in 2001, 4 numerous articles have identified ongoing reporting quality deficiencies in both the medical 3,5-7 and surgical [8][9][10] literature.
Trials of operative intervention carry inherent methodological challenges 11 and so the CONSORT statement in this setting is not as useful as for trials of pharmacological intervention. Such challenges include accounting for variation in the recruiting and consenting practices of surgeons, difficulty in blinding patients and outcome assessors, the presence of confounding factors such as the surgeon's technical ability, differing anaesthetic technique and the need to standardise operative technique between surgeons who have different training backgrounds. As a result, the emphasis of reporting is different than for trials of pharmacological intervention. The reporting of certain methodological features gains increased importance while other features are entirely unique to surgical trials.

GRAy SULLIvAN ALTMAN GORDON-WEEKS
The CONSORT statement for non-pharmacological treatment (CONSORT-NPT) provides criteria specific for such features, enabling assessment of the reporting quality of trials of operative (and other non-pharmacological) intervention while maintaining important reporting elements of the original CONSORT statement. 12 Such items specific to the CONSORT-NPT checklist include the way in which interventions are standardised between surgeons, the assessment of a surgeon's adherence to such standardisation, the volume of participants treated per surgeon and the surgeon's experience with the intervention.
To date there has been no published analysis of the adherence of trials of operative intervention to the CONSORT-NPT statement or assessment of whether the CONSORT-NPT extension has led to an improvement in the reporting quality of such trials. This study analyses the adherence of trials of operative intervention to the CONSORT-NPT statement at time periods before and after its publication in order to determine whether the CONSORT-NPT extension has contributed to an improvement in the reporting standard of trials of operative intervention.

Search strategy
Electronic searches of MEDLINE ® and Embase™ were performed by a librarian at the John Radcliffe Hospital, Oxford. The following journals were searched: British Journal of Surgery, Archives of Surgery, European Journal of Cardio-Thoracic Surgery, European Urology and Journal of Bone and Joint Surgery (British volume). These publications were chosen to represent high impact factor journals providing information from a range of surgical specialties. The Annals of Surgery was excluded as a preliminary search revealed that it had published relatively few RCTs for the time periods studied. All included journals (with the exception of the European Journal of Cardio-Thoracic Surgery) have endorsed the CONSORT statement in their instructions to authors since 2007. No journal had mentioned the CONSORT-NPT statement as of February 2012 and the European Journal of Cardio-Thoracic Surgery had not mentioned the CONSORT or CONSORT-NPT statements.
The search was limited to two time periods: January to December 2004 (sufficiently long enough following the 2001 CONSORT statement revision 4 that trialists should be aware of its publication) and January to December 2010 (similarly with regard to the publication of the CONSORT-NPT statement in 2008). 12 The search history for an example journal can be seen in Figure 1. The search was repeated for each journal at each time point to ensure that all published RCTs within the search limits were retrieved. The NHS Evidence advanced search software (http://www.library.nhs. uk/) was used to perform electronic searches.

Inclusion/exclusion criteria
RCTs were included if they reported the comparison of at least one non-pharmacological intervention. This included trials of surgical technique, surgical access, technology and instrument design as well as anaesthetic interventions. RCTs comparing solely pharmacological interventions were excluded. All journals included in this study publish their manuscripts in English.
Studies were included in chronological order until a sample size of ten trials per journal per time point had been reached. This limit was set based on labour constraints rather than performing a sample size calculation, which, following personal communication with the study's lead statistician (DA), was felt to be unnecessary given that a defined hypothesis was not being tested.
Publications of retrospective, observational, experimental and animal studies were excluded as were trials reporting follow-up data of a previously published trial. Journals that endorse the CONSORT statement were defined as those that reference the statement in their instructions to authors or those journals that are referenced on the relevant CONSORT webpage (http://www.consort-statement.org/ about-consort/consort-endorsement/consort-endorsers--journals/). Journals were not excluded on the basis of their lack of endorsement of the CONSORT statement.
All abstracts retrieved from database searching were reviewed for selection by two authors (AGW and RG). Studies in which it was not clear whether the inclusion criteria had been met were reviewed in full text and discrepancies between the two authors were resolved by discussion with the remaining authors.

Data extraction
All publications were reviewed in full text by two authors (AGW and RG). From the CONSORT-NPT statement, a 30-point scoring system was devised (Table 1) giving equal weighting to CONSORT-NPT items and resulting in a score out of 30 for each trial. Two further items not included in the CONSORT-NPT statement (method of anaesthesia and sources of funding) were added as it was felt that these factors could be significant confounders in the design of trials of operative intervention such that their reporting was key. The reporting of the effect of clustering on sample size calculation and the estimate of the effect size and its precision were allocated their own point on the scoring system.
Reporting of intention to treat was not included as an item in the scoring system as previous studies have pro-  vided evidence to suggest that this method of analysis is frequently misreported by trialists. 13 Any addition or retraction of items from the original CONSORT-NPT checklist was done following discussion with all listed authors including a senior member of the CONSORT Group (DA). It has been noted that the CONSORT Group does not recommend the use of the checklist to provide a quality 'score'. 14 However, this approach gives a useful summary of overall reporting standard when comparing time periods to complement comparison of the reporting of specific items. The data extracted by each author were compared and any discrepancies resolved with discussion between the authors and re-review of the publication. Any remaining discrepancies were discussed with a third author (MS) until a consensus had been reached and a final score obtained. Assessors were not blinded to the time period or journal in which the RCT was published.

Outcome measures
The primary outcome measure was the difference in mean total modified CONSORT-NPT statement score between the 2004 and 2010 time periods. The secondary outcome measures were both percentage of adherence to each CON-SORT-NPT item and the difference in mean total modified CONSORT-NPT statement score between those journals endorsing the CONSORT-NPT statement and those not doing so.

Statistical analysis
Normally distributed means were compared using Student's t-test and non-normally distributed data with the Mann-Whitney U test. Normality was determined by visualisation of histograms and p-values of <0.05 were considered statistically significant.

Study flow, demographics and CONSORT-NPT endorsement
The study flow can be seen in Figure 2. A total of 191 RCTs were identified with 81 and 110 published in 2004 and 2010 respectively. Of these, 55 publications were excluded because the maximum number of RCTs had already been reviewed for specific journals within specific time points. The demographic details for publications at each time point were similar ( Table 2).

Comparison of mean CONSORT-NPT score
The mean CONSORT-NPT score was 15.2 (standard deviation [SD]: 3.8) for RCTs published in 2004 and 19.1 (SD: 4.1) for those published in 2010. The improvement in mean CONSORT-NPT score from 2004 to 2010 was 3.95 points (95% confidence interval: 3.61-4.29, p<0.001). There was considerable variation in the reporting of individual CON-SORT-NPT items with several items being underreported at both time points (Table 3). No single trial scored 100%.

Adherence to the CONSORT-NPT statement
Regarding methodological issues, for RCTs published in 2010 there was a significant increase in the reporting of sur-geon's/centre's eligibility criteria, sample size calculation, method for random sequence generation and allocation concealment (Table 3). For the results section, a significantly higher percentage of RCTs published in 2010 included a flow diagram or at least sufficient information to determine participant flow. There was also a significant improvement in the reporting of the number of participants treated per surgeon or centre, the study population's demographics and the experience of the surgeon or centre with the intervention technique. Finally, a significantly higher percentage of studies published in 2010 highlighted potential areas of bias.  Methodological items reported by <50% of RCTs in 2010 include the eligibility criteria on which surgeons or centres were selected (27%), the type of anaesthesia used (40%), how interventions were standardised between surgeons (43%), the methods used to monitor surgeons' adherence to the intervention or comparator techniques (8%), the effect that clustering has on the sample size calculation (3%) and the method of blinding participants or outcome assessors (48%). For the reporting of results, the number of participants treated by each surgeon or centre (43%), the surgeon's/centre's experience (38%) and the confidence of the effect estimate (35%) were all poorly reported (Fig 3).

CONSORT-NPT score and journal practice
The mean CONSORT-NPT score (both time points) for those studies published in CONSORT endorsing journals (mean: 17.5, SD: 4.5) was higher than that for the European Journal of Cardio-Thoracic Surgery (mean: 15.6, SD: 4.0). However, this did not reach statistical significance (p=0.064). When this was analysed per time period there was also no significant difference.

Discussion
Trials involving operative rather than pharmacological intervention bring inherent methodological challenges. Failure to overcome such challenges in the conduct of a trial is likely to lead to considerable bias, potentially invalidating the results and limiting their interpretation. Adherence to the CONSORT statement enables trial authors to maintain a transparent system of reporting so that the reader can draw considered conclusions from the trial findings. Previous studies have found that trials published in the surgical literature are lacking in their adherence to the CONSORT statement. [8][9][10] Balasubramanian et al found that trials published in high impact surgical journals reported only 69% of CON-SORT items. 9 Similarly, studies analysing the reporting quality of publications in spinal 15 and cardiothoracic 16 surgery found on average 65% and 66% of CONSORT items were reported respectively while the figure for urological trials was lower still at 52%. 8 All such studies acknowledge the difficulties in performing surgical trials. However, all determined CONSORT statement adherence rather than CONSORT-NPT adherence and included trials of pharmacological as well as operative intervention in their assessment. This makes them less specific for the analysis of operative trials and the reporting of methodological features, which make such trial design difficult. Furthermore, no previous study has analysed reporting adequacy in trials of operative intervention at two time points such that until this time any change in reporting standards could not be quantified.
It is recognised that in conducting this study the analysis is limited to four surgical specialties and that although data extraction was not performed in a blinded fashion, consensus was reached between multiple authors when scoring trials. The high quality of the search strategy and well defined inclusion/exclusion criteria enabled the analysis solely of trials of surgical intervention at two time points separated by the introduction of the CONSORT-NPT statement.
Equal weighting was given to all items in the CONSORT-NPT statement to create an overall score although some items may in fact assume greater importance than others. Nevertheless, presentation of the figures for the reporting of each item individually (Table 2) enables a clearer understanding of the items most frequently underreported. Finally, although the demographics of the included trials differed little between the two time periods studied (Table 2), it is recognised that improvements seen in the CONSORT-NPT score in the period studied could result from secular trends, in particular an improved awareness of an evidence-based approach in the surgical community or stricter ethical regulations imposed on trials rather than publication of the CONSORT-NPT statement alone.
Importantly, the significant improvement in reporting from 2004 to 2010 resulted from improved reporting of items such as sample size calculation, allocation concealment and participant flow (all items found in the original CONSORT statement). Comparison with prior estimates of reporting practice supports this improvement. For example, previous estimates for the number of trials in the surgical literature reporting sample size calculations were 20% (2003) 16 and 44% (2008) 15 compared with 80% (2010) here. Similarly, the percentage of trials reporting study flow was estimated at 51% (2000-2003) 8 and 52% (2008), 15 and documented as 80% (2010) here. This trend is repeated for reporting of participant blinding, random sequence generation and allocation concealment among other items.

Conclusions
Although CONSORT items improved significantly, there was little improvement in CONSORT-NPT specific items, all of which were reported in less than 50% of trials in 2010 (Fig 3). While journals give specific instructions to authors regarding the use of the CONSORT statement, no mention of the CONSORT-NPT statement was found in the journals included in this study. Peer reviewed journals' instructions to authors are likely to have played a large part in improving the awareness of reporting standards throughout surgical academia by insisting on adherence to the CONSORT guidance. The evidence presented here strongly suggests that journals publishing trials of operative intervention should pay equal attention to the CONSORT-NPT statement with the aim being to improve both authors' and reviewers' awareness of this CONSORT extension. This will, in turn, help to improve the quality of reporting of methodological issues specific to such trials, enabling clearer interpretation of their outcomes.

Acknowledgement
Many thanks to Tatjana Petrinic, who created the search strategy.