Background

The randomised controlled trial (RCT) is the most robust design to assess the efficacy and effectiveness of treatments.1 As a result of this realisation, clinical decision-making in recent years has been directed away from reliance based solely on the doctor's clinical experience towards a paradigm based on evidence derived from RCTs. The results of large RCTs have subsequently been translated into guidelines containing evidence-graded recommendations which clinicians are encouraged to use as the basis of good clinical practice.2 If, however, the 'raw material' is flawed, the conclusions cannot be trusted, hence the need to appraise critically the quality of the underpinning trial evidence.3

Quality is a multidimensional concept which relates to the design, conduct, and analysis of a trial, its clinical relevance, and its reporting.3 In most cases, the RCT report is the only source for clinicians, guideline developers, and other researchers to judge the validity and generalisability of the results, so the quality of reporting of trials is of inherent interest.4 It is then of considerable concern that the quality of reporting of RCTs is often sub-optimal.5

In response to these concerns about the quality of reporting of RCTs, in the mid-1990s an international group developed the Consolidated Standards of Reporting Trials (CONSORT) Statement.6 This was first published in 1996 and then updated in 2001.5 After an expert meeting in January 2007, this was further revised resulting in the latest iteration — i.e. the CONSORT 2010 Statement.5

The current CONSORT Statement (hereafter referred to as CONSORT) comprises a checklist of essential items that should be included in reports of RCTs and a diagram for documenting the flow of participants through a trial. It is formulated primarily for use with reports of two-arm parallel-group RCTs. Many of the CONSORT data fields are, however, also relevant to a wider spectrum of trial designs such as non-inferiority, equivalence, factorial, cluster, and crossover trials.5 Extensions to the CONSORT checklist for reporting trials with some of these other designs have been published,7–9 as have those for reporting particular types of data (i.e. harms10), types of interventions (i.e. non-pharmacological treatments11 and herbal interventions12), and abstracts.13 In this review we will, as appropriate, use the non-pharmacological, non-inferiority and equivalence, cluster and pragmatic extensions of CONSORT.

CONSORT criteria have been used to assess the reporting quality of RCTs in several disease areas4,14,15 and journal types.16,17 There have, however, been no recent assessments of the quality of RCTs reporting in the asthma literature. The only previous study on clinical trials of asthma treatments was undertaken for the period 1984–1997 and was published in 2002 in two reports.18,19 This initially involved a comparison between RCTs published in Spanish and English language journals,19 and this was then followed by a secondary analysis of a subsection of the same dataset focusing solely on the quality of RCTs in English.18 The first article showed poorer reporting quality of the RCTs in Spanish publications and a strong association between the type of journal, type of intervention, and the comparison measure used and reporting quality. Moreover, this study highlighted the necessity for better reporting in general in the asthma literature, leading the authors to advocate the more widespread use of a checklist by authors and editors in order to improve reporting standards.19

Building on this earlier work, we will examine the quality of reporting of asthma clinical RCTs in the contemporary asthma literature. Our secondary aim is to investigate if there is an association between specific trial characteristics that have previously been identified in the literature in influencing reporting quality and the actual quality of the trial reports.4,14,17,20–22

Objectives

The primary objective is to assess the contemporary quality of reporting of RCTs in the asthma literature for the period 2010–2012.

The secondary objectives are to identify factors associated with better reporting quality, that is:

  • Are trials published in general medicine journals associated with better quality than those published in specialist journals?

  • Is a high impact factor of the journal of publication associated with studies of better quality than those published in lower impact journals?

  • Are studies conducted or led by teams in high income country settings (defined using World Bank Group definitions)23 associated with better quality than those in middle and low income country settings?

  • Does the funding source have an impact on study quality?

  • Are trials evaluating a pharmacological intervention associated with better quality than those evaluating a non-pharmacological intervention?

  • Are studies with multiple participating centres associated with better quality than single-centre studies?

Review methods

Search strategy

We will search the electronic database MEDLINE (via Ovid) using the search terms of the Cochrane Airways Group Specialised Register for asthma and RCTs for the time period January 2010 to July 2012. We will include studies that have been published in the top 10 impact factor journals in general medicine and respiratory specialty journals using the most recent available (i.e. 2011) rankings,24 as long as they publish clinical trials and include articles related to asthma. Our complete search strategy is presented in Appendix 1.

Inclusion criteria

  • Types of studies: RCTs with parallel or cluster study design that involve only human subjects

  • Types of participants: All study populations with asthma as the only condition being examined

  • Types of interventions: Pharmacological and non-pharmacological interventions evaluating the clinical effectiveness of a treatment with any conceptual framework (superiority, non-inferiority, equivalence). We consider that a trial is evaluating the effectiveness of a treatment as long as it has at least one clinical outcome (primary or secondary).

Exclusion criteria

  • Reviews, systematic reviews, and meta-analyses

  • Non-randomised trial designs (quasi-experimental, observational studies)

  • Studies with crossover and factorial design, n-of-1 trials, split body trials

  • Studies evaluating diagnostic tests, prevention, prognosis, cost-effectiveness, pathophysiological mechanisms, pharma-cokinetics, pharmacogenetics, validation of questionnaires, tolerability of drugs, and economic studies

  • Trials not reported as full papers (abstracts), editorials, comments, letters, case reports, audits, guidelines, historical articles

  • Methodological, epidemiological and qualitative studies

  • Study protocols

  • Pilot studies and phase I, II, and IV trials

  • Secondary analysis of trials

  • Studies reporting updates of previously published RCTs.

Review strategy

Searches will be undertaken independently by two reviewers (CN and PB) with support from AW and AS. The references will be imported into EndNote and duplicates will be deleted. Both reviewers will independently review the titles for potentially eligible studies. They will not be blinded to study details. If they are unsure or there are disagreements they will read the abstract also. Full text copies of potentially relevant studies will be obtained and CN and PB will assess their eligibility for inclusion against the criteria mentioned above. A kappa statistic will be calculated to measure the level of agreement.25 Where the reviewers agree, they will either include or exclude the study as appropriate. Disagreements will be resolved through discussion with AW or AS as arbiters. The studies that will be excluded after reading the full paper ('near-misses') will be reported in a table with reasons for exclusion. The whole process will be documented on a PRISMA flow chart.26

Data extraction and quality assessment strategy

Data will be extracted independently by two reviewers (CN and PB) from the selected studies using an appropriate electronic customised data extraction form (see Appendices 2,3,4). The reviewers will not be masked to study details. There will be pilot testing of the data extraction sheet, disagreements will be discussed, and modifications will be made if required. In case of multiple reports of the same study, we will extract data directly into a single data extraction form. Disagreements will be resolved through discussion with AS as arbiter. We will extract data on general characteristics of the trials (see Appendix 2) and use a modified 38-item CONSORT-based checklist (see Appendix 3) that consists of all the CONSORT checklist items plus one additional item from the non-pharmacological treatments extension. The assessment of the adequacy of reporting will be done according to the CONSORT 2010 guidelines and its extensions.5,8,9,11,12 Each item can be characterised as ‘yes’ if it is clearly and adequately reported, or ‘no’ if it is partially unclear or not reported at all. If an item is not applicable to a specific study we will characterise it as ‘N/A’. Each ‘yes’ answer will receive a score of 1 and each “no” answer will be scored as 0. The overall quality scoring of the trial will be calculated as a proportion of the ‘yes’ rated applicable items (possible range 0–38 points). In addition, we will score the overall quality of reporting using key parameters of internal validity summarised in the Cochrane Risk of Bias tool (see Appendix 4) and we will categorise the studies into those at (1) low risk of bias and (2) moderate/high risk of bias.25

The following data will be extracted:

General characteristics

  • Journal name

  • Journal type (general medicine or specialty)

  • Journal impact factor

  • Country of study (high-income, middle-income, low-income)

  • Funding source (solely industry, part industry, non-industry, none, unknown)

  • Trial design (parallel or cluster)

  • Conceptual framework (superiority, non-inferiority, equivalence)

  • Type of intervention (drug or non-pharmacological)

  • Number of participating centres (multiple or single centre).

Analysis and data synthesis

We will calculate the proportion of the trials that have clearly and adequately reported each CONSORT item with a 95% confidence interval (CI). An overall quality score will also be calculated for each trial as a percentage of all the adequately reported applicable items with a 95% CI, which will be used to inform a global assessment of the quality of reporting. The general characteristics data will be presented as numbers and percentages with 95% CI when categorical and as mean and SD or median and IQR with 95% CI when continuous. SPSS software will be used to identify the variables associated with 'low risk of bias' studies with Fisher's exact test, and overall quality scores for subgroups with different trial characteristics will be compared with appropriate two-sample methods (rank-based or Normality-based, depending on the distributional characteristics of the overall quality score). We will report on the quality of reporting of asthma trials and make recommendations for researchers and journal editors regarding the conduct, reporting, and publication of asthma trials. In our description of the studies we will make reference to the setting and population in which the study was undertaken. In concluding, we will consider the quality and relevance of the body of work for informing clinical decision-making.