Protocol for a systematic review on the extent of non-publication of research studies and associated study characteristics

Background Methodological research has found that non-published studies often have different results than those that are published, a phenomenon known as publication bias. When results are not published, or are published selectively based on the direction or the strength of the findings, healthcare professionals and consumers of healthcare cannot base their decision-making on the full body of current evidence. Methods As part of the OPEN project (http://www.open-project.eu) we will conduct a systematic review with the following objectives: 1. To determine the proportion and/or rate of non-publication of studies by systematically reviewing methodological research projects that followed up a cohort of studies that a. received research ethics committee (REC) approval, b. were registered in trial registries, or c. were presented as abstracts at conferences. 2. To assess the association of study characteristics (for example, direction and/or strength of findings) with likelihood of full publication. To identify reports of relevant methodological research projects we will conduct electronic database searches, check reference lists, and contact experts. Published and unpublished projects will be included. The inclusion criteria are as follows: a. RECs: methodological research projects that examined the subsequent proportion and/or rate of publication of studies that received approval from RECs; b. Trial registries: methodological research projects that examine the subsequent proportion and/or rate of publication of studies registered in trial registries; c. Conference abstracts: methodological research projects that examine the subsequent proportion and/or rate of full publication of studies which were initially presented at conferences as abstracts. Primary outcomes: Proportion/rate of published studies; time to full publication (mean/median; cumulative publication rate by time). Secondary outcomes: Association of study characteristics with full publication. The different questions (a, b, and c) will be investigated separately. Data synthesis will involve a combination of descriptive and statistical summaries of the included methodological research projects. Discussion Results are expected to be publicly available in mid 2013.

1. To determine the proportion and/or rate of non-publication of studies by systematically reviewing methodological research projects that followed up a cohort of studies that a. received research ethics committee (REC) approval, b. were registered in trial registries, or c. were presented as abstracts at conferences.

2.
To assess the association of study characteristics (for example, direction and/or strength of findings) with likelihood of full publication.
To identify reports of relevant methodological research projects we will conduct electronic database searches, check reference lists, and contact experts. Published and unpublished projects will be included. The inclusion criteria are as follows: a. RECs: methodological research projects that examined the subsequent proportion and/or rate of publication of studies that received approval from RECs; b. Trial registries: methodological research projects that examine the subsequent proportion and/or rate of publication of studies registered in trial registries; c. Conference abstracts: methodological research projects that examine the subsequent proportion and/or rate of full publication of studies which were initially presented at conferences as abstracts.
(Continued on next page) (Continued from previous page) Primary outcomes: Proportion/rate of published studies; time to full publication (mean/median; cumulative publication rate by time).
Secondary outcomes: Association of study characteristics with full publication. The different questions (a, b, and c) will be investigated separately. Data synthesis will involve a combination of descriptive and statistical summaries of the included methodological research projects. Discussion: Results are expected to be publicly available in mid 2013.
Keywords: Publication bias, Selective reporting, Abstract(s), Research ethics committees, Trial registration, The OPEN Project Background Full information about completed and ongoing clinical studies is the indispensable base for decision-making about medical therapies, treatments, and diagnostic procedures by patients, doctors, and policy-makers. Equally, researchers and research organizations, research ethics committees, governments and health system agencies, courts for social justice, pharmaceutical companies, and all professional groups of the healthcare system are dependent on unbiased information. Methodological research projects have shown that for approximately 40% to 50% of all launched studies, results or reasons for their failure are never published [1][2][3][4]. Evidence shows that these unpublished studies do not significantly vary in methodological quality from their published counterparts [5,6]. The lack of publication of completed studies was termed 'the file drawer problem' by Rosenthal in 1979 [7].
The frequency of unpublished studies is particularly problematic because several methodological research projects have found that non-published studies often have different results than those that are published, a phenomenon known as publication bias [8]. It has also been found that studies with non-novel or negative findings take longer to reach publication than studies with novel findings or findings in favor of the study arm [9,10]. When results are not published, or are published selectively based on the direction or the strength of the findings, healthcare professionals and consumers of healthcare cannot base their decision-making on the full body of current evidence.
The inability to make evidence-informed decisions impacts the healthcare system at various practical and financial levels. At the individual level, the overestimation of treatment effects caused by publication bias may result in patients receiving treatments that may be more harmful or less efficacious than previously believed. Furthermore, non-publication of studies results in considerable financial investment by funders without any return. Further costs include those incurred by hospitals and medical centers, insurance companies, and individual patients whom all continue to pay for treatments that may not be the most effective or efficient. Although the full extent of financial impact of non-publication of studies is currently unknown, the waste of funds is likely to be high.
In response to these concerns the OPEN Project (To Overcome failure to Publish nEgative fiNdings) was developed with the goal of elucidating the scope of nonpublication of studies. The OPEN Project is a 24-month project co-funded by the European Commission under the Seventh Framework Programme. With an international work group composed of key opinion leaders (for detailed information see Appendix), the project will first examine the current evidence on publication bias through a series of systematic reviews such as the one described in this protocol. Second, the OPEN Project will examine current practices by key groups in the field of biomedical research (namely funding agencies, the (pharmaceutical) industry, research ethics committees, research institutions, researchers, trial registries, biomedical journals, regulatory agencies, and benefit assessment agencies), through surveys, interviews, and analysis of current policies and guidelines. These findings will be presented and discussed during an international workshop, which aims to develop recommendations for implementing effective measures to avoid non-publication of studies and related publication bias at all levels.
Non-publication impacts the influence of the results of specific studies as well as of any systematic review in which the individual study could be included. The inability of systematic reviews to include all current evidence may result in misleading conclusions [11]. Previous reviews have been conducted on different aspects of non-publication of studies and publication bias [12][13][14]. However, some of these reviews were done several years ago, while others are quite specific in scope, focusing only on specific aspects of non-publication of studies and/or publication bias.
Non-publication of studies and publication bias can be revealed through a variety of sources. Accordingly, this review seeks to summarize the current knowledge on non-publication of studies and publication bias by examining cohorts of studies approved by research ethics committees, by examining cohorts of studies registered with trial registries, and finally by examining cohorts of studies that have been initially presented in abstract form at conferences.

Objectives
This review has two objectives: 1) To determine the proportion and/or rate of nonpublication of studies by systematically reviewing methodological research projects that followed-up a cohort of studies that were: a. approved by research ethics committees, or b. registered in trial registries, or c. presented in abstract form at conferences.
2) To assess the association of study characteristics (for example, direction and/or strength of findings) with likelihood of full publication.

Search methods for identification of methodological research projects
To identify the relevant research evidence we will conduct electronic literature searches in the following databases: No language restrictions will be applied. The search strategy was designed with consideration of similar reviews' strategies and with the support of a librarian/information specialist. The strategy was adapted according to the requirements of each database (for the full search strategy, please see Appendix). We will also search reference lists of articles retrieved from our database searches to further help us to identify relevant articles, authors, and opinion leaders who might not have been captured in our original search or known to us previously. These individuals will be contacted and asked for further relevant articles.

Data collection and analysis Selection of methodological research projects
Reports of methodological research projects including a cohort of studies will be selected using a specifically designed full text screening form with predefined inclusion and exclusion criteria (see below). The criteria will be deliberately broad in order to capture as large a scope of biases related to non-publication of studies as possible.
Published and unpublished methodological research projects on the fate of a cohort of studies will be included. The inclusion criteria for the individual aspects are as follows: (1) Methodological research projects of studies approved by research ethics committees (RECs): methodological research projects that examine the subsequent proportion and/or rate of publication of studies that received approval from ethics committees; reports of methodological research projects must provide the number of studies approved and the proportion or rate of approved studies which have been published. presented as abstracts at a conference or as summary reports: methodological research projects that examine the subsequent proportion and/or rate of full publication of study results which were presented at conferences as abstract or as summary report; reports of methodological research projects must provide the number of abstracts (summary reports) identified and the proportion or rate of abstracts (summary reports) published as full articles.
The included methodological research projects do not need to apply a minimum follow-up to assess time to full publication; however, to calculate the proportion of published studies and/or publication rate, a minimum follow-up of 24 months will be required. Two reviewers will screen the titles and abstracts of all references identified by the literature search. Full text articles will be evaluated for all papers identified as potentially relevant.

Data extraction and management
A specifically designed data extraction form is being developed and pilot-tested for each aspect (RECs, trial registries, abstracts). Two reviewers will independently extract all relevant data from eligible studies. Data will be recorded in an Excel spread sheet.

Assessment of study quality in included methodological research projects
We will systematically consider the validity and generalizability of the identified evidence provided by each of the methodological research projects by evaluating the following domains: Sampling: Choice of sampling frame (for example, all abstracts submitted, all accepted as oral presentations, and so on) and sampling method (for example, all abstracts, random sample of abstracts, and so on). Literature search: Timing of search (for example, how many months after abstracts were presented at a conference was the literature search conducted) and methodology used for identifying full publications (for example, two independent researchers, direct contact with investigators, number and choice of databases, database search for first and/or last author, and so on). Matching of publications: Methodology used to match retrieved references with studies included in cohort of studies (for example, matching by keywords, title, author names and so on, matching completed by one or more researchers, adjudication if differences, and so on). Analysis: Adjustment for confounding factors if analysis for factors associated with full publication was done.
Methodological research projects will be categorized as low study quality if the search for publications was likely to miss a substantial number of full publications (for example, if it were only conducted in a single database or if choice of databases is judged to be not appropriate by two reviewers independently). A similar approach will be taken for the other three domains.

Outcome measures
Our primary outcomes will be the proportion and/or rate of studies published and the time to publication. When possible, pooled proportions of unpublished studies will be calculated. For outcomes pooled, we will explore heterogeneity by standard measures.
The following definitions will be used: Study completion is defined as the last day of follow-up of participants; Time to publication is defined as the interval between study completion and publication. If the end of follow-up is not known (for example for studies presented as abstracts) time to publication will be calculated based on the interval between abstract presentation and subsequent full.
Secondary outcomes include association of study characteristics with full publication. We will evaluate factors that could be associated with the publication of studies, such as direction and/or strength of findings. We planif reported or estimable -to collate information on costs/resource use incurred by studies that were not published.

Unit of analysis issues
The unit of analysis is the methodological research project. At the time of writing this protocol, we do not anticipate any unit of analysis issues.

Dealing with missing data
In the instance of missing data, lead researchers of the methodological research projects will be contacted for further information.

Assessment of heterogeneity
Heterogeneity for pooled outcome measures will be assessed by standard methods including Chi 2 -test and calculation of the I 2 value.

Assessment of reporting biases
Funnel plots will be used to assess the association between point estimates of log odds ratio (a measure of extent of association between study characteristic and likelihood of publication) and a measure of precision if more than 10 methodological research projects provide necessary information. Funnel plots will be visually assessed and appropriate formal statistical tests following recommendations formulated by Sterne et al. will be used to test for asymmetry [15].

Data synthesis
The three aspects (research ethics committees, trial registries, and conference abstracts) will be considered separately. Data synthesis will involve a combination of descriptive and statistical summaries of the findings of the included methodological research projects. We will calculate the pooled proportion and/or rate of publication with 95% CI using a random effects model for all included methodological studies by averaging the individually reported publication rates after weighting by the square root of the total number of studies included in each methodological research project. If possible, time to publication will be estimated by survival analyses and depicted in Kaplan Meier curves. If required data are not available, we will simply present proportions at the following time points: 12, 24, 36, 48, 60 months, and so on.
Strength of association of study characteristics with full publication will be expressed as risk ratios with 95% confidence intervals. We will also explore the possibility of meta-regression analysis to examine the impact of factors specified for subgroup analyses on likelihood of future publication.
Subgroup analysis and investigation of potential risk factors for non-publication Subgroup analyses will be conducted for the following factors depending on availability of data in the included methodological research projects: Acceptance for presentation at a scientific conference (abstract aspect only).
We will use definitions applied in included methodological research projects for categorization of, for example, clinically relevant positive/negative results. Further subgroup analyses will be considered if important subgroups of study characteristics arise during the conduct of this systematic review. These exploratory analyses will be clearly labeled as post-hoc analyses.

Sensitivity analysis
Sensitivity analyses are planned with regard to the four domains used to assess the quality of the methodological research projects (that is, consider only methodological research projects of low study quality) and whether research projects themselves were published as full publications only.

Discussion
This review seeks to synthesize the growing body of research that is related to non-publication of studies. By examining multiple aspects of biases related to nonpublication of studies in one review, we hope to provide an up-to-date, comprehensive picture of this important problem. The findings, including risk factors for nonpublication, will serve to raise awareness about the extent of non-publication and the complexity of this issue. In conjunction with results from other work packages of the OPEN project, this review will serve as a foundation for a recommendations workshop. This workshop will enable key members of the biomedical research community (for example, funders, research ethics committees, journal editors) to develop future policies and guidelines to lessen the frequency of non-publication and related biases.