Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? *

BACKGROUND
Since 2005, International Committee of Medical Journal Editors (ICMJE) member journals have required that clinical trials be registered in publicly available trials registers before they are considered for publication.


OBJECTIVES
The research explores whether it is adequate, when searching to inform systematic reviews, to search for relevant clinical trials using only public trials registers and to identify the optimal search approaches in trials registers.


METHODS
A search was conducted in ClinicalTrials.gov and the International Clinical Trials Registry Platform (ICTRP) for research studies that had been included in eight systematic reviews. Four search approaches (highly sensitive, sensitive, precise, and highly precise) were performed using the basic and advanced interfaces in both resources.


RESULTS
On average, 84% of studies were not listed in either resource. The largest number of included studies was retrieved in ClinicalTrials.gov and ICTRP when a sensitive search approach was used in the basic interface. The use of the advanced interface maintained or improved sensitivity in 16 of 19 strategies for Clinicaltrials.gov and 8 of 18 for ICTRP. No single search approach was sensitive enough to identify all studies included in the 6 reviews.


CONCLUSIONS
Trials registers cannot yet be relied upon as the sole means to locate trials for systematic reviews. Trials registers lag behind the major bibliographic databases in terms of their search interfaces.


IMPLICATIONS
For systematic reviews, trials registers and major bibliographic databases should be searched. Trials registers should be searched using sensitive approaches, and both the registers consulted in this study should be searched.

Background: Since 2005, International Committee of Medical Journal Editors (ICMJE) member journals have required that clinical trials be registered in publicly available trials registers before they are considered for publication.
Objectives: The research explores whether it is adequate, when searching to inform systematic reviews, to search for relevant clinical trials using only public trials registers and to identify the optimal search approaches in trials registers.
Methods: A search was conducted in ClinicalTrials.gov and the International Clinical Trials Registry Platform (ICTRP) for research studies that had been included in eight systematic reviews. Four search approaches (highly sensitive, sensitive, precise, and highly precise) were performed using the basic and advanced interfaces in both resources.
Results: On average, 84% of studies were not listed in either resource. The largest number of included studies was retrieved in ClinicalTrials.gov and ICTRP when a sensitive search approach was used in the basic interface. The use of the advanced interface maintained or improved sensitivity in 16 of 19 strategies for Clinicaltrials.gov and 8 of 18 for ICTRP. No single search approach was sensitive enough to identify all studies included in the 6 reviews.
Conclusions: Trials registers cannot yet be relied upon as the sole means to locate trials for systematic reviews. Trials registers lag behind the major bibliographic databases in terms of their search interfaces.
Implications: For systematic reviews, trials registers and major bibliographic databases should be searched. Trials registers should be searched using sensitive approaches, and both the registers consulted in this study should be searched.
Clinical trials registers such as ClinicalTrials.gov and portals to trials registers such as the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) are increasingly used to identify ongoing or completed clinical trials. These resources offer important information on the methods and progress of trials likely to be of interest to a range of users, including researchers, clinicians, and patients. The extent to which these resources can be relied upon as a sole source of trials for inclusion in systematic reviews, including Cochrane systematic reviews (CSRs), is the subject of the research study reported here. This study also investigates the most efficient ways that librarians, information professionals, and other searchers can search these resources. Search efficiency was investigated by evaluating the overlap and unique yield of searches in the two resources and by testing four search approaches. The tested search approaches ranged from the very precise (single specific condition search term combined with a single specific intervention search term) to the very sensitive (at least two interventions terms).

BACKGROUND
Since July 2005, all International Committee of Medical Journal Editors (ICMJE) member journals have required that clinical trials must be registered in publicly available trials registers before they are considered for publication [1]. Such registration is designed to ensure that trial details are available to all: health care professionals, patients, librarians, program managers, and researchers, including systematic reviewers. It also means that information about trials is available, irrespective of the results and in the event of non-publication of results. Even if trials are not published in journals, increasingly their results may be published in the registers (as is the case with ClinicalTrials.gov) or in results registers [2]. The increased availability of information about trials may encourage both improvements in trial methods and a reduction in publication biases through knowledge of the existence of a trial and potentially the wider availability of trial results [3]. The introduction of the ICMJE requirement and other initiatives is likely to have contributed to an increasing number of trials being registered in resources such as ClinicalTrials.gov [4,5] and in the registers accessible via the WHO ICTRP [6]. The extent to which trials registers can now be relied upon as a sole source of clinical trials and trials results for research, including CSRs, is now being discussed. If all trials are recorded in registers, then the need to search other resources and databases such as MEDLINE may be reduced, with resulting economies for the systematic review process. However, as trials registers proliferate and the searching of trials registers becomes more widespread, especially in the context of systematic reviews [7], questions also arise about which registers need to be searched and whether ClinicalTrials.gov needs to be searched at all, since it is included in ICTRP. ClinicalTrials.gov and ICTRP offer different search engines and search options. The most efficient search approaches, in terms of finding as many of the available relevant trials and the fewest irrelevant trials for the fewest searches, remain to be established.
These questions are particularly important for searchers in the Cochrane Collaboration. Searching trials registers, including ClinicalTrials.gov and ICTRP, is now mandatory for Cochrane Collaboration members [8]. These questions are also important for librarians and information professionals who provide support to researchers, clinicians, and patients looking for trials in a range of information-seeking scenarios.
The study reported here investigated interrelated questions to increase understanding of the current adequacy of clinical trials registers for conducting systematic reviews, as well as to identify the optimal approach to searching the two largest trials resources (ClinicalTrials.gov and ICTRP) for clinical trials: & To locate trials for inclusion in systematic reviews and health technology assessments, is it adequate to search only public trials registers, or should major bibliographic databases, such as MEDLINE, be searched as well? & In the case of ClinicalTrials.gov and ICTRP, what are the optimal search approaches and is it necessary to search both collections to maximize trial identification?

Original search strategy
ClinicalTrials.gov and the ICTRP were initially searched for the trials included in two recently updated systematic reviews from the Cochrane Injuries Group [9,10]. Following this pilot investigation, six additional CSRs [11][12][13][14][15][16] were identified by searching for ''New'' reviews in the 2012 issue 1 version of the Cochrane Database of Systematic Reviews (CDSR), and their included trials were extracted as well. To identify the trials in the registers, their bibliographic details were collated into an Excel spreadsheet. Additional identifying information (trial number/unique identifier, trial name, lead investigator, location, and sponsor) was located by searching PubMed and Google. PubMed provides trial numbers in the PubMed Secondary Source ID (SI) field.
Having retrieved this information, ClinicalTrials. gov and ICTRP were then searched to identify the included trials, using the following approaches: To achieve geographical focus, the ''Locations'' field was used in ClinicalTrials.gov and the ''Countries of recruitment'' option was used in ICTRP. If the included study was carried out in the United States and the search results were large, the ''State'' location limit in ClinicalTrials.gov was employed.

Trials register searches
Once registered trials had been identified, the search strategies (search terms and combinations of terms) reported in the reviews were tested to see whether they would actually have identified these trials in the two registers. The original MEDLINE search{ strategy reported in each of the CSRs was adapted for use in searching the two trials registers. In addition, four types of searches were executed for each CSR: & a very precise strategy combining a single specific condition search term with a single specific intervention search term using the Boolean ''AND'' & a precise strategy using just one term for the named intervention or a single combination of terms such as ''sterile water'' & a sensitive strategy using a two concept approach with at least two condition search terms ''AND'' at least two intervention search terms (specific and generic) & a very sensitive strategy using only one concept captured by at least two intervention search terms These strategies were developed based on the assumption that phrase searches would tend to be more precise than searches using the Boolean ''AND,'' and that searches using ''AND'' would tend to be more sensitive than searches using phrase searching. An additional assumption was that searching using two concepts combined with ''AND'' would usually be more precise (fewer irrelevant records to assess) and less sensitive (finding fewer relevant records) than searching with only one concept. An example of an adapted sensitive strategy for one review [13] for the two interfaces in the two resources is shown in Table 1.{ These four different search approaches were tested in ClinicalTrials.gov and ICTRP for each of the { Although five of the six systematic reviews reported searching at least one of the trials registries [11,[13][14][15][16], only one review [15] provided a search strategy for a registry. Therefore, the MEDLINE search strategies from the remaining reviews were modified for use in searching the trials resources. { In practice, it was often difficult to differentiate the very precise from the precise strategies and the sensitive strategies from the very sensitive strategies. In some cases, the search concept was already a compound, for example, ''autologous blood transfusion.'' reviews that had studies in those resources. Each search approach was tested using both the advanced and basic search options available in ClinicalTrials .gov and ICTRP. A total of sixteen searches were planned for each review.
The authors analyzed the search results and the number of studies included in the CSR that were retrieved. We assessed CSRs according to how many of their studies were published since 2005, the year in which the ICMJE issued the requirement that clinical trials must be registered in a public trials register before they are considered for publication. The assumption was that trials conducted since 2005 might be more likely to be registered and retrievable than trials conducted prior to the ICMJE requirement.

Included studies
The eight systematic reviews addressed a variety of interventions and conditions: & anti-fibrinolytic use for minimising perioperative allogeneic blood transfusion [9], & preoperative autologous donation for minimising perioperative allogeneic blood transfusion [10], & parenteral versus oral iron therapy for adults and children with chronic kidney disease [11], & intracutaneous or subcutaneous sterile water injection compared with blinded controls for pain management in labour [12], & intramedullary nailing for tibial shaft fractures in adults [13], & tranexamic acid for upper gastrointestinal bleeding [14], & intravesical gemcitabine for non-muscle invasive bladder cancer [15], and & continuous glucose monitoring systems for type 1 diabetes mellitus [16].
Despite 1 review [9] having an unusually large number of included studies (252), only 4 of those studies (1.59%) could be located in ClinicalTrials.gov and 8 (3.17%) in ICTRP. The remaining 7 CSRs varied in terms of the proportion of their included studies that could be identified in the registers. In ClinicalTrials .gov, the number of included studies varied between 0 out of 7 for 2 reviews [12,14] and 8 out of 22 (36.36%) for 1 review [15]. In ICTRP, the number of included studies varied between 0 out of 7 for 2 reviews [12,14] and 12 out of 22 (54.55%) for 1 review [16]. The reviews for which no included studies could be located in either resource were a review of sterile water injection for pain management in labor [12] and a review of tranexamic acid for upper gastrointestinal bleeding [14]. On average, 84.00% of included studies were not listed in either resource.
The low yield of registers might have been due to the age of the trials ( Table 2). For 1 review [12], 18 of its 22 (81.82%) studies were published after 2005 and might have been expected to be registered, but only 12 of the 18 (66.66%) were identified in the registers. For another review [9], only 8 of the 206 post-2005 studies (3.88%) were identified in the registers. Even taking into account the age makeup of included studies in the reviews did not really improve the picture on study registration.
All of the records identified in ClinicalTrials.gov were also identified in ICTRP. Nine additional unique trials were identified in ICTRP (Table 2).

Search strategies
Search strategies were developed for all eight reviews (Appendix, online only); however, sensitivity could only be calculated for six reviews, because no records for included studies could be found in either  ClinicalTrials.gov or ICTRP for two of the reviews [12,14]. Four searches were planned for each review for both of the interfaces for Clinicaltrials.gov, a total of eight searches per review. The same approach was planned for ICTRP, with another eight searches per review. Ninety-six search strategies were planned in total. In practice, a strategy that was different than one of the other strategies could not be compiled for five strategies in Clinicaltrials.gov and for five strategies in ICTRP. For example, for one review, it was not possible to create a highly precise strategy that was different than the precise strategy. In addition, one ICTRP highly sensitive strategy caused the ICTRP advanced interface to freeze, so that strategy did not produce any results [13]. This meant that performance figures for thirty-eight searches (out of a possible forty-eight) were achieved in Clinicaltrials.gov and thirty-seven (out of a possible forty-eight) in ICTRP. For the searches tested, the highly sensitive search approach, using just 1 concept in the basic search interface, produced the best results in retrieving known relevant studies. The highly sensitive searches in the basic interface retrieved 100% of the relevant records for all 6 reviews in Clinicaltrials.gov and 5 out of 6 reviews in ICTRP. The searches in the advanced interface retrieved 100% of relevant records for 5 out of 6 reviews in Clinicaltrials.gov and 4 out of 6 reviews in ICTRP.
There were differences between searches run in the basic and the advanced search interfaces in each resource (Table 3). In 16 out of 19 (84.21%) searches in the ClinicalTrials.gov advanced interface, sensitivity was maintained or improved, and in 15 out of 19 (78.95%) searches, precision was improved when compared to the basic search. In 3 out of 19 (15.79%) searches, the advanced search reduced sensitivity. In ICTRP, sensitivity was maintained or improved in 8 out of 18 (44.44%) searches, and in 4 out of 18 (22.22%) searches, the precision was improved. In 10 out of 18 (55.55%) searches, the advanced interface produced reductions in sensitivity. The highly sensitive search for 1 review [13] would not run at all. In 1 case [10], the advanced search was more sensitive than the basic search.
To assess whether it is possible to provide guidance on the best search strategy approach for each resource and each interface in the resources, the type of strategy that first achieved 100% sensitivity (''successful'' search) in finding the known relevant studies was assessed. The underlying assumption was that in an ideal search situation, most searchers would prefer using the fewest and most precise terms possible to achieve 100% sensitivity. Table 4 shows how many of each search approach offered 100% sensitivity, using the basic and the advanced interfaces. Nineteen strategies were tested for Clinicaltrials.gov (out of a possible 24) in each interface. Fifteen out of 19 (78.95%) strategies achieved 100% sensitivity using the basic interface and 13 out of 19 (68.42%) using the advanced interface. Nineteen strategies were tested for ICTRP (out of a possible 24) in each interface: of the 38 possible searches, 37 yielded results. Thirteen out of 19 (68.42%) strategies achieved 100% sensitivity in the basic interface, and 8 out of 19 (42.11%) strategies were successful in the advanced interface.
One-third (2/6) of the searches using a highly precise search approach in ClinicalTrials.gov sensitivity with a precise strategy. For 3 reviews [10,11,16], sensitive or highly sensitive searches were required to identify all included studies. For the basic interface to ICTRP, one-third of searches achieved 100% sensitivity using a highly precise search approach. No single search approach was sensitive enough to identify all included studies in the 6 reviews. Only 1 of the highly precise searches in ICTRP was the same as for Clinicaltrials.gov. For 1 review [10], ClinicalTrials.gov required a highly sensitive search approach to reach 100% sensitivity, and ICTRP required a highly precise search. For another [9], ClinicalTrials.gov required a precise search and ICTRP required a highly sensitive search to reach 100% sensitivity. Successful searches in the advanced interfaces of the 2 registers showed similar variability. In ClinicalTrials. gov, there was 1 case of a highly sensitive search moving to a sensitive search with the move to the advanced interface, 1 case of a sensitive search becoming less sensitive, and 1 case of a change from highly precise to precise. For ICTRP, 6 successful strategies changed with the move to the advanced interface, the majority in the direction of requiring more sensitive searching to achieve 100% sensitivity in the advanced interface.

DISCUSSION
One of the objectives of this research was to assess whether it is possible to search for trials to include in systematic reviews using only publicly available trials registers rather than major bibliographic databases such as MEDLINE. The data presented here suggest that current trials registration still seems to be sparse. Even a very large and current CSR with 252 included studies of anti fibrinolytics [9] had few trials included in either of the studied resources. Searchers cannot yet rely on trials registers instead of major bibliographic databases, because registers do not contain all of the research evidence that is required for systematic reviews. This is true even for very new or current topics: the authors found that the majority of studies published post-2005 were unregistered. In one systematic review [15], all of the included studies were published post-2005 and yet only two were registered in the trials registers. This reinforces the findings of Gill's 2012 study of US studies registered on ClinicalTrials.gov, which found that, prior to 2004, the majority of trials for serious and lifethreatening conditions were not registered and that registration of studies often did not occur until a long time after a trial begins [17]. The latter conclusion has also been reached by other authors [18].
A considerable number of the post-2005 trials in this research were published in journals that do not publish in English or were conducted in countries other than those in the western English-speaking world: the United States, Canada, Australia, and the United Kingdom. In these circumstances, it was surprising that ICTRP did not have a higher yield, because it offers access to, among others, the Chinese Clinical Trial Register, Brazilian Clinical Trials Registry, Iranian Registry of Clinical Trials, and Japan Primary Registries Network. The search for trials is still a challenge, even with the advent and development of important resources such as Clinicaltrials.gov and ICTRP [19].
In both resources, highly sensitive single concept searches in the basic interface seemed to be the most reliable approach to identifying the known relevant records in the test reviews. Other search approaches (highly precise, precise, sensitive) worked, but not in all of the cases. This suggests that adopting a highly precise approach will result in missed trials. This is unfortunate, as the authors assume that most searchers would prefer a highly precise search involving few search terms. In the case of ClinicalTrials.gov, the use of the advanced interface seems to improve precision without loss of sensitivity, and this interface might be preferred when large numbers of search results are anticipated. The current study suggests that use of the ICTRP advanced interface may be problematic because of reductions in sensitivity. Recommendations from a recent conference presentation go further than this study's and suggest that searchers should search both registers using ''multiple basic and advanced searches'' to detect all potential ClinicalTrials.gov records because searching ICTRP identified additional ClinicalTrials.gov records [20].
Using the ICTRP basic interface requires that searches be structured in a particular way, as shown in the strategies in Table 1. Combining search terms in a specific order and using Boolean operators (in the absence of parentheses) in the correct order is quite challenging. The required search structures are unlikely to be intuitive to lay searchers who are used to Google-style interfaces, and even skilled searchers may need to plan out their searches beforehand. In these circumstances, it seems likely that many searchers are not using the ICTRP basic search interface in the most optimal way, leading them to Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform miss relevant studies and resulting in highly imprecise searches, unless they rely on single concept searching. Graham's exploration of the search patterns of searchers using transaction log analysis in ClinicalTrials.gov over 3 months in 2005 showed that searchers entered ClinicalTrials.gov at a trial record level (39%) more often than via a home page search (24%) [21]. Many users therefore did not directly use the provided search options and did not spend time refining their searches [21]. Indeed, the majority of users in Graham's study accessed ClinicalTrials.gov via external websites including Google, suggesting that precise searches might be being used [21]. This suggests, in turn, that at least the precise strategies presented here are likely to be similar to those used by busy or lay searchers who are used to searching single line interfaces such as Google. Our sensitive searches are likely to be more extensive (and possibly more successful in terms of retrieval) than those designed by many of the searchers who use ClinicalTrials.gov or reach it via websites such as Google.
Effective searches depend not only on imaginative searchers and good search interfaces, but also on informative abstracts to search. Several studies have reported that trials registers have incomplete records and poorly standardized descriptions, both of which are likely to hamper retrieval further and to encourage the adoption of sensitive search approaches, and that records may sometimes not comply with mandatory registration requirements [22][23][24][25][26]. These deficits may well have affected retrieval in this research and are certainly issues to consider when designing searches: sensitive searches should also seek to take into account inconsistent reporting and variability in reporting detail.

Limitations of the study
This project might have failed to find all of the included studies that were available in the trials registers. Despite using various search approaches, it was difficult to identify trials records and to link them to published studies. It is still not common practice to include trial numbers such as National Clinical Trial (NCT) or International Standard Randomized Controlled Trial (ISRCT) numbers in journal abstracts and articles. More consistent recording would facilitate assessments of trial publication patterns. The increased population of the PubMed Secondary Source ID (SI) field should also assist trial and publication linkage in the future. However, the current picture indicates that substantial numbers of trial reports do not include information on the trial registration in the abstract of the final publication [27], and this impedes retrieving and matching publications to register records.
Another limitation was the absence, in most of the recently updated Cochrane reviews, of the search strategies used to search trials registers. This meant that we had to develop strategies that we hoped would reflect the Cochrane reviewers' approaches. We may, however, have searched in very different ways than the Cochrane reviewers. The implementation of the new Methodological Expectations of Cochrane Intervention Reviews (MECIR) includes guidance on reporting all searches in Cochrane reviews, so we anticipate that future investigations will benefit from both the increased use of trials searches by Cochrane searchers [8] and more detailed reporting of those searches.
Our strategies were designed to reflect four search approaches from very precise to very sensitive. However, for some topics, we were not able to test all four approaches, and other searchers might use very different strategies and thus obtain very different results. Even our precise strategies tended to be sensitive, as can be seen by the limited use of phrase searching.
ClinicalTrials.gov and ICTRP have very different interfaces and offer different search options within their interfaces. The two resources are potentially most similar when searching their advanced interfaces, which both offer the option to use a field-limited search for interventions and conditions. However, the differences in the interfaces do make comparative strategy testing problematic. In particular, the ICTRP mapping of terms to the Unified Medical Language System (UMLS) is not transparent, and the implementation of phrase searching in ICTRP also seemed to produce unexpected results.
The study reported here only explored eight reviews and searches in six of those reviews, and its recommendations are cautious and conservative. Further research is needed to confirm the most efficient approaches to searching for information on clinical trials and to monitor the extent to which trials registers are comprehensively recording trials in progress. In addition, searchers should be aware that other trials registers are available that are not yet incorporated into ICTRP, so identification approaches need to be wider ranging. A guide to searching some available trials registers can be accessed at https://www.sites.google .com/a/york.ac.uk/yhectrialsregisters/.

CONCLUSIONS
Searching only trials registers to identify studies for inclusion in systematic reviews, CSRs, and health technology assessments is not advised at present because many trials are not being registered in ClinicalTrials.gov and/or ICTRP. Until trials are consistently registered in ClinicalTrials.gov and/or ICTRP, searchers cannot ignore major bibliographic databases, such as MEDLINE.
Search strategies to search ClinicalTrials.gov and ICTRP should ideally be structured to search for one concept and to use a range of synonyms and related terms to ensure sensitivity. Searches of more than one concept in ICTRP should be constructed carefully with attention to the order of processing of the Boolean operators. Glanville et al.