Prostate Cancer Research, 2000-16, its Citation Impact and its Influence on Clinical Practice Guidelines

We evaluated prostate cancer research outputs from leading countries to see if they reflected the countries’ research expenditure and disease burden and determined their impact. Were the countries making the largest contribution to the evidence base of clinical practice guidelines (CPGs) for prostate cancer also those whose papers received the most citations on papers? We selected papers in the Web of Science (WoS) from 2000-16 with a complex search filter and analysed their characteristics and citations. We compared countries’ outputs with their overall research expenditure and their burden of disease from prostate cancer. We collected 71 clinical practice guidelines (CPGs) from 28 countries and downloaded their references. Although world output increased from 1696 to 4329 papers over the study period, prostate cancer research represented only 3.6% of all cancer research in 2016. Europe’s relative output was less than half its relative cancer burden and that of Africa only one sixth, but Asia, whose men are less likely to suffer from the disease, published a proportionate amount. The USA still has the largest output (31% of the total, down from 53% in 2000) but China’s output has risen very rapidly and is now second. The US and Netherlands papers were the most cited in the WoS and those from Belgium, the Netherlands and Sweden were the most cited on the CPGs. These CPG references involved research on the main treatments but relatively few on genetics. Some countries’ CPG references were rather old. Prostate cancer research is relatively neglected in Europe and particularly in Africa, but receives more attention in North America, the only continent where its disease burden relative to all cancer has actually declined. The best-performing countries in terms of their influence on CPGs differed from those with the best citation records on the WoS.


INTRODUCTION
Prostate cancer is a disease that affects older men and as men are living longer in all continents, its burden is becoming progressively greater, see Table 1. This shows the burden in Disability-Adjusted Life Years, DALYs, which take account both of the shortening of life (compared with that in Japan) and time lived with a disability, including pain. Many cells in the bottom third of the table show that prostate cancer has increased between 2000 and 2015, particularly in Eastern Europe (EEU). It is noticeable that the burden is relatively much higher in Africa (AFR) and in Latin America and the Caribbean (LAT), than in Asia (ASI). This genetic variation was previously noted for different racial groups within the UK, the USA and elsewhere. [1][2][3] However, it is higher still in Oceania (OCE), Europe (EUR) and North America (NAM).
The main current challenges are to detect the first signs of prostate cancer, mainly with the Prostate Specific Antigen (PSA) test, although it is not reliable [4,5] and the search for an indicative gene has not yielded positive results. [6] It is necessary to determine whether it will advance rapidly or be quiescent, when "watchful waiting" may be the recommended procedure. [7,8] Several treatment options exist in localized disease including surgery, radiotherapy, brachytherapy and hormone therapy. Metastatic disease is usually treated with systemic agents, with primary treatment based on hormone therapies and with an increase in different treatment options when the cancer becomes resistant to hormone therapy "castrate-resistant".
Although there are many bibliometric studies on cancer research, we were only able to find one that specifically treated prostate cancer publications in a quantitative way. [9] It examined prostate cancer papers in the Scopus database in 2004-13, found an Indian contribution of 1.5% and also analysed international collaboration and the division of Indian outputs between subject areas. This paucity of studies on prostate cancer contrasts with the analysis of outputs of papers on lung cancer (three between 2000 and the present) and on breast cancer (eight). Our paper examines not only the outputs of prostate cancer papers, but also their impact. This will be measured in two ways: through counts of their citations on other papers in the Web of Science (WoS, © Clarivate Analytics) and through an analysis of how often they have been cited in the evidence-base of clinical practice guidelines (CPGs) globally. This is a relatively new means of research evaluation as until recently there has been no comprehensive database of these references, so individual studies have perforce needed to undertake their own work of collection of the CPGs and transformation of their lists of references into a standard format that can be matched to existing bibliographic databases, such as Scopus (© Elsevier) or the WoS. However, this is changing as Minso Solutions AB in Sweden have recently developed a large database of such references [10] which can be used for research evaluation.
The use of the references on CPGs as a means of research evaluation began with two papers from the Wellcome Trust. [11,12] We have continued this work on a larger scale [13][14][15][16][17][18] and applied this systematic methodology to several disease areas. The conclusions have been consistent, namely that the papers cited on CPGs are very clinical (as opposed to basic); they preferentially cite authors from the CPG country; and that the gap between citing and cited documents is very variable and depends on the country of the CPG. It is clear that this methodology provides a valuable complement to the normal counts of citations in the serial literature and gives a more direct indication of the utility of research for diagnosis and treatment of patients.

MATERIALS AND METHODS
Papers in prostate cancer research were identified in the WoS by means of a proprietary complex filter for cancer (ONCOL), which contained the names of 185 specialist cancer journals and 323 title words or phrases. This filter has been calibrated [19] and its precision (specificity) was 0.95 and its recall (sensitivity) was 0.98. A subset of these papers was then identified by means of a small prostate filter, consisting of three specialist prostate cancer journals, four more general prostate journals and the title words, Gleason, Prostat* and PSA. All papers in the three specialist journals were taken, together with those identified by the ONCOL filter that were also in one of the four prostate journals or had one of the three title words. Bibliographic details of articles and reviews from the 17 years 2000 to 2016 were downloaded and converted into an MS Excel spreadsheet by means of a program written by Philip Roe of Evaluametrics Ltd.
Each paper was marked with the fractional contributions of the countries in the address field. For example, a paper with one French and two Italian addresses would be marked as FR=0.33, IT=0.67. Our attention was focused on the leading 18 countries, which are listed, with others, in Table 2 with their International Standards Organisation (ISO) digraph codes. The outputs of the leading countries were plotted against the national gross research expenditure, measured in billions of US dollars. Papers from the different countries were also normalised with respect to their overall cancer research outputs and the ratio plotted against the fraction of their cancer burden attributable to the prostate.
Five-year citation scores to the papers from 2000 to 2012 were also downloaded from the WoS and matched to the individual papers; these were designated Actual Citation Impact or ACI. We determined the mean value of this indicator for leading countries. We also ranked them by the percentage of their citable papers with enough citations (56 or more) to put them in the top 5% of ACI values, which we designated as their "World Scale" value by analogy with world oil tanker charter rates [20] This shows what proportion of their papers were of high impact. The research level (RL) of the individual papers, from clinical observation through to basic research, was determined from words in their titles [21] as clinical (RL = 1.0) or basic (RL = 4.0), or both (RL = 2.5), from which the mean RL of groups of papers could be determined. The papers were also classified by their type of research, or domain, such as genetics or surgery, by means of another program, based on title words and journal name strings. Some of the papers involved clinical trials, which were identified from their title words only, such as controlled trial, double-blind and phase.
We searched the Web for prostate cancer CPGs from as many countries as we could find (n=28) and from international organizations (mainly European ones). Most of the texts were in English, but for those that were in national languages it was easy to identify the lists of references and to process them by copying and pasting them into an MS Excel spreadsheet. Their individual components (authors, title, year, journal abbreviated name, pagination) were then separated out and search statements constructed for use on the WoS. These usually consisted of the name of the first author, the title (or the three longest words from it) and year. They were concatenated into groups of 20 and these papers were sought on the WoS. Some were not present, either because they were not in journals, or not in journals processed for the WoS in that year. Some additional papers were found, mostly comments on existing papers, or authors' replies, that had the same title and publication year as the main article; these were eliminated from the file. The resulting downloaded text files of CPG references were then added to another MS Excel spreadsheet, with each reference also annotated with details of its citing CPG, including its country and date. This spreadsheet was analysed in a similar way to the spreadsheet of the prostate cancer papers (PROON), see above.

Prostate cancer research papers and citations
During the 17 years, 2000 to 2016, we found 52,943 prostate cancer research papers in the WoS. They represented 4.5% of the tally of all cancer research papers. The proportion reached 5.2% in 2007 but dropped to 3.6% in 2016. This is barely half the percentage of the world cancer burden attributable to prostate cancer in 2015 (Table 1) and only about half of this percentage in Europe and North America, where most biomedical research takes place. Figure 1 shows these percentages for the seven continental regions, averaged over the study period.
The volume of output from 19 leading countries for the last four-year period (2013-16) is plotted against their Gross Expenditure on Research and Development (GERD) in 2015 in Figure 2. This usually gives a better correlation than a plot against population and in this instance, the correlation is also somewhat better than a plot of output versus wealth, shown  Figure 3, which also gives their respective outputs as a percentage of the total. We identified 1807 papers (3.4%) as having reported clinical trials. The highest percentage was for the Netherlands (7.8%), followed by Denmark and Finland (6.4%) and Belgium (6.2%). They were followed by Canada with 5.3%, the UK with 4.9% and the USA with 3.4%. The lowest percentages were for Taiwan (0.4%), China (0.6%) and India and Israel (0.7%). The distribution of the papers among the countries is shown as two pie diagrams, one for 2000-04 and the second for 2013-16 and reveal the changes that have occurred over the study period, see Figure 4. The most striking change is the relative decrease in the share of North America and the corresponding increase in that of Asia. This increase was mainly owing to the output of China, which went from 71 papers in 2000-04 to 1932 in 2013-16, or by a factor of over 30. The output of South Korea also increased rapidly, from 43 to 514 papers between the two periods. Citation scores of the leading countries are shown in Table 3. Papers in the different research domains received different citation scores, see Figure 5.

Citations on clinical practice guidelines
Our web search yielded a total of 71 CPGs from 28 countries and several international organizations, published from 2001 to 2019. In total there were 10,273 references that we were able to identify in the WoS, of which 5,962 were unique.  by Gross Domestic Product (GDP). The correlation is moderate, with Canada (CA) publishing the most relative to its research expenditure, followed by Italy (IT), Australia (AU) and the  Note: abscissa is on log scale, so straight correlation line is an exponential fit.  Table 4 shows the countries and international organizations whose CPGs were found and processed, with the total numbers of references from the country -some countries had several different CPGs. Many references were cited multiple times: one on 26 CPGs and another on 25 of them; both papers were in the New England Journal of Medicine. Figure 6 shows the numbers of references with given numbers of CPG citations (between 2 and 10). The least-squares correlation line is straight, showing that these numbers decreased exponentially with citation scores. For country codes, see Table 2.
There was a marked concentration on research domains concerned with treatment, as well as prognosis, see Figure 7, but there were relatively very few papers on genetics. This is not surprising as little genetic testing is carried out on prostate cancer patients to identify the best treatment and there are no The individual references from CPGs from different countries could be characterised by the mean number of CPGs that cited them. This "citation count" is not as rigorous as the WoS mean value of ACI, or its World Scale value, as most prostate cancer research papers are not cited at all on CPGs, our selection of them is by no means exhaustive and the citation window is very variable. Nevertheless, we can compare the mean Guideline Citation Impact (GCI) with the WoS mean ACI and with other parameters such as the mean RL of a country's prostate cancer research papers and its percentage of these papers that report clinical trials. The latter graph is shown in Figure 9. It is striking that Sweden (SE) and Finland (FI) show to advantage in all three correlations. Their spots lie well above the line in Figure 9 but are not so prominent in Figure 8.
On the other hand, China, Japan and South Korea perform less well, probably because their clinical journals, designed for the information of national medical personnel and therefore in national languages, are not covered in the WoS. Thus, the numbers of papers in the prostate cancer research file in their languages were each in single figures, whereas 12% of the German papers and 23% of the French ones, were in German and French, respectively.
Because the selection of CPGs is far from uniform and the results may well be biased because of the tendency of medical researchers to over-cite their fellow countrypersons, [22] it is better to show CPG citation scores (GCI values) for all CPGs other than those (or that) of the cited authors' own country. On this basis, the best performing countries are the Netherlands, Sweden and the UK. However, probably of greater importance is the influence of each country's research on its own CPGs as this will directly affect the quality of clinical care in the country. New Zealand (NZ) is the country most intensively citing its own country-people, its Over-Citation Ratio (OCR) being as high as 26, although its own papers account for only 6% of its CPG references. On the other hand, the US CPGs good means to provide a useful prognosis of the likelihood of successful treatment.
The research level of the cited references was very clinical (RL p=1.14), compared with that for prostate cancer research papers (RL p=2.02).
The analysis of the countries that contributed to the CPG references ( Figure 8) showed that the smaller European   For country codes, see Table 2. Log-log scales. numbers of citations received by papers on genetics, which is of less utility in the treatment of patients. If the library of CPGs and their references could be systematically analysed, this may be a very useful indicator for research evaluation and could benefit clinical researchers who may currently feel disadvantaged when they apply for grants, as clinical work is often less cited than basic research.
Our second observation is that some country CPGs rely on much more recent research evidence than others ( Figure 10). This means that the recommendations on the CPGs from those countries on the right of the chart may be somewhat outdated, as the field is advancing rapidly, particularly in screening and diagnosis. The guidelines span an 18-year time period for some countries, although the majority are from 2012 onwards. As this is quite a long time, given the importance of the topic, perhaps those organisations should consider a shorter time frame, such as three years. [23] We emphasise the importance of a balanced compromise between time and pragmatism in the production of scientifically valid and rigorous clinical guidelines. [24,25,23,16] Our third point concerns the factors that appear to be positively correlated with frequent citation of a country's papers on CPGs. This is a parallel investigation to the many papers that have sought to try to explain the variation in numbers of WoS citations with the parameters of individual papers. These include numbers of authors, numbers of acknowledged funding sources, research level and the amount of international collaboration. [26][27][28] Such factors are unlikely to be relevant here and our database excludes the large majority of prostate cancer research papers that are not cited on CPGs. What we have demonstrated, without of course proving causation, is that there is a positive correlation between a country's CPG citation performance and (a) its papers being well cited by other papers in the WoS; (b) its papers being clinical rather than basic; and (c) its relevant research papers including many clinical trials. The last of these is a novel finding and could influence the composition of its research output portfolio if a country wished to improve the basis for new and updated recommendations for good clinical practice through CPGs.
Fourth, we have noted the tendency of a country's CPGs to over-cite its own research and that this over-citation ratio is higher for countries with small research outputs. [22] South Korea (KR), Belgium (BE) and Italy (IT) appear to be outliers in that they self-cite less than might be expected. This suggests that their research is not having the effect on their national guidelines that they might have expected. This inter-country comparison could be rather useful as a means to evaluate whether a country's investment in research was likely to lead to good recommendations for the health care of its citizens, which must surely be one of the main reasons why countries carry out medical research.
are far more US-centric, with 56% of their evidence-base coming from US researchers.
The final analysis is of the mean gap, in years, between the date of publication of a CPG and the average publication date for all the references that it cited. This shows the uptake of newer research in the evidence-base of some CPGs. [16] This is shown in Figure 10. The columns are tinted according to the continent to which the countries belong.  Table 2.

DISCUSSION AND CONCLUSION
The principal conclusion is that prostate cancer is underresearched within cancer, relative to its disease burden and that the situation is not improving. Europe falls short in its relative research output by a factor of about two. In Africa, where prostate cancer is still only a minor burden, but where for genetic reasons it has a disproportionate likelihood of occurrence; research output falls short by a factor of six. On the other hand Asia, particularly China, is now carrying out an appropriate volume of prostate cancer research. Output from Canada and the USA has historically been high and perhaps as a result this continent uniquely now suffers a somewhat lower burden (relative to all cancer) than it did in 2000.
It appears that countries whose research is more clinical and who carry out relatively more clinical trials have more influence on the CPGs in our collection and on ones from foreign countries, see Figure 9 above. We found, by way of illustration, that prostate cancer research papers from 2000-12 with a journal research level between 1. This means that citations on CPGs provide a welcome balancing indicator. Specifically, they are not influenced by the large Our study has several limitations. The first is that the outputs of papers in our analysis of the WoS were quite low, only 40% of those in Scopus found earlier. [9] This may reflect the wider geographical coverage of journals by Scopus, but also the Guptas' search strategy, whose precision was not determined. Nor did they give details on how they determined the subject areas within their collection of Indian papers.
Secondly, although we sought diligently for prostate cancer CPGs from as many countries (and international organizations) as we could find, we may have missed some from countries that we know are active in the publication of CPGs, such as Brazil, Latvia, Norway and Switzerland, particularly if they were published in languages other than English.
The third limitation is the uneven coverage of the guidelines from different countries and specifically the bias introduced by some having many more references than others, see Table 4. There appears to be a much bigger variation in the numbers of references from individual CPGs than there is from prostate cancer research papers, probably because there are no formal limits on numbers whereas many journals impose such limits on papers submitted to them.
The fourth limitation is that we have only included references in the serial literature and we are aware that some CPGs rely also on reports, book chapters and other guidelines and some of these will embody recent research. Moreover, we confined our search for the bibliographic data on the cited references to those ones that were processed for the WoS. In practice, this meant that only relatively few journal papers were not included in our database.
A fifth limitation, by no means confined to this study, is that we did not distinguish between the relative importance of the different references to a CPG. Clearly when recommendations for treatment are being made in a CPG, they must be based on the best-conducted clinical trials, involving the largest numbers of patients and a double-blind procedure. We have shown that clinical trials are an important route whereby research does get translated into recommendations in CPGs.