Introduction and background

This study is a bibliometric analysis of a highly complex research discipline, namely Geography. Within these different science fields, different publication habits exist, which are explored in this contribution based on four different individual evaluation exercises.

The discipline Geography deals with any observation, analysis, modelling, and theoretical and conceptual approach on any action and development in the space and time domain. It is indeed much more than a “regional” science. It often has a spatial focus, but it also addresses contextual thoughts and developments. Geography includes the study of earth surfaces, landscapes, people, places and the physical and mental environments of humans (Gebhardt et al. 2011b). Hereby, the Geographic domain is either addressed from specific disciplines, for example Spatial Planning or Geomorphology, but also and in particular addresses cross-cutting topics such as Natural Hazards and Risks (Dikau and Pohl 2011), utilization of natural or human resources, or global environmental changes (Glaser and Gebhardt 2011). The computerized assessment and analysis of spatio-temporal pattern has been significantly changed with the introduction and improvement of Geographic Information Systems (GIS) and the introduction of computerized visualization within the traditional field of Cartography. Today, the “Modern Geography” (Gebhardt et al. 2011a) contains fields, which can be attributed to social sciences (e.g. Population -, Economic Geography, Spatial Planning), to natural sciences (e.g. Geomorphology, Geoecology, Biogeography, Climate Geography) and to engineering and IT sciences (e.g. Geoinformation, Cartography). Besides the research focused on these sciences, a traditional focus is also on cross-cutting topics such as climate, environmental and global changes, utilization of resources or natural hazards and risks, since these are research fields which need a multi-disciplinary view on the resulting matters in order to contribute to the significant challenges humans and societies are currently facing. Since not everything can be investigated in one analysis, this analysis focuses on the research areas of Human- and Physical Geography, and does not consider the GIS and IT-domains.

From a bibliometric point of view, Geography is a very challenging discipline. The relation to the natural sciences (e.g. Physical Geography) as well as to the social sciences (e.g. Human Geography) results in an interesting bibliometric analysis of this overarching discipline. These two main directions are perfectly illustrated in the two different editions of the Journal Citation Reports (JCR). They show that the median as well as the aggregate category impact factor is twice as high in the Science than in the Social Science edition.

Although many bibliometric studies have demonstrated the differences between natural and social sciences and the different discipline-dependent publication cultures (e.g. Nederhof 2006; Australian Research Council 2012; Ossenblok et al. 2012; van Leeuwen 2013; Moksony et al. 2014), there is no literature available focusing on specific Geography related contexts.

Consequently, the main research questions of this study are:

  • Which are the most usual and most cited publication channels?

  • Which data sources are most applicable for Geography bibliometrics? Is Google Scholar indeed a useful complementary data source?

  • Are there any bibliometric differences detectable between natural science and human science oriented research fields?

Data sources and methodology

This study is primarily based on publication data collected for four evaluation exercises performed at the Department for Geography and Regional Research at the University of Vienna. These four exercises have been classed in three parts. The first and second evaluation exercises were related to “Geoecology” and to “Social and Economic Geography”, and comprised twelve and ten individuals or candidates respectively. Both of these analyses were performed during 2013 and are presented in part one. The third exercise was related to “Population Geography and Demography”, comprised nine candidates, was performed in August 2014 and is discussed in part two. The fourth and final exercise discussed in this study was related to “Economic Geography”, comprised nine candidates again, was performed in July 2015 and is presented in part three.

In all exercises, the publication data were delivered directly by the candidates, whose identity has to remain anonymous. Since it can be assumed that the candidates presented all presentations, the publication data are regarded as complete. All bibliometric indicators added to the list of publications were controlled or recalculated in order to enable a correct and comparable analysis (Gorraiz and Gumpenberger 2015a, b). Document types used by the authors in their list of publications were manually reassigned to the following standard groups: Monographs (Books), Book Chapters, Journal Articles, Proceedings Papers, Conferences (including Meeting Abstracts and Talks), Reports (Working Papers), Book Reviews, Edited Books and Journals Issues, and Other Publications (or Miscellaneous).

A clear distinction between “Proceedings Papers” and “Conferences” was not always possible when referring to the provided details in the individual publication lists. However, the resulting possible erroneous classification of entries is regarded as negligible, especially for citation analyses.

The main data source for coverage and citation analyses was “WoS - Core Collection” including the Conference Proceedings and Book Citation Indexes. Since coverage of many human science oriented research fields of Geography in the usual multidisciplinary bibliographic and citation databases (Web of Science, Scopus) is very low and therefore has to be treated carefully within common citation analyses, we have included Google Scholar (GS) as an additional data source in a first explorative attempt (Jacso 2005; Kousha and Thelwall 2007; Meho and Yang 2007).

The analysis in GS was performed by using the GS Citations Profiles. The candidates were invited to create their individual profiles and make them publicly available for at least a couple of weeks. In addition, the tool ‘Publish or Perish’ particularly designed for citation analyses in GS was used to check and amend these profiles (Harzing 2007). In the cases where individual profiles were not available, respective queries have been made in order to assemble a complete data set. It was aimed to be similar and comparable for all candidates in the further analyses.

In spite of the fact that citations were checked and the percentage of self-citations was determined, citation analyses in GS should be taken with a pinch of salt. GS is rather a search engine than a database, and therefore indexing remains non-transparent and documentation is lacking. That is why the analyses were also performed in WoS, including the Cited Reference Search. This means that citations originating from WoS ‘core journals’ were also considered for all document types without any restrictions. Furthermore, Scopus has been utilized as another primary source within the third exercise, corresponding to the following second part (Labbé and Labbé 2013; López-Cózar et al. 2014).

The used publication windows were either the last 10 years or the academic age of each applicant for all four excercises. In order to distinguish individual academic age periods, the year of the first publication activity is always included.

The observed citations window was identical for all applicants for each exercise. It covered the date from the first publication until April–May 2013 for the two first ones (1 and 2), until July–August 2014 for the third one 3, and until July 2015 for the fourth one. Visibility analyses were performed based on the data available from the Journal Citation Reports (JCR), editions 2012 (exercise 1 and 2), 2013 (exercise 3), and 2014 (exercise 4) respectively. The quartiles (Q1 = Top 25 %; Q2 = Top 25–50 %; Q3 = Top 50–75 % and Q4 = Top 75–100 %) were calculated according to the 2-years impact factor (IF) in the corresponding WoS category.

Furthermore, a statistical analysis has also been performed and includes the calculation of the mean, median, standard deviation and the confidence T functionFootnote 1 (significance level of 0.05 what equates to a confidence level of 95 %). These values are provided at the bottom of the respective tables.

Results

The results are reported in three parts according to the different excercises. This will avoid any further bias in the interpretation of the data, since the analysis and respective values always refer to the similar period of analysis (Table 1).

Table 1 Publication spectrum and WoS coverage according to provided publication list for exercise no.1 (geoecology—12 candidates)
Full size table

Part 1: Comparison between exercises 1 (geoecology) and 2 (social and economic geography)

The most important publication document types used by the candidates for both appointments are presented in Tables 2 and 3. The spectrum is much more heterogeneous in the social sciences, where articles in peer-reviewed journals are not always the most common publication channel. Furthermore, many publications in the social science domains of Geography are published in German, since these are the dissemination products which reach the targeted audience.

Table 2 Publication spectrum and WoS coverage according to provided publication list for exercise no. 2 (social and economic geography—10 candidates)
Full size table
Table 3 Visibility (Q1 and  %Q1) and citation analysis in WoS for exercise 1 (geoecology—12 candidates)
Full size table

The mean and median number of publications in Books, Edited Books and especially Book Chapters is higher in Social and Economic Geography than in Geoecology. Tables 2 and 3 also show that almost twice as many publications were produced via other channels (see Miscellaneous) in Social and Economic Geography. This reflects the already addressed importance of this research field to reach the targeted German-speaking audience.

Miscellaneous mostly comprised of Reports and Working Papers in both appointments. Therefore this document type was considered separately in the second part of the study.

In exercise 2, other document types such as Films, Policy Briefs, Newspapers and Special Issues were mentioned, albeit only individually, and counted under the category Miscellaneous. This can again be explained that in Social Science research fields, other means of dissemination of research results play an important role for making scientific research accessible. For two candidates (one in each exercise), articles in Other (non-scientific or non-peer-reviewed) journals were also assigned to the group Miscellaneous.

Concerning the coverage in WoS, both tables corroborate the still low inclusion of Books and Book Chapters in both editions of the Book Citation Index (Gorraiz et al. 2013). For articles in peer-reviewed journals, the WoS coverage in exercise 1 ranges from 60 to 100 %. This trend has been constantly increasing in the last 10 years until having reached a value of almost 90 % for all candidates. In exercise 2 the coverage was lower, with values from 30 to 60 %. However, a similar trend could also be observed, despite of the fact that the increase was not so steep. This reflects the observed situation. Within natural science oriented Geography (= exercise 1), it is more common to publish the research results in English peer-reviewed journals. However, it has been recognized by many researchers within the various fields of Human Science related Geography, that they have to increase their participation in the English research journals. As the numbers show, the numbers of respective publications raise, as well as the increased trend is visible in both science domains.

Tables 3 and 4 show the results of the visibility and citation analyses performed for both exercises. Only publications indexed in WoS in the last ten complete years (2003–2012) were considered. These results corroborate the higher number of publications and citations in the discipline related to the natural sciences (approximately twice as many). Nevertheless the number of co-authors and the percentage of self-citations, which is almost twice as high in the natural sciences, need to be taken into account as well. By doing so, no considerable difference can be observed.

Table 4 Visibility (Q1 and  %Q1) and citation analysis in WoS for exercise 2 (social and economic geography—10 candidates)
Full size table

These numbers reflect again the differences in the social and natural science oriented Geography. With the natural science domains, it is much more common to publish also many papers during the research process. In particular if this research is embedded in third-party funded research projects, a respective output also in terms of research publications in peer-reviewed journals is required. Although this is also true for social science, the respective research directions in Geography traditionally prefer to publish the results in larger contributions, thus summarizing findings in book chapters or monography rather than publishing also “smaller” research findings individually. Since one has to cite previously published work, the candidates in natural science show higher numbers in self-citations than the candidates in social science research.

The visibility analysis (number of Q1—Journal Articles) shows that publishing in top journals with high impact factor result in a much higher visibility for the appointment related to natural sciences than for the one related to the social sciences. As already previously explored, this can be related to the different value of publishing in peer-reviewed journals, which is much more important for natural than for social science related research.

Finally, Tables 5 and 6 show that the citation differences, according to the aggregate impact factor of the main WoS category, are higher for exercise 1 than for 2. For example, the WoS category “Ecology” has an aggregate impact factor of 3.095, which is almost three times higher than for “Geography” with 1.469.

Table 5 First and second research field according to WoS Categories for exercise 1 (geoecology—12 candidates)
Full size table
Table 6 First and second research field according to WoS Categories for exercise 2 (social and political geography—10 candidates)
Full size table

Both tables demonstrate, that the natural science domains contribute significantly to highly specialised international journals in order to place the respective research results in the relevant research community. The same is indeed true for social science publications; however, the respective applicants in these exercises tend to publish in Geography related journals and not so much in journals of other disciplines with higher median category impact factors. Nevertheless, it can be recognized that the researchers have become aware of this situation, and that most recently the numbers of publications in interdisciplinary journals with higher IF have increased.

Part 2: Results obtained in exercise 3 (demography and population geography)

Within this evaluation exercise, applicants were invited to create their individual GS Citations profiles and make them publicly available for at least a couple of weeks. Out of nine applicants six created their GS Citation Profile, two refused to do so, and one followed the invitation, but the profile was delivered in an incomplete status. Therefore, the tool ‘Publish or Perish’,, was then used for collecting the missing and checking the provided data (see “Data sources and methodology”).

According to the WoS Categories, the candidates were publishing mainly in Geography, Demography and Urban Studies, categories with very similar aggregate and median impact factors according to the JCR Edition 2014.

Figure 1 represents the publication activity for each scientist according to the most relevant publication types. The data are based on the list of publications submitted by the candidates. The table reveals that some candidates published their geography related research more in journals which are summarized within the category “Geography”, whilst others publish rather in journals which are categorized to other specific disciplines, e.g. “Urban Studies”. Besides this different publication strategy, also the assignment of specific journals in the different data bases varies. Consequently, the usage of such categories has to be handled with care, since it these are also dependent on the different internal classification schemes of the data bases.

Fig. 1
figure 1

Percentage of publication types for each candidate (Exercise 3)

Full size image

Book chapters and journal articles are the most frequent publication types. This confirms the already provided statement, that in the social science domains of Geography, it is much more common to distribute the research findings in these means of dissemination. The results hint at very heterogeneous and individual publication strategies based on the analysed publication types. However, a clear trend towards an increased publication within scientific journals was observed in all exercises.

The next three sections contain coverage and citation analyses performed in the three considered data sources. Citation analyses include following indicators for each data source: (1) number of indexed publications; (2) percentage of publications covered according to the provided publication list; (3) number of cited documents; (4) total number of citations; (5) number of citations per cited publication; (6) maximum number of citations attracted by a publication; (7) total h-index and (8) i-index (number of publications with more than 10 citations). The percentage of self-citations in GS was only calculated if the number of citations was of sufficient significance.

Table 7 shows the percentage of coverage in GS for each publication type. In order to distinguish individual academic age periods, the year of the first publication activity has always been included. Monographs (Books) and Edited Books or Issues are very well covered, most likely due to the inclusion of Google Books (Kousha and Thelwall 2009).

Table 7 Coverage (%) in GS for each publication type (Exercise 3)
Full size table

The coverage of Journal Articles is also much higher in GS than in WoS or Scopus (see a more detailed insight in Gorraiz and Gumpenberger 2015a, b). This might also refer to the fact, that WoS and SCOPUS include much more natural and engineering science journals than social science ones. This bias within the data base is not so evident within GS, since this database contains much more extensively also other means of publications.

The high coverage of Reports (Working Papers) is remarkable as well. Again, in the social science domains of Geography, the results have to be presented also to the specific groups on stake, and therefore, the preferred place of publishing these findings are often reports and working papers. Chapters in Books are not that well covered, but this is probably due to incidental incorrect citations.

Table 8 shows the results of the coverage and citation analyses performed in WoS, including the Cited Reference Search, in Scopus, and in GS. The higher coverage scores in WoS compared to Scopus are due to the inclusion of the Cited Reference Search, which enables the retrieval of citations not only restricted to journal articles and book indexed in WoS, but also from other books, reports and other document types cited by the core journals in WoS.

Table 8 Coverage and citation analysis in the three data sources for each candidate (Exercise 3)
Full size table

Table 8 confirms that the values of the main citation indicators (number of citations, citations per cited publication and h-index) are different in absolute values in GS, WoS and Scopus, but are comparable in terms of relative values. This observation refers again to the better coverage of social science related research in GS rather than in WoS and Scopus.

However, spearman correlations performed for these indicators (number of citations, citations per cited publication and h-index) in the three data sources (WoS, Scopus and Google Scholar) were very strong (varying from 0.8 to 0.95).

A detailed coverage and citation analysis for the three most cited document types in GS, namely Monographs, Book Chapters and Journal Articles is presented in Table 9.

Table 9 Detailed citation analysis GS for each candidate and the three most cited publication types (Exercise 3)
Full size table

The results show that the most cited publication types differ for each candidate on an individual level. As a consequence, a separate citation analysis of these publication types is then recommended for evaluation purposes. In general, Monographs have the highest percentage of cited items (more 90 %), followed by Journal Articles (~75 %) and Book Chapters (~70 %). Moreover Monographs attract the highest average of citations (cited mean), whereas Journal Articles contribute most to the total impact (total number of citations) for all the individuals. Book Chapters receive almost twice as many self-citations than Journal Articles, and Monographs show the lowest percentage of self-citations. The number of publications covered in GS and not included in the candidates’ publication lists remained very low and were irrelevant for the citation analysis.

Part 3: Results obtained in exercise 4 (economic geography)

Apart from the research focus of candidate no. 8, who was rather related to Economics than to Geography, no considerable differences were perceived, neither in the WoS category nor in the value of the aggregate impact factor (1.28 versus 1.66).

Figure 2 represents the publication activity for each scientist according to the most relevant publication types. The data are based on the list of publications submitted by the candidates. Like for the previously discussed appointments (see Fig. 1), the results hint at very heterogeneous and individual publication strategies based on the analysed publication types. Some researchers prefer to publish their research within high ranking journal (incl. Q1-journals), other do not consider journals as an important publication channel and prefer to publish in reports and book chapters. The already extensively explored different publication philosophies and strategies are clearly visible.

Fig. 2
figure 2

Percentage of publication types for each candidate (Exercise 4)

Full size image

Table 10 shows the percentage of coverage in WoS, WoS Cited Reference Search and GS for each publication type.

Table 10 Coverage (%) in WoS, WoS cited reference search and GS for each publication type (Exercise 4)
Full size table

Monographs (Books) are completely covered in GS (100 %), whereas coverage is less complete in WoS Cited Reference Search (~75 %) and almost zero in WoS Core Collection including both Book Citation Indices. The good results in GS probably originate from the inclusion of Google Books (Kousha and Thelwall 2009) as already noted in the previous section. Similar trends are reported for the other publication types. However, the percentage of book chapters covered in GS is still low (around 55 %) but high in comparison to WoS (14.1 %) and WoS Cited Reference Search (32.9 %). Less than five percent of the publications retrieved in WoS Cited Reference Search and approximately 6 % in Google Scholar could not be assigned to one specific publication type due to missing bibliographic information. It is assumed, that these related to the so-called “grey-literature” (publications, which have been published as internal reports and papers, without an ISBN or ISSN). Furthermore, both data sources, WoS Cited Reference Search and GS included other publications not mentioned by the candidates in their publication lists. These are responsible for values > 100 % given in Table 10. The bias between the two data bases of WoS’s and GS are again clearly visible.

Since the ranking of journals in Q1 to Q4 journals is getting more and more important, this classification has also been performed for both number and percentage of publications in top journals according to their impact factor (IF) in WoS (see Table 11). The Impact Factor data are based on Journal Citation Reports (JCR) Editions 2014. The quartiles (Q1 = Top 25 %, Q2 = Top 25–50 %, Q3 = Top 50–75 %, Q4 = Top 75–100 %) were calculated according to IF in the corresponding Web of Science category.

Table 11 Number and percentage of publications in Q1-Journals in WoS (Exercise 4)
Full size table

These results reveal clear publication strategies concerning journal articles. Except candidates 3, 5, 6 and 7, most of the candidates publish predominantly in top Q1 journals according to their impact factor. This reveals and confirms the already previously mentioned observation, that it is recognized also within the social science fields of Geography to publish the research results in highly visible journals.

Table 12 provides the results of the co-authorship analysis and shows very different publication habits. Authors with a higher number of co-authors are usually not only the most productive ones, but also the ones who attract the highest impact scores.

Table 12 Co-authorship analysis in WoS Core Collection (Exercise 4)
Full size table

It must be stressed, however, that a high number of co-authors can be interpreted in many ways (strong cooperation but also undeserved or gift authorship) and needs in each particular case a deeper analysis basing on the role played by each author. Therefore, many journals are already asking for the precise role of each co-author (see for example, instructions for authors in PLOS journals).

The results of the citation analysis performed in WoS, WoS CRS and GS are summarized in Table 13. Following indicators were used in WoS Core Collection: percentage of cited documents, total number of citations, citations per cited publication, maximum of citations, h- and m-indices, i10 (number of publications with at least 10 citations), and the number and percentage of Top 10 % and Top 1 % most cited publications calculated for each WoS category. The two last indicators related to the top most cited publications were not available in WoS CRS and GS, since these data sources do not provide reference citations values per subject category and year of publication. So these were substituted by different variations of the i-index (i10, i50, i100, and i100), according to the number of citations attracted in each source. Table 13 shows that the values of the main citation indicators might differ in absolute values in GS, WoS Core Collection and WoS Cited Reference Search. Nevertheless, they correspond very well overall and thus, they are all valid and applicable in bibliometric studies.

Table 13 Citation analysis in the three data sources for each candidate (Exercise 4)
Full size table

Last, but not least, Table 14 shows a detailed citation analysis (including the total and mean number of citations, and the number of citations per cited items) in Google Scholar for each candidate of Exercise 4 and for the three most cited publication types monographs, book chapters and journal articles.

Table 14 Detailed citation analysis GS for each candidate and the three most cited publication types (Exercise 4)
Full size table

Concerning the percentage of cited documents the results of the citation analysis are very similar to the ones reported for exercise 3. It is evident for this research field, that journal articles contribute most to the total impact assessed by the number of citations. The most collaborative candidates no. 2 and 8 also achieve the highest citation scores.

Conclusions and discussion

This study illustrates the special situation of the Geography discipline with its foundations as a natural and a social science discipline as well as a discipline which covers and addresses cross-cutting topics. Since research initiatives and networks are commonly oriented towards external cooperation to other disciplines or to focused national or international networks, the respective publications are also positioned within these links. This observation is explored in the following three different subsets.

Publication strategies

Differences between research fields related to natural and social sciences are clearly visible. While in natural science, publications in highly ranked international peer-reviewed journals is of highest importance, the publications within the social science domain refer often to reports, book-chapters and also monographs. However, higher productivity (number of publications per year) and citation counts are relativized when also considering the higher number of co-authors and percentage of self-citations.

The results hint at very heterogeneous and individual publication strategies considering the selection of adequate publication channels. This is even true within the same research fields. Rather heterogeneous and individual publication strategies regarding the number of co-authors and the percentage of single-authored publications have been revealed even within the same research field. Certainly authors with a higher number of co-authors are usually the most productive ones, which results in attracting the highest impact scores. Hence this study corroborates one more time the benefits of co-operation also in this discipline.

General publication strategies, especially those based on the impact factor, are still more evident in the fields related to the natural sciences but they are also getting increasingly important in fields related to the social sciences. This might also be attributed to the publication language. With social science research, it is much more common to publish in the native language German. The reasonable argument herein is, that the study objects are embedded in the cultural language environment, and consequently, the publication of the research findings have to be made available in particular to this targeted group or entity. In contrast, research publications in natural sciences focus increasingly on the international audience, and thus, the researchers are much more present at international conferences, contribute more commonly also in international networks. Consequently, this leads to a higher percentage of English publications in international highly ranked peer-reviewed journals. Nevertheless, it has been recognized by all researchers in the Geography discipline that research needs to be distributed through all channels in order to raise the awareness of the performed analysis and achieved results. This is also visible in many time-line investigations of bibliometric analysis.

Publication channels and their impact

Journal Articles and Book Chapters are the most common publication channels within Geography. A research output analysis performed a posteriori in the Current Research Information System (CRIS) of the University of Vienna (http://bibliothek.univie.ac.at/ucris.html) for the Department of Geography and Regional Research (IfGR) during the years 2012, 2013 and 2014 can be summarized as follows: (a) almost one third of the publications are published as contributions in journals, another third as contributions in books and less than one tenth as monographs, (b) 60 % of the journal articles were indexed in peer review journals, (c) half of the journal articles were indexed in the Web of Science (Science Citation and Social Science Citation Indexes), and (d) 42.5 % of the items indexed in WoS were published in Q1-Journals according to their impact factor in the corresponding Journal Citations Reports Edition (Top 25 %). The results gained in this study are in very good agreement with this analysis performed in the CRIS of the University of Vienna. This suggests that the general observations discussed in this contribution can also be seen in the IfGR of the University of Vienna.

Monographs, Journal Articles (including Proceedings Papers) and Book Chapters are the most cited document types in the fields strongly related to the natural sciences as well as in the ones related to the social sciences, with a different distribution within these three types, however.

In social science Geography, other publication channels including also non-scientific or non-peer-reviewed sources, like Films, Policy Briefs, Newspapers and Special Issues are very usual, even if they do not attract a high number of citations. Working Papers are also a relevant document type and require special attention, since they can attract a considerable number of citations. All these publications are commonly written in German. Their frequency certainly depends on the addressed target group. Tracking these channels can be used in order to assess the societal impact of the research performance in these fields within the regional setting and how these studies are embedded in the surrounding, often local communities.

Comparison between data sources

As the analysis has shown clearly, citation analyses for articles, monographs, book chapters and reports (working papers) should be performed separately and require the inclusion of complementary data sources. Otherwise relevant publications can easily be missed, resulting in wrong interpretations. The results show that no considerable differences are observed when using WoS or Scopus. The inclusion of the Cited Reference Search allows for a significant coverage and is more practicable in WoS than in Scopus.

The coverage, especially of books (almost 100 %), is much higher in GS and suggests the recommendation of this complementary data source, although it is still a “black box” from a data base perspective (no transparency, missing content information, etc.). In this study the accuracy of the citations in GS was very high (~95 %). Nevertheless further analyses are needed to reduce the noise of GS data in order to increase the significance of the results.

The values of the main citation indicators might differ in absolute values in GS, WoS, WoS Cited Reference Search and Scopus, but are comparable in terms of relative values. This fact suggest an “iceberg analogy”. The citation analysis in WoS and Scopus reveals only the ‘visible part’, which usually can be measured. However, the much bigger ‘invisible part’ remains inaccessible for these citation databases, but can be explored to some extent by Google Scholar. Still, only future methods, products and metrics will be able to surface the currently hidden parts of the iceberg.

General conclusions

Peers have to be aware of blind spots in ‘citation analyses’ (e.g. ‘non cited’ document types and publications, manuscripts wrongly assigned to specific types) with potentially harmful consequences in evaluation exercises. This should, however, not result in the conclusion that such an analysis is not useful. In contrast, it should be stressed, that the bibliometric analysis is most important and useful, but still has to be handled with great care. As a consequence, it has to be stressed that citations can only be used as a proxy for impact (and not for the quality) of publications produced in the ‘publish or perish’ community (i.e. the scientists who are committed to publishing their results). One has to use these metrics with caution. High “numbers” cannot always be related to “high quality” science. For example, a really bad publication including possibly even wrong data might be cited very often as an example of a bad paper—but consequently receives high bibliometric values. Although it is an extreme example indeed, it demonstrates the necessity for cautiousness when dealing with—and interpreting—such bibliometric indices.

Last but not least, the scientific community does not only serve for academics. It comprises also teaching academics as well as representatives from government or industry, who rather use that cite scientific output directly. Furthermore, our society has become progressively informed (‘societal impact’). Unfortunately alternative metrics (like usage metrics and altmetrics) are still in their infancy (Kurtz and Bollen 2010; Priem et al. 2012; Gorraiz et al. 2014; Hammarfelt 2014) to measure the impact beyond citations and could not yet be applied to the described appointment procedures due to the current lack of available and reliable data. Nevertheless, the provided data are an important accompanying information source in any selection process.