Evaluation of 50 Greek Science and Engineering University Departments using Google Scholar

In this paper, the scientometric evaluation of faculty members of 50 Greek Science and Engineering University Departments is presented. 1978 academics were examined in total. The number of papers, citations, h-index and i10-index have been collected for each academic, department, school and university using Google Scholar and the citations analysis program Publish or Perish. Analysis of the collected data showed that departments of the same academic discipline are characterized by significant differences on the scientific outcome. In addition, in the majority of the evaluated departments a significant difference in h-index between academics who report scientific activity on the departments website and those who do not, was observed. Moreover, academics who earned their PhD title in the USA demonstrate higher indices in comparison to scholars who obtained their PhD title in Europe or in Greece. Finally, the correlation between the academic rank and the scholars h-index (or the number of their citations) is quite low in some departments, which, under specific circumstances, could be an indication of the lack of meritocracy.


Introduction
Scientometric evaluation and ranking of universities, departments and scholar constitutes a widely accepted topic which informs decisions both for students as well as for academics. In general, two main factors are subject to evaluation: The quality of the educational process offered by a specific institute or scholar and the quality and the quantity of the scientific outcome. The latter is largely based on scientific indices such as as the total number of publications, citations, h-index, i-10 index and others, such as m-index, which are subsequently calculated using the before mentioned indices (Hirsch, 2005). The process of collecting such data is greatly facilitated by web-based scientific databases, namely Google Scholar, Web of Science and Scopus. In addition, there are other databases such as EconLib and PubMed, which focus on a specific scientific discipline.
One of the most popular indicators for scholars' scientific quality evaluation is the h index Hirsch (2005). The main advantage of the index is that it combines both scientific productivity and quality as measured by the impact of his work to the rest of the scientific community in terms of citations (Glänzel, 2006). The h index is based on the distribution of citations of a specific scholar and is calculated as follows: a researcher has h-index equal to n if she has n publications with at least n references each and every other publication received less than n citations (Hirsch, 2005).
A second advantage of h index is that it is quite easy to evaluate objectively and quantitatively a researcher, a process which is necessary for decisions related to professors' hiring and promotion, research funding and for nominating awards such as the Nobel and Turing prize (Hirsch, 2005). For instance, Hirsch (2005)  At the same time, by using the h index the comparison between researchers is accomplished in a more complete manner (Hirsch, 2005). Not suprisingly h-index is used many times as a measurement instrument of scientists, journal and departments of different disciplines (Abramo, Cicero, & D'Angelo, 2012, Altanopoulou, Tselios, & Dontsidou, 2012, Egghe, 2010, Glänzel, Schubert, & Lazaridis, 2010, Kazakis, 2014, Kazakis, Diamantidis, Fragidis, & Lazarides, 2014, Shin, Toutkoushian, & Teichler, 2011, Thijs, & Debackere, K., 2011.
Moreover, during university rankings of computer science departments, it was found that the rankings based on measurement reports for a scholar and their ratings according to the h index, showed a strong positive correlation (Cronin & Meho, 2006). Thus, nowadays it is widely accepted that the h-index offers a meaningful way to identify differences in scholars, departments or journals (Cronin & Meho, 2006, Glanzel, 2006, Mingers, & Leydesdorff, 2015.
Beyond the obvious advantages of h-index, there are some drawbacks (Mingers, & Leydesdorff, 2015, Moed, 2006. The h index grows with new publications which can receive a number of citations, but it also grows with new references added to already existing articles. So a researcher can increase her h-index, without having to publish for a time (Glänzel, 2006). Another drawback is that by using h for measuring the quality of a younger scientist the results are not always representative. That is because a new researcher is not active for the same period of time, and time is required to receive citations in new articles. This is especially true in the Social sciences and Humanities, an article can take five to ten years to receive a significant number of citations (Harzing, 2010, p. 10). Another shortcoming of the h-index is that it doesn't take into account any attention to the number of writers who participate in a paper as well as the contribution of each one in this (Hirsch, 2005). Thus, a researcher can increase the h index through collaborations where their role is not significant, apart from contributions where he is the main or the second author.
Apparently, such an approach could lead to superficial results since the publication practices as well the mean impact factor of each field vary significantly (Costas & Bordons, 2007).
Differences in standard h values in the various fields, are mainly influenced by the average number of citations to a paper, the average number of publications produced by every scientist in the field, and the size (number of scientists) in the field (Egghe, 2010). Therefore, it would be better if the comparison is carried out strictly between researchers belonging to the same scientific field or at least normalize the results in order to be comparable. For instance, Batista et al. (2006) report that the ratio between the mean h indices for the scientific disciplines of biology and mathematics is 3:1. Scientists working in 'smaller' or marginal scientific areas will not achieve the same high h values, compared with those who work in extremely topical areas (Hirsch, 2005).

Study objectives and questions
In general, this paper aims to highlight the positive contribution of scientometrics and its usefulness in issues related to higher education quality evaluation (Marchant, & Bouyssou, 2011). In specific, it attempts to provide answers to the following research questions: 1. Are there any significant differences in publications, citations, h index and i10 index between departments of the same discipline which are located in different universities?
2. Are there any differences on the academics' scientific performance (as expressed using the h-index) between those who report detailed information about their research on the department's website and those who don't?
3. Are there any differences on the scholars' performance according to the location in which they completed their PhD (namely Greece, other European countries, or USA)? 4. Is there any correlation between the academics' rank (Full Professor, Associate Professor, Assistant Professor, Lecturer) and their h-index and total number of citations?

Correlation (correlation) and (a) h-index (b) citations/references
The rest of the paper is organized as follows. First, the research design is described in detail and the tools used to collect the data are discussed in brief. Subsequently, the results of the analysis are presented for each department and for each research question. Finally, the obtained findings are discussed and future goals are derived.

Research Design
Fifty departments from Science and Engineering disciplines were selected for the study (see Table 1). All in all, 31 Science and 19 Engineering departments were evaluated. The procedure proposed by Altanopoulou, Dontsidou and Tselios (2012) was followed to record and analyze the data. The names, surnames and academic grade of all the academics were recorded. The program Publish or Perish (PoP) was used to calculate the total publications, citations, h-index, i-10 index, m-index. If a scholar used at the time Google Scholar Profile the related data were collected from there, instead of using PoP. Subsequently, for each department the following indices were calculated: the mean median and standard deviation on publications, citations, h-index, i-10 index and the mean m-index, the percentage of academic members who report information on their website and the percentage of academic members who retain a Google Scholar Profile. Subsequently, the aggregate results were calculated for each department, as well as for the departments of the same discipline. In some cases, synonymy could slightly affect the presented data, since there is always the possibility of having two scholars with the same name and surname. In such cases, a Google Scholar profile greatly assisted the procedure. If this was not the case the data were cleaned and the affiliation of each author was closely examined. However, it is difficult to claim 100% success while in the process of evaluating 1978 faculty members (Altanopoulou, Dontsidou, &Tselios 2012).
As mentioned above, the Google Scholar database was the source to retrieve the scientometric data. In addition to free access offered by Google Scholar, there are still 3 advantages which characterize its use. Google Scholar is easy and straightforward to use. It is also quite efficient, because the search of information takes place immediately without needing additional registration steps to access the available data. Finally, the main advantage of Google Scholar is the wide coverage of scientific disciplines and publication venues which surpasses both Scopus and Web of Science (Bar-Ilan, 2008b). The information related to the scientific activity of a specific scholar, covers not only notable and reputable scientific journals, but it also contains references from books or book chapters, conference proceedings and technical reports reports which are not indexed in Web of Science and Scopus databases (Harzing, 2010, Mingers, & Leydesdorff, 2015. As far as the collection of data is concerned each of the databases follows a different approach, which in turn affects the results in total numbers of publications and reports (Bar-Ilan, 2008a, Bar-Ilan, 2008b. For this reason, and in conjunction with the margin of error in the algorithm of Google Scholar, there is the possibility that the number of citations for a specific to is actually smaller (or even higher) than it appears. This can happen for a variety of reasons such as unrecognized text format, or error in recognition of the date of publication (Jacso, 2006). In the process of scientometric evaluation, various indicators are nowadays widely accepted and used such as the total number of publications, citations, h-index, i-10 index and others, such as m-index, which are calculated using the before mentioned indices (Hirsch, 2005). The data for all faculty members and department were collected from 10 March 2015 to June 1 st , 2015. The data were recorded and analyzed using Google Sheets and SPSS v.21 and are presented in the following section.

Presentation of the results
In this section, the aggregate evaluation results are presented by subject area. The following data are presented: the number of academics in a department; median and mean number or publications per faculty member (as well as standard deviation); median and mean number of citations per academic (and standard deviation); median and mean number of h-index (and standard deviation) mean and median number of i10-index (number of papers which have at least 10 citations each). Moreover, percentage of academics who report scientific activity on the departments' web site and the percentage of scholars who retain Google Scholar Profile is presented. The departments were ranked according to their median h-index.

Departments of Science
In the departments of Mathematics (Table 1)  One may notice significant differences in mean and median number of publications, citations and h-index between the departments. This demonstrates that departments in the same scientific subject, which have the same resources (for example, financial support from the Ministry of Education, comparable infrastructure and exactly the same wage for each academic according to their grades), have notable differences in research outcomes. However, no official national report states those differences. This is also evident in other scientific disciplines, as discussed below.    Among the departments of Chemistry (Table 4), the department at the University of Crete is ranked first in all indices: h-index, publications, citations, i10-index, m index.
The department of Athens has the lowest indexes (citations, i10-index, h-index), but 100% of its members maintain Google Scholar profile. The department at the University of Ioannina has the lowest mean number of publications.  Thessaloniki. As far as mean h-index is concerned, third is ranked another department located in Athens but at the Athens University of Economics and Business (AUEB).
On the other hand, the departments at the Harokopio and Ionian University have the lowest scores on publications, citations, h-index and, i10-index. In general, the online reporting rates are quite high for all departments, as well as the degree of adoption of GS profile among the scholars. Among the three departments of Chemical Engineering (see Table 7) the department at the University of Patras presents the highest numbers in most evaluation indexes in relation to the other two similar departments. It is also remarkable that the number of Academics at the Department of Athens is almost the double in comparison to the other two departments. However, the values of all indexes are quite lower than those at the other two departments. Among the Civil Engineering departments (Table 8), the department at the University of Athens is ranked first in all 5 evaluation indexes, while the department at the Democritus University of Thrace scores the lowest numbers. The department at the University of Patras has the highest rates to members who report their research work at the department's web site and maintain GS profile. From Table 9, it emerges that the department at the National Technical University of  The results in Table 10 show that the department of Mechanical Engineering at the University of Thessaly is the first in the ranking, while the department at the Concluding, the biggest differentiations between departments of the same discipline are presented in the departments of Sciences. In some cases, the difference in the h index between the first and the last Department exceeds 11 points, and the difference in the mean number of publications 100. The deviations in Engineering Departmetns are rather smaller (up to 6 h-index points and 82 publications, respectively).

RQ2. Differences between academics who report detailed information about their research on the department's website and those who don't
Further analysis was conducted to examine possible link between scientific output and scholars' who report academic activity. The purpose was to determine whether there was any difference in the level of their research, among those who report their scientific activity and those who do not. In Table 11 the data collected for the departments of each academic discipline are presented.   In specific, in 7/10 of departments' categories a statistically significant difference in all indicators of assessment emerged depending on the region where PhD was obtained. As mentioned above, further analysis for each pair was conducted (i.e. GR-EU, GR-US, EU-US). It seemed that differences were evident in the pair of GR-US, in which 76% (38/50) of the departments statistically significant differences were detected in indicators in favor of scholars who earned their PhD in the US. Less significant differentiations were evident in the other pairs: As far as the GR-EU pair is concerned, in only 12/50 of the departments were found significant differences in all indicators in favor of the scholars who obtained their PhD in a European country other than Greece. In the 40% of the examined departments a statistically significant difference was found at least to one index (publications, citations or h-index). As for the EU-USA pair, in only 20% of the examined departments a statistically significant difference at least in one indication was found.

RQ4. Correlation between academic rank, h-index and number of citations
The last research question was related to the investigation of the link between the members' academic rank and their h-index and number of citations. In Table 12 the data obtained for the departments of each academic discipline are presented. In general, higher correlation suggests better hiring practices, since a higher scientific production is required to achieve higher academic ranks.

Conclusions
The purpose of this study was to evaluate faculty members' research performance in departments of Sciences and Engineering in Greece using scientometric indices.
Using the Internet and the citation database Google Scholar as well Publish or Perish software (PoP, Harzing, 2010), indices such as publications (publications), references (citations), h-index, i10-index and m-index were collected. The process was quite efficient and accurate. In the majority of the evaluated departments, a significant difference in h-index was observed between academics who report scientific activity on the departments' website and those who do not. Moreover, academics who earned their PhD title in the USA have higher indices in comparison to scholars who obtained their PhD title in Europe or in Greece. Finally, the correlation between the academic rank and the scholars' h-index (or the number of their citations) in some cases is quite low in some departments which, in some cases, could indicate lack of meritocracy.
From the before mentioned discussion, it arises that one of the ways to assess the quality of Greek universities, is by focusing on the research output of faculty members. From the results obtained some useful conclusions were derived, which could contribute to the improvement of the departments. Moreover, such studies could inform elected officials and policy makers and better shape the public opinion. For instance, it would be advisable for potential university students to choose departments based on the reported level of research and not on criteria such as distance from their place of residence. Apparently, in many cases other parameters, such as socioeconomic status of each family, are involved. Moreover, rankings based on quantitative and widely accepted criteria would help to shape incentives for further research by all the Institutions. Appropriate interventions and policies could also aid the Universities to reach satisfactory scientific output.
The implementation of bibliometric evaluation in Greek Universities on an annual basis, will enable them to self-monitor the progress of the scientific output and the degree to which each University qualifies and meets tangible objectives laid down by the Greek Ministry of Education. Research for faculty members of academic departments, is one of their basic obligations as faculty members (apart from teaching and administrative tasks) and one could classify them into scientifically active or inactive. In this way, with the appropriate reform impetus an improvement of the current state of the Greek educational system could occur. More specifically, the evaluation indicators used in this work (h-index, m-index, publications, citations, i10index) could be taken into account, with new legislation, to further incorporate transparent and merit practices into Tertiary Education. For instance, in some U.S.
universities and disciplines science teachers are requested to have an index at least 12 to be promoted to the rank of associate professor and h equals 18 or higher enables their promotion to the rank of full professor (Lazaridis, 2008, p. 75). Similar policies in our country will provide greater research incentives to the faculty members to further improve the quality of their research work. In addition, transparent hiring practices based on tangible criteria adopted by hiring committees could motivate young people to pursue careers in academia.
However, the study is not without limitations. Our data are reliant on websites to determine the academic ranks of scholars. Therefore, we may have been inaccurate in assigning academic ranks to some of the academics in the present study. Also in some cases a synonymity occurred thus inflating the scholar's indices. In general, in only a few cases such a problem was occurred, since a lot of academics retain Google Scholar Profile. The latter is considered as accurate, which in turn in some cases could not be the case.
It is quite evident that the research questions answered in this paper, cover only a fraction of the possibilities provided by the bibliometric evaluation and statistical analysis of research data of the faculty. Numerous research questions and evaluation indicators of academic performance can be thoroughly studied (Mingers, & Leydesdorff, 2015). Our research was mainly focused to indices such as h-index, publications and citations. In addition, questions related to the area in which the scholars' PhD is obtained and possible relation to their output, as well as tendencies such as maintenance of GS profile and reporting scientific activity on the departments' web site, were also investigated. Other issues such as academic inbreeding (Inanc & Tuncer, 2011), relation between state funding and performance, number and characteristics of doctorate students and their scientific output as well as relation between scholars' gender and salaries and performance could be closely monitored and explored. A useful extension of this work is to incorporate representative international departments in order to better monitor the scientific progress and examine whether it is calibrated to an international level.
Finally, in order to publicly provide more reliable and representative results data should be collected by all Greek Universities and departments, preferably by using a suitable, usable and accessible web application (Katsanos, Tselios, Tsakoumis & Avouris, 2012, Orfanou, Tselios & Katsanos, 2015. Thus, the data related to scientific output could be instantly available to any stakeholder without the need to further process them and gives the ability to inform decisions. Therefore, a more organized, comprehensive and official effort to evaluate all the departments of the universities in the country, could greatly assist improvement of Greek Tertiary Education.