International University Ranking Systems and Their Relevance for the Medical and Health Sciences – A Scoping Review

,


The Role of International University Ranking Systems and the Importance of the Disciplines of Medical and Health Sciences
International university ranking systems aim to foster the improvement of academic institutions through a "virtuous competition" and are a common reference tool used by various stakeholders in the environment of universities.Their results provide relevant information for policy-and decision-makers on a national and international level in regards to financial and political support (Fauzi et al., 2020).Even though international university ranking systems all share the same objective-to independently and transparently rank higher education institutions on a global scale based on their academic performances-they often have different scopes and indicators with distinct weight-attributions.In addition, these university ranking systems are based on various criteria and data(bases), are developed in different countries with diverse cultural and socioeconomic settings, and therefore often lead to different outcomes when ranking an academic institution (Kayyali, 2020).According to Dill and Soo (2005), the globalization of the higher education system is one of the main contributing factors to the success of rankings.Students and their parents are motivated to choose the best university to proceed in their studies.Thus, university ranking systems find an attentive audience by providing easily interpretable information (Dill & Soo, 2005).In addition, Aguillo et al. (2010) highlighted the scientific context underlying the rationale for ranking higher education institutions, including the need for universities to recruit doctoral students, faculty members, and researchers from around the world to expand their international collaboration.These two factors are strongly influenced by the universities' objectives of finding and justifying funding to ensure their financial stability (Aguillo et al., 2010).Butler (2010)  much aware that national policy and multimillion-pound decisions are influenced by these rankings" (p.16).For all these reasons, ranking systems endure over time and are an integral part of the landscape of universities-on a national and international level.
Originally, university ranking systems were created to rank higher education institutions as one institution.With increased access, growth, and diversification of higher education institutions, this approach is no longer covering the whole range of modern academia.Accordingly, the demand for discipline-specific ranking systems has increased (Fauzi et al., 2020).This is also true for the fields of medical and health sciences.According to the Oxford language dictionary Lexico (2022), medical science is defined as "the branch of science concerned with the study of the diagnosis, treatment, and prevention of disease".Health sciences comprise various disciplines and are defined by the journal Nature (2022) as: "Health sciences study all aspects of health, disease and healthcare".In light of recent pandemics, the fields of medical and health sciences are inseparably linked.Furthermore, they have grown and expanded in their relevance as autonomous disciplines as well as one integrated discipline (World Health Organization [WHO], 2022).The increased prominence and relevance of medical and health sciences is also confirmed by Professor Michelle A. Williams, the dean of the Faculty Harvard T.H. Chan School of Public Health (Boston, United States of America).In her graduation ceremony speech in 2021, she highlighted: "The world is realizing that public health is just that-public.That it's everyone's business" (Roeder, 2021).
Consequently, it is important that medical and health sciences receive the academic, financial, and political support needed to further grow and expand.To be able to continue to strive for academic excellence within these disciplines, international university ranking systems are crucial and should address discipline-specific ranking criteria and indicators for the disciplines of medical and health sciences.This is where this scoping review finds its relevance.It is unclear how many existing international university ranking systems address medical and health sciences as specific disciplines; have accordingly adapted indicators, criteria, and weighting of indicators; or if there even exists an international university ranking system created uniquely for medical and health sciences.Therefore, the research question of this scoping review is the following: Are international university ranking systems focused on and adapted to the disciplines of medical and health sciences?

History and overview of international university ranking systems
The first attempts to rank higher education institutions on a national level were introduced in the 1870s in the United States.However, it was not until the 1980s that university ranking systems were developed in a systematic manner.
At that time, a first trial to rank universities was introduced by the US News & World Report's Best Colleges and gained much prominence.Subsequently, many more national ranking systems were developed (Doğan, 2018).China launched the first international university ranking system in 2003, the Shanghai Academic Ranking of World Universities (ARWU).One year later, the Times Higher Education-QS Ranking was developed.A number of international university ranking systems followed (International Ranking Expert Group [IREG], 2022).
As a response to the numerous rankings which have emerged since 2003, the IREG was established by the UNESCO European Center for Higher Education (in Bucharest, Romania) and the Institute for Higher Education Policy (in Washington, USA) in 2004.In 2016, the IREG developed the Berlin Principles on Ranking of Higher Education Institutions.These 16 principles represent "a set of principles of quality and good practice in higher education institutions rankings" (IREG, 2022).These principles highlight the importance of the methodological aspect of existing international university ranking systems, especially in regards to the weight-attribution of the different indicators and criteria on which the rankings are based.By establishing those principles, the IREG also addressed the challenge that different ranking systems can lead to different outcomes when ranking the same academic institutions (IREG, 2022).We created a table of the 10 most relevant international ranking systems.The aim of this table is to establish whether and how existing international university ranking systems address the disciplines of medical and health sciences, with what methodology, which indicators and weighting of those indicators and whether the methodology is specifically adapted to the field of medical and health sciences.We selected the rankings included in this table based on the relevance established through the screening of the scientific literature for this review.

Published by Sciedu Press
Because the fields of medical and health sciences encompass various different disciplines and because the authors work in the field of public health, we chose the discipline of public health as an example to elaborate how existing international university ranking systems may or may not adapt their methodologies to a specific discipline within the field of medical and health sciences.Through the selected examples of international university ranking systems, it could be noted that only a few of the existing global university ranking systems change the weighting of their indicators in regards to a specific discipline (for example in the case of public health: the ARWU).None of them adapt their indicators specifically to the discipline of public health.This highlights the lack of discipline-specific rankings and weight adjustments of indicators for academic disciplines in the field of medical and health sciences.

Search Strategy
We conducted a scoping review of scientific literature to identify and analyze the existing international academic ranking systems for higher education institutions as well as their respective indicators.The scoping review was based on three different search rounds with distinct sets of keywords.We kept the first set broad to search for existing international university institutional ranking systems.We narrowed down the second search round by adding to the existing set of keywords the term "medical schools" and the third round by adding the terms "health sciences" and "medical sciences."We used the same sets of keywords for each round for both databases.The keywords can be found in Appendix A.
For Web of Science, we screened all the articles identified with each set of keywords (a total of 485 articles).For Google Scholar, we limited the screening to the first 1,000 articles for each set of keywords (a total of 3,000 articles).The initial screening was based on the title (and in case of doubt, on the abstract) of each scientific article, which had to include or address international university ranking systems.

Inclusion and Exclusion Criteria
A broad scope of scientific literature (scientific article, book chapter, conference summary) was considered for this review.The literature had to be written in English and be published within the past 3 years (2019-2021, including the start of 2022 because this review was conducted during the months of February and March 2022).They had to address the subject of international academic ranking systems (either in general or specifically addressing medical or health sciences).Therefore, we included any scientific article addressing existing international university ranking systems, their indicators or articles proposing a new international university ranking system, or possible new indicators.National university ranking systems were not considered (a minimum of two countries had to be addressed by the scientific investigation).We only confirmed the final time-frame criteria after the first screening because it was initially unknown whether the different sets of keywords would provide sufficient results or not.
For the screening, we applied the following exclusion criteria: literature not published in the past 3 years, articles not written in English, articles that did not address the topic of academic ranking systems, or articles that only focused on national ranking systems.Additionally, we excluded literature solely assessing the personal academic ranking of members of universities and not the performance of the universities themselves or literature only addressing the ranking of medical departments in the university hospitals and not on the universities.Various websites of the best-known international university ranking systems were not part of the review itself, but we still analyzed them to extract data on their methodology to deepen the understanding of existing international university ranking systems.A table summarizing the inclusion and exclusion criteria can be found in Appendix B.

Data Extraction
For the data extraction and the removal of duplicates, we used EndNote 20 software.After removing all the duplicates and applying the time-frame criteria, we screened the remaining articles based on title and abstract by applying the inclusion and exclusion criteria to assess the eligibility for full-text reading.In a second step, we read the included articles and extracted relevant information (using an Excel software, 2022 version).Details regarding the extracted information of the included articles can be found in Appendix C. Further information can be found in the Prisma chart (Figure 1).

Prisma Chart
Figure 1.Scoping review Prisma chart

Results
We reviewed and included a total of 55 scientific articles, book chapters, or conference summaries for full-text reading and data extraction.

Background (Problems Addressed)
We created six different categories of "problems addressed" and classified articles per those categories (Table 2).
The most addressed category was problems in relation to the methodology of existing international university ranking systems (addressed 38 times).Most prominent in this regard were "Challenges associated with indicators/criteria" (addressed 12 times), followed by "Divergences in methodologies of different ranking systems" (addressed seven times) and "Challenges associated with weighting of indicators/criteria" (addressed six times).The "Lack of inclusion of certain indicators" and the "Lack of analysis of methodologies of international university ranking systems" (both addressed five times) as well as the "Lack of transparency/validity/coverage" (addressed three times) completed this category.
The categories "Universities" and "Socio-economic environment/Political influence" were both addressed 16 times.
In the "Universities" category, the "Influence of ranking systems on university policy" was the most prominent (addressed seven times).The "Overflow of/influence on scientific output," the "Reinforcing reputational factors or prestige," and "Publication-purchase/Data manipulation" were each addressed three times.In regards to the "Socio-economic environment/Political influence" category, for instance, the "Influence on policy makers" was addressed seven times and the "Influence on funding" was only addressed three times.
General challenges of university ranking systems (pros and cons, what they actually measure, etc.) were addressed a total of nine times (most of them addressed general challenges except one article that addressed the overflow of existing international university ranking systems).
The Western focus of existing international university ranking systems was also an important topic (addressed eight times).The focus in this category was on the lack of adaptation of existing ranking systems to non-Western contexts (addressed five times) as well as on the favouritism of scientific output written in English and not in other languages (addressed three times).Challenges with digital technology 1 20 Figure 2. Background (problems addressed) in percentage.

Objectives (Types of Scientific Investigations)
We created five different categories for "Objectives (types of scientific investigations)."The included literature could again be matched with multiple categories.
In total, 10 articles (or book chapters/conference summaries) proposed a new ranking system, nine proposed a new method or approach to analyze or improve existing international university ranking systems, three proposed new indicator(s), and two proposed a new database.Almost all the documents included an analysis or a review, and around half of them were solely an analysis or a review (which means they did not propose any new indicator, database, methodology, or ranking system).
In regards to the new ranking systems proposed, only one of them was aggregated from existing university ranking systems (the other nine were developed de novo).Weight attribution and indicators also played a major role when Published by Sciedu Press 114 ISSN 1927-6044 E-ISSN 1927-6052 developing new ranking systems (each of them was addressed three times).Less prominent was the focus on challenges associated with research output (this was addressed twice) and the development of international university ranking systems to non-Western contexts (addressed only once).
In the category "Proposal new method/approach to analyze/improve rankings," the focus was mostly on the databases, indicators, and algorithms (which were addressed six times).The role of sustainability was addressed twice, and the goal to improve rankings in non-Western countries was only addressed once.
The most common type of scientific investigation was focusing on methodologies, indicators, or data sources of existing international university ranking systems (addressed 15 times).Reviews of challenges were also addressed often (nine times), as were analyses on the impact of scientometric, bibliometric, and webometric indexes on scientific progression and analyses of specific indicators (both addressed seven times).The impact of existing university ranking systems on policy makers was addressed seven times.Less often, articles about the interaction of existing ranking systems (addressed four times) and analyses about the research output in non-Western countries (addressed three times) were found.The impact of languages of websites and social media of universities on international university ranking systems and the improvement of credibility of the ranking processes analyzed were only addressed once.Proposal new database Total (n=2) 1, 30 Figure 3. Objectives (types of scientific investigations) in percentage.

Innovation Factor
Ten new international university ranking systems were proposed in total for that category.Approximately half of those new ranking systems were based solely on bibliometric indicators.Many of these new ranking systems are based on preexisting rankings, either by integrating part of their indicators or by aggregating preexisting ranking systems (details can be found in Appendix C).In general, the weight attribution of indicators and the development of new indicators were most commonly the basis for the development of new international university ranking systems.Yet, only in three out of the 55 papers were discipline-specific indicators found.

General Information Extracted from the Included Literature
The three most cited existing international university ranking systems are the ARWU, the THE, and the QS.All of the included papers are based on an international scope (which means the document addressed at least two countries).However, some articles focused specifically on the Arabic world, Eastern Europe, or Southeast Asia.In regards to the authors of the articles, they came from all over the world (details can be found in Appendix C).

Discipline-Specific Focus
The vast majority of the included literature was not specifically addressing medical or health disciplines.However, three exceptions can be highlighted.One article has the aim to evaluate the h-index to identify young medical researchers with future potential in research, using an example of a medical faculty in South Africa.A second article has the objective to perform bibliographic comparisons among three databases and addresses the medical faculty in a Published by Sciedu Press 116 ISSN 1927-6044 E-ISSN 1927-6052 way that they recognize health sciences as an individual research discipline within the search engine.The last article highlights that the capacity to produce good research seems to be concentrated partially in the field of medical sciences in Southeast Asian countries.No included document was particularly focused on medical schools.Furthermore, no international university ranking system specifically designed for medical schools or health sciences could be identified.

Discussion
In this scoping review we investigated the role of the disciplines of medical and health sciences in existing international university ranking systems.The results highlight that there does not exist an international university ranking system specifically adapted to or designed for the disciplines of medical and health sciences, or at least not found per the methodology used in this research.Furthermore, existing international university ranking systems lack discipline-specific adaptations to the field of medical and health sciences.
To address this lack of discipline-specific university ranking systems in the field of medical and health sciences, we have identified three main barriers: 1. Methodological challenges and shortcomings of existing international university ranking systems, 2. The influence of the ranking systems on university policy and scientific output, and 3.
The lack of inclusion of the socio-economic environment into the development of international university ranking systems.

Methodological Challenges and Shortcomings
In this scoping review we have identified methodological shortcomings and challenges of existing international university ranking systems as the main subject addressed in scientific research in this area as well as the main burden in the development of discipline-specific international university ranking systems.The identified methodological challenges and shortcomings include problems associated with indicators, the divergences in the methodologies of various university ranking systems, as well as the lack of transparency of those methodologies applied.This is also confirmed by many researchers, including Qureshi and Daud (2021), who highlighted for example the controversial methodologies.
First, we have identified challenges associated with the weighting of indicators as a repeated criticism.This is also confirmed by other researchers, such as Stoupas et al. (2021), who indicated that the choice and weighting of indicators often occur on an arbitrary basis, and that, as a consequence, university ranking systems tend to "compare apples-to-oranges" (p.244).Subsequently, there exists an affiliated risk that these university ranking systems are diverted from their primary purpose and used by the general public without in-depth knowledge to compare incomparable factors.
The lack of inclusion of certain indicators in existing international university ranking systems is the second main methodological challenge identified by this review.For example, online education, which has grown in its importance tremendously since the COVID-19 pandemic, is only mentioned once.The same applies to sustainability.Even though there is a whole ranking system dedicated to sustainability (the UI GreenMetric Ranking), its resonance remains rather low compared to other existing ranking systems.As highlighted by Muñoz-Suárez et al. (2020), universities "must integrate the sustainable development concept" because this topic currently represents a significant weakness of many universities (p.2).According to Holmes (2021), the inclusion of "third mission indicators" in ranking systems, indicators which include basically anything other than teaching and research, is a current discussion (Holmes, 2021, p. 134).These few examples underline the importance of having university ranking systems that do not only include bibliometric or research-focused indicators but also criteria that assess other factors such as sustainability, gender equality, or informatic tools for online education, especially for the development of discipline-specific rankings.

International University Ranking Systems as Policy Guides for Universities and Impact on Scientific Output
The risk that university ranking systems, instead of being consulted as information resources, become policy guides for universities or have an impact on the scientific output is another of the main barriers to developing discipline-specific university ranking systems.As quoted by Nassa and Arora (2021) (citing Taylor et al., 2007), "In any case, it [ranking systems] should not dictate university policy, either at a national or institutional level" (p.5).
According to Holmes (2021), ranking systems are not only increasing in their number, but they are also becoming more sophisticated, now often including standardization and field normalization (Holmes, 2021).This point underlines once again the need for all the stakeholders in the environment of universities to have competent knowledge of the existing international university ranking systems and their methodologies to be able to use them effectively and to avoid unwanted impact on university policy or research output.Accordingly, some attempts at  (Holmes, 2021).Our results showed that these issues are identified but not sufficiently addressed.Therefore, existing ranking systems and their methodologies are reinforced.This is a barrier to the development of discipline-specific international university ranking systems.

Lack of Inclusion of Socio-Economic Environment
The third barrier to the development of discipline-specific university ranking systems is the lack of inclusion of the socio-economic and geopolitical environment of different countries in existing international university ranking systems.Many of the existing international university ranking systems are developed in high-income countries.Therefore, there is always an inherent bias within the structure of those rankings, which are mainly focused on the socio-economic and cultural context of the Western world.The question as to how to overcome this bias bears many challenges.Not only do existing international university ranking systems need to be adapted to non-Western and low-and middle-income contexts, but their structures, methodology, indicators, databases, and application of criteria should be adapted to a more international and diverse setting that resonates with the different contexts and countries.

The Example of Public Health
Only when university ranking systems are created specifically for a certain discipline or address them in a specific manner can indicators be refined and address those challenges in a more comprehensive approach.This was rarely addressed by the literature included in this scoping review.As we have seen in Table 1, there is no such thing as a discipline-specific ranking.If we once again look at the example of the discipline of public health, we see that some rankings do rank public health schools (ARWU, NTU, SCimago, CWUR, US News Rankings), but with criteria and weightings common to other disciplines.We believe that this is a major weakness of most of the existing international university ranking systems.It is true that creating discipline-specific rankings would take considerable resources, but we believe that this is the only way to obtain a robust ranking: ranking systems should definitely be adapted on a discipline-by-discipline basis.To illustrate this, let us again take the example of the Shanghai Ranking's Global Ranking of Academic Subjects 2021 for schools of public health.This ranking takes into account five indicators with different weightings (Q1 -100%, CNCI -100%, Top -100%, IC -20%, Award -0%).It seems appropriate that the Award indicator should not be given any weight because it is rare for a public health researcher to win a Fields Medal or a Nobel Prize, given these categories are only remotely related to the potential subject of research in this field.Nevertheless, what should we think of the weighting of the international collaboration indicator?Is international collaboration not of paramount importance in public health and relatively underestimated by a weighting of 20%?Why does this ranking not include parameters concerning sustainability or gender equality, given the importance of this subject in the discipline of public health?Why are bibliometric indicators, which can be developed in great detail in certain rankings, summarized globally by only three indicators in a domain where academic research is primordial?These are some of the questions we can raise in regards to this university ranking system applied to the public health domain.

Link to the Disciplines of Medical and Health Sciences
The ongoing COVD-19 pandemic has highlighted the importance of the interdisciplinary and transnational exchange between different actors of the medical and health field.In that sense, quality education is a requisite element not only for pandemic prevention, preparedness, and response, but also in the general view of population health.The medical and health sciences disciplines play a prominent role in our societies, whether from an individual point of view or from a global point of view.This bears a challenge to achieve the "highest attainable standard of health" as defined by the WHO constitution (1946), which is "one of the fundamental rights of every human being without distinction of race, religion, political belief, economic or social condition" (p. 1).Furthermore, these disciplines represent an absolute necessity for the future of humanity, whether it be at the human, animal, or global level.As defined by the U.S. Centers for Disease Control and Prevention, "One Health is an approach that recognizes that the health of people is closely connected to the health of animals and our shared environment" (2022).Supporting academic excellence and research in the different disciplines that approach this topic is therefore essential.To promote and encourage the development of medical and health sciences as leading disciplines, the development of a specific ranking for schools of medical and health sciences is mandatory.

Strengths and Limitations of This Review and Future Perspectives
According to our results, no recent articles about the ranking of medical or health schools could be found.This review therefore highlights the need to develop a specific university ranking system designed for or adapted to medical and health sciences.Furthermore, to our knowledge this study appears to be the first to investigate the need for adapted university ranking systems in this field.Because many stakeholders in the landscape of academia rely on global university ranking systems, the quality assessment of higher education institutions in the fields of medical and health sciences is of great importance.We believe that the methodology used in this research avoided major biases in the analysis conducted.
This scoping review has several limitations.First, all the articles included were written in English.Therefore, some relevant publications in other languages may not have been included.However, this choice was made to improve coherence within the search strategy.A second limitation could be the choice of only two databases (Web of Science and Google Scholar).These databases are developed by high-income countries and focus mainly on scientific output produced in English.Still, owing to the limitations of resources and the scope of this review, the literature research had to be limited to the two search engines mentioned (which still cover a wide range of diverse scientific research output).Another limitation is that this research only focused on international ranking systems because one could argue that a national ranking system for medical and health sciences could have been developed in a specific country and that this could have been missed by this scoping review.Furthermore, the literature search was only limited to the past 3 years, which could exclude certain possibly relevant scientific articles published before 2019.

Conclusion
This scoping review has highlighted the existing lack of an international university ranking system specifically designed for higher education schools of medical and health sciences.Future researchers could investigate how to develop indicators (and their weighting) specific to the field of medical and health sciences and thus promote a university ranking system dedicated to these disciplines.

Methodology
of transparency/validity/coverage Analysis -Of existing global university ranking systems and 1.No 2. N/A*

Table 1 .
Example of existing international academic ranking systems and their application to medical and health sciences 104 ISSN 1927-6044 E-ISSN 1927-6052

to the product Π of their Eigenfactor and Article Influence Score, from
largest to smallest.A list Li of journals with non-zero Π can then be obtained.

. First year 2. Frequency 3. Country Indicators & weighting 1. Addresses medical schools (Yes/No) 2. Addresses public health (PH) schools (Yes/No) 3. Indicators & weighting (for ranking addressing PH schools)
If Ni is the total number of articles in Li listed in the most recent edition of JCR, the journals chosen are the ones with the highest Π in Li and containing between them 0.25 Ni articles in total.Repeating this algorithm, we obtain 227 lists Ji of the top journals for all Subject Categories in JCR." -"Using the Science Citation Index Expanded and the Social Sciences Citation Index databases, an

institution's pre-final score in a given Subject Category SCi is given by log (ni + 1), where ni is the institution's number of "Article" publications in Ji during the last 10 full years.
Pre-final scores are then scaled to the top performing institution to arrive at final scores."

US News Best Global Universities Rankings
(US News Best Global Universities Rankings, 2022) Best Global Universities for Public, Environmental and Occupational Health" (subjects: epidemiology, hygiene and health; parasitic diseases and parasitology; tropical medicine; industrial medicine; occupational medicine; infection control; and preventive medicine, resources on environmental health, cancer causes and control, aviation, aerosol, and wilderness medicine) -"Best Global Universities for Social Sciences and Public Health" (subjects: social policy, political science, education, demographics, law, public health and administration, ethics, social aspects of health, and addiction) 1. Yes: "Best Global Universities for Clinical Medicine" 2. Yes: two specific rankings: -"

Number of publications that are among the 10% most cited - 15% 9. Percentage of total publications that are among the 10% most cited
-7.5% 10.

Number of highly cited papers that are among the top 1% most cited in their respective field
-7.5% 11.

Percentage of total publications that are among the top 15 Published
by Sciedu Press 109 ISSN 1927-6044 E-ISSN 1927-6052

Number of highly cited papers that are among the top 1% most cited in their respective field -5% 11. Percentage of total publications that are among the top 15 most highly cited papers
-5% 12. International collaboration (relative to country) -5% 13.International collaboration -5% *Note regarding abbreviation: "N/A" means "not applicable."

Table 2 .
Background (problems addressed) addressed by the articles included in the scoping review.

Table 3 .
Objectives (type of scientific investigation) addressed by the articles included in the scoping review.

types of scientific investigations) N° Articles Article reference number Proposal new method/approach to analyze/improve rankings Total (n=9)
external validation have been made because several rankings are now audited (for example, the THE Ranking is audited by the accounting firm PricewaterhouseCoopers) Like many authors,Stoupas et al. (2021)also noted that existing ranking systems promote universities of English-speaking countries.Selten et al. (2020)demonstrated the same result: English-speaking universities have an advantage on the research performance scale.This was also pointed out byKassim et al. (2020), who explained why a new ranking specialized to Asian countries (called ASIANur) should be developed, highlighting three preexisting economic, demographic, and academic strengths in Asia.This last example shows that efforts are being made to counter this inherent flaw.However, it is important to note that following these various criticisms, several teams have tried to propose innovative solutions.For example,Kudela (2021)proposed a new methodology that allows universities to choose the respective weighting of each indicator to optimize their own rankings.Another team based in Jordan developed a database focused on the Arab-context (called eMarefa database), together with a new indicator called Arcif (Al-Shorbaji, 2020).