Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations

We analyzed how often and in what ways the Journal Impact Factor (JIF) is currently used in review, promotion, and tenure (RPT) documents of a representative sample of universities from the United States and Canada. 40% of research-intensive institutions and 18% of master’s institutions mentioned the JIF, or closely related terms. Of the institutions that mentioned the JIF, 87% supported its use in at least one of their RPT documents, 13% expressed caution about its use, and none heavily criticized it or prohibited its use. Furthermore, 63% of institutions that mentioned the JIF associated the metric with quality, 40% with impact, importance, or significance, and 20% with prestige, reputation, or status. We conclude that use of the JIF is encouraged in RPT evaluations, especially at research-intensive universities, and that there is work to be done to avoid the potential misuse of metrics like the JIF.

10.2 Reappointment, Tenure, andPromotion Handbook For Candidates (2016-2017 (2009) provide an estimation of this merit using the following criteria: relative contribution of the faculty member for coauthored manuscripts (e.g., a percentage value representing overall proportional contribution to the finished product); quality of the journal (both common knowledge and weighting factors, such as impact indices); and the impact of the contribution to the discipline.
" Sentiment: Supportive Claims to measure: Quality " Three levels of scholarly contribution quality are recognized: A, B, and C. These three levels provide a means by which the quality of contributions can be ranked in addition to quantity (e.g., publication rate and total number of contributions) in assigning annual review scores. In multiauthored endeavors, the faculty member should state the significance of his/her contribution to the project. For all items, the faculty member should state the appropriate level (see below) with a brief explanation of this choice (e.g., impact factor of journal, refereed publication with an undergraduate, etc.) " Sentiment: Supportive Claims to measure: Quality 2 Boise State University 2.1 Guidelines for Promotion and Tenure (2017), Department of Psychological Science, College of Arts and Sciences " Although quantity of scholarship is certainly important, quality is also important. In addition to meeting quantitative requirements in the area of scholarship, successful candidates for tenure and promotion must make a compelling case for excellence in scholarship. Examples of evidence related to quality include the type of outlet (e.g., quality of journals), impact on the field (e.g., citations and journal impact), and/or impact on the greater community for technical reports (e.g., audience of the outlet, such as a small private business versus a major Government agency).
" Sentiment: Supportive Claims to measure: Quality and impact/importance/significance 2.2 Tenure and Promotion Policy (2017), College of Arts and Sciences " If a department employs specific metrics of impact or quality for the purpose of evaluation relative to tenure and promotion (e.g., H-index or critical reviews), the processes and standards shall be described in departmental policy. year over the five years preceding promotion. The length of articles and scientific significance, as measured by citations and journal impact, may also be considered, as will authorship on contributions to other scholarly works (e.g., reference and text books). " Sentiment: Supportive Claims to measure: Impact/importance/significance 4.2 Tenure and Promotion Guidelines (2015), Institute of Environmental Sustainability " Evaluation of a candidate's research will consider the quality of the work, as well as the quantity of published products and the degree to which the candidates' research is original and advances the field. Web-based reference services, such as The Web of Science, will be used to quantify how many times the candidate's publications have been cited by others, and the quality of the journals will be assessed by their average impact factors. " Candidates should have at least four manuscripts in peer-reviewed journals published or in-press in the five years preceding application for tenure and promotion to Associate Professor. The length of articles and scientific significance, as measured by citations and journal impact factor, will also be considered, as will authorship on contributions to other scholarly works (e.g., reference and text books).
" Sentiment: Supportive Claims to measure: Impact/importance/significance " For promotion to Professor, the candidate must have an average publication rate of at least one article per year published in peer-reviewed journals in the five-year period preceding the application for promotion. These articles should be regularly cited by other researchers in the field. We will consider both the quality of the journal (as measured by the journal's impact factor, or JIF) as well as the number of citations of each publication. We will employ the metric: Article Impact Factor (AIF) = (JIF * citations) where "citations" represents the number of citations for the particular publication. Employing this metric, faculty have incentive to publish in the highest quality journals (which will increase the JIF) and simultaneously produce the highest quality research manuscripts, potentially increasing the number of citations, and increasing the AIF. " • Encourage to submit research to peer-review process as soon as possible.
-Time lags for submission, editorial responses, acceptance and publication.
• Impact factors and rejection rates . " The Russ College of Engineering and Technology has established metrics for the evaluation of departments, programs and faculty, and for establishing college workload policy. These serve as guidelines for those making promotion and tenure decisions in the college but candidates must recognize that other factors are taken into account as well.
For example, the quality and impact of publications must be considered as well as the quantity. Specifically, the quality/reputation of a given journal or refereed conference (e.g., impact factor) will be considered along with impact of a given paper (e.g., citation count).
" Sentiment: Supportive Claims to measure: Quality and prestige/reputation/status " List relevant peer-reviewed journal and conference papers published over the last five years (or since last promotion or initial appointment, whichever is less) related to pedagogy or other relevant areas of education. Include the journal's impact factor (or equivalent journal ranking data) and the number of citations of the article(s). " Quantity is neither the primary nor the driving factor in assessing scholarly/creative activity. Schools and departments may choose to identify various quantitative markers, and it is a combination of quantity and quality of scholarship that is ultimately assessed in evaluating the candidate. The quality of the work, influence the work produces, and the level of contribution to a particular body of knowledge are the substantive issues that will be reviewed for promotion and/or tenure. Markers of quality of publications may include impact factors of journals, number of citations of published work, and audience of journal. " Publications and quality of journals: For excellence in research, candidates should demonstrate their ability to publish their work in reputable journals, scholarly anthologies or collections, and conference proceedings. Refereed publications, and/or peer-reviewed journals, with a high impact factor carry special weight. For books and monographs, the quality of the press is important. Similarly, creative activities at recognized venues are highly regarded.
" Sentiment: Supportive Claims to measure: Quality and prestige/reputation/status " Some suggested subheadings for appendices include: ... Impact Factor and Citation Reports. Include citation of your work, (e.g. by other authors or speakers), impact factor and acceptance rate of journals your work appears.  (2014), Department of Anthropology " The quality of a candidate's research will be an important component of the assessment of scholarly contribution. Members of the primary committee have the responsibility to examine the publications of candidates when making judgments of quality. In addition, the quality of the journals as indicated by such things as reputation, editorial board members, impact factors, and rejection rates will be considered in judgments of quality.
The department expects that some of the candidate's publications will have appeared in major influential journals in anthropology or in the candidate's area of specialization. " The candidate should publish at least 3 peer-reviewed publications each year, or less than 3 but in high impact journals, prior to promotion. The candidate for Full Professor should show the continuity of a strong publication record, at least 3-4 peer-reviewed publications per year in high impact journals, since the candidate's promotion to the associate professor rank.
" Sentiment: Supportive Claims to measure: Unspecified " While quoting the Scientific Citation Index of each paper is not mandatory, publications in high impact journals covering the candidate's own research field is expected.
Candidates are encouraged to provide information for evaluating the quality of the journals in which their work is published. This may include impact factors (relative to other publications in the field), rejection rates, editorial board membership, and overall ranking of the journal in the field. This information will be combined with the judgments of outside reviewers in evaluating the quality of the candidate's research scholarship. " " Sentiment: Supportive Claims to measure: Impact/importance/significance " It is expected that the candidate will be productive in terms of generating refereed journal articles from his or her ongoing independent research program. No specified number of publications is required, but rather, emphasis will be placed on the quality of the publications, and impact on the field. Factors used to assess quality may include such measures as citations of publications, evaluations of the work by external referees, and the impact factor of the journals in which they are published. " For promotion to Full Professor, the candidate will need to present evidence of an international reputation in his or her field. This will normally be assessed by establishing that there is a significant body of research that has been published in peer reviewed journals. In addition to evaluations solicited from external referees, the quality of this research may be assessed using the citation figures for publications and by examining the impact factor of the journals involved.  (2007), Faculty of Medicine and Dentistry " Performance will be judged to be "Superior" when it exceeds expectations. There will be evidence of: • a substantial record of research productivity with either a landmark paper in a prestigious international journal or multiple papers in high-impact journals that make a significant impact on the field " Sentiment: Supportive Claims to measure: Impact/importance/significance " Examples of superior administrative contributions can be related to research or education and include contributions to the discipline such as serving as Chair of a grants panel or site visit, serving as editor of a high-impact journal, organizing a major national or international conference, or serving as president of a professional organization; recognition by receipt of a service award from a professional society would be considered meritorious.
" Sentiment: Supportive Claims to measure: Unspecified " Performance will be judged to be "Superior" when it exceeds expectations. There will be evidence of: . . .
• exceptional administrative service such as serving as Chair or Deputy Chair of a grant review panel or editor of a high-impact journal " Sentiment: Supportive Claims to measure: Unspecified " Criteria for superior performance in the research/scholarship category represent guidelines, not mandatory requirements (see C. Performance Standards). Superior performance will have been demonstrated if the candidate has published a substantial number of papers in good journals with some (or one landmark paper) making significant contributions to the field;...and/or has served as Chair of a grant review panel, editor of a high impact journal or organizer of a major national/international scientific or educational meeting. The CALL's concern that the candidate be "continuously and effectively engaged in creative activity of high quality and significance," should further be demonstrated through other publications that include peer reviewed articles in high impact journals, as well as evidence through submitted work in progress of new, or developing, research projects. The Committee realizes that there are fields in which the writing of peer reviewed articles may take precedence over the completion of a book, or book manuscript. In such cases the published work shall be judged for its creativity, originality, and high impact by the standards of that field, or sub-field, based on the careful evaluation of the candidate's published work by both departmental and external reviewers. " ...scholarship in the discipline relates to a range of clinical and research specialties; thus, publication outlets for these specialties include an array of highly respected journals focused in these areas, some of which may include a limited readership due to the narrow focus of the specialty area. Therefore, it is not possible to interpret such measures as impact factors for these journals in the same manner as it may be possible to interpret them via benchmarks in other fields. Rather, it is necessary to interpret impact and citation metrics within the context of the identified specialty areas, which have more narrowly defined readership and scholarly foci. In accordance with the Faculty Senate Resolution on this matter, "if citation indices are used relative to the faculty or the institution of the University of Central Florida, their limitations will be addressed. " Evidence of the impact of the candidate's work is more important than the sheer number of publications. Scholarly impact is indicated by various publication metrics and other recognitions. Publication metrics may include such examples as citation count, journal impact factor, journal rankings, journal acceptance rates, journal distribution (international), and the extent of full-text publication downloads. Works that are widely read and cited are generally more significant in promotion decisions than obscure, unread and uncited work. In addition, publication in respected, highly cited journals will be given more weight than publications in unranked journals. This will include the quality and number of peer-reviewed publications. A faculty member may want to point out the specific qualities of a paper by submitting additional material such as reviewer comments or other documentation. Additionally, the faculty member may provide a few sentences describing the significance of these papers and the original contribution that they represent (similar to NSERC F100).
Discussion of journal quality (by those familiar with the field) may be included in the assessment in addition to consideration of the quality of individual research contributions. However, citation analyses and impact factors are problematic indices, particularly in comparisons across fields, and their use in the review process is not encouraged. A record of a substantial number of original, peer-reviewed research papers in widely respected refereed journals, judged on the quality as well as the quantity of research publications, since the faculty member became an assistant professor. Typically 1-2 publications on average per year as first or senior author since the candidate became an assistant professor is expected, although consideration is also given to the type of research, the impact factor of the publications, and to faculty whose work is primarily part of team research. The evaluation of research should include an explanation of practices and conventions of publishing or exhibiting in the discipline that may not be clear to outsiders...In addition, information on the following points is very helpful: • The prestige and selectivity of the venues in which the candidate has published, performed, or exhibited. Impact factors or citation analyses may be included but are not required since such measures are not available in many disciplines. If acceptance rates are available for journals or presses, those should be included. " Productivity Index] and media coverage, particularly if in a professional periodical or a leading science journal (e.g., "News and Views" type of highlighting in a major journal). Impact measures such as citation data can also be useful for documenting impact, but a low citation index should not be used to disallow a promotion, especially if it can be argued that the research challenges a ruling paradigm.

University of North Carolina at Greensboro (continued)
22.3 General Evaluation Guidelines for Reappointment, Promotion and Tenure, and Post-Tenure Review (2014), Department of Nutrition, School of Health and Human Sciences " The successful candidate will demonstrate scholarly or creative contributions in a combination of the major areas listed below.
• Papers in refereed journals (also state impact factor, percent acceptance, and circulation volume of each journal) ...  " While ability and potential as a researcher cannot be inferred from quantitative criteria of publications alone, the following relative weights for types of publications are generally applicable: ... f. Consideration will be given to publication quality as measured by the following items (though not exclusive of other quality measures not listed here): journal/press rankings, journal/press reputation in the field, journal impact factors, journal acceptance rates, awards, citations, reviews and/or reprints. served as the basis for promotion to Associate Professor. However, our primary concern is with assessing the quality of the work. At this level we are looking for the kind of contribution that makes a notable positive impact on the field. Factors described in the accounts of quality assessment in I.B. are relevant. Evidence of scholarly impact and quality can also include favorable reviews of the book, references to the candidate's work by other scholars, and inclusion of the work in anthologies or textbooks. It should also include positive assessments of published work by respected scholars with expertise in the candidate's area from other universities.  (2015), College of Arts and Sciences " The evaluation of quality normally consists of an assessment of the originality, skill, and importance of the candidate's contribution to the field. Quality is assessed through unbiased, expert analysis and judgment (e.g., internal and external peer evaluation, book reviews, etc.), as well as inferred through indicators such as the prestige and reputation of publication outlets, contextualized impact factors, and the selectivity of the journals, presses, galleries, venues, etc. where the work appears. " Criterion 3: Evidence of independent and original contributions to research or creative activity, which have an impact on the field of expertise.
Associate Professor (Level A) Evidence of original contributions that are influencing the evolution of the field, practice, or thinking within the discipline or as practical applications. 2 30 University of Windsor (continued) 29.2 Sample Research Evaluation Rubric (2016) " Criterion 2: A record of high quality refereed publications, juried creative activity or other demonstrated scholarly outputs a) Publishes in journals or with publishing houses with a strong academic reputation 2 2 Departments may wish to provide quantitative metrics such as journal impact factors as an element of their standards. Factors such as low acceptance rates, high levels of readership, importance to the field are also suggestive indicators in assessing quality and reputation. Some departments have used an illustrative model to articulate publications that are at acceptable levels. Departments may also wish to identify the range of publications or other products that should be taken into account.

"
[Note: The first part of this quote (up to 'reputation') also appears in the Working Session on Developing Promotion and Tenure Criteria for Research (2016) from University of Windsor.] Sentiment: Supportive Claims to measure: Quality and prestige/reputation/status " Criterion 3: Evidence of independent and original contributions to research or creative activity, which have an impact on the field of expertise. a) Original contributions to the field of study or creative practice that influenced thinking and/or practice in the field, including extent to which research or creative activity is considered, referred to, read; citation in documents; impact factors, citation counts, publication rates, confidential external reviews of impact " [Note: A nearly identical quote also appears in the Working Session on Developing Promotion and Tenure Criteria for Research (2016)  " The Department acknowledges that journals and books vary in quality. At the same time, the Department recognizes that it is difficult to measure journals' or books' worth by using impact factors or other similar indicators. Generally, books published by university presses and journals linked to scholarly associations or published by well-recognized publishers (e.g. Taylor and Francis, Routledge, Sage, etc.) will be assigned greater value than other publications. Candidates will be encouraged to submit a statement that explains the importance of their publications, which may include factors such as journal impact factors, citation rates, publication in journals with low acceptance rates, high levels of readership, demonstrated importance to their field.
" Sentiment: Supportive Claims to measure: Impact/importance/significance " Publications and contribution are in well reputed and appropriate venues with significant impact in her research area. This may include community, local, national and international venues that are highly regarded and of significance to the field. It may include both contributions in traditional and non-traditional venues and both publicly engaged and peer reviewed contributions.
The extent to which the publications influence the field based on number of citations, citation rate, impact factor, confidential reviews, invitation as keynote speaker or peer reviewer. a. Papers published in, or accepted by, refereed archival publication venues (e.g., journals and highly selective conferences with archival proceedings). Papers must be accessible for reference. Comment on the standards of the publication venue, and provide a list of the top (not more than 10) publication venues in the candidate's research areas, identifying both broad and specialty venues. Include acceptance rates and/or impact factors where available (be sure to specify the research areas), and provide in an appendix a copy of the letter of acceptance if not yet published. Citation counts and/or electronic download counts may be included to document impact.