The Development of the Journal Evaluation Tool to Evaluate the Credibility of Publication Venues

INTRODUCTION A shared concern among librarians who work in an academic environment is finding effective mechanisms to help faculty identify suitable publication venues. Determining the suitability is now also complicated by the need to determine the credibility of the venue itself, to ensure that faculty select a venue that is held in esteem. DESCRIPTION OF PROJECT At Loyola Marymount University (LMU), a medium-sized, private institution in the United States, three librarians developed a tool to assist faculty in determining the credibility of a publication venue, specifically for open access journals. This article outlines the development of a tool to evaluate journals, the pilot testing process, and some of the measures taken for the promotion, outreach, and implementation of the tool. The goal of the tool is to inform publishing decisions using an objective measure of credibility and to empower authors to make publishing decisions for themselves. NEXT STEPS The authors have released the tool with a Creative Commons CC-BY license in order to enable the broad dissemination, use, and enhancement of it by anyone interested in using or developing the tool further. It will be valuable to understand the adapted use cases of the tool and learn about experiences from other librarians using this tool at their institutions.


INTRODUCTION
A shared concern among librarians in academic environments is finding effective mechanisms to help faculty identify suitable publication venues. Advances in technology have had a tremendous impact on the scholarly publishing landscape, further impacting how, where, and how quickly publishing occurs. Such advances have, however, brought with them some unethical publishing practices that make it difficult for librarians and faculty to determine the credibility of venues themselves to ensure that faculty select venues that are held in esteem.
At Loyola Marymount University (LMU), a medium-sized, private institution in the United States, three librarians developed a tool to assist faculty in determining the credibility of a publication venue. The Journal Evaluation Tool was originally developed with the intent of evaluating open access journals; however, it can be used to evaluate non-open access journals as well. This article discusses the work undertaken by these librarians at their institution to test and implement the Journal Evaluation Tool. It also examines various other freely available tools developed by other librarians and organizations to assist with evaluating journals. The authors also reflect on the limitations of the methods used to conduct the testing of the tool.

LITERATURE REVIEW
The scholarly communication landscape is ever-changing, and researchers have various options for publishing their scholarly works in the digital environment. While the open access (OA) movement began in the 1990s, the scholarly world is still seeking ways to embrace, adapt, and successfully implement the open access publishing model. Among some of the barriers to OA publishing, several studies have found credibility and the perception of quality of open access journals to be a primary concern among faculty (Reinsfelder, 2012;Rodriguez, 2014, Rowley et al., 2017. Thus, libraries and scholarly communities have developed various freely available tools to help researchers evaluate OA journals and publishers.

Credibility in Open Access Publishing
Exploitative practices in some open access publishing venues have stigmatized open access journals as predatory, questionable, or non-credible. Berger (2017) discusses the challenges of defining predatory publishing, as the subject is complex: "Understanding the detailed characteristics and practices of predatory publishing as well as the research on publishers, authors, and editors is critical to moving towards the praxis of educating users" (p. 206). The term non-credible will be used here to describe journals that contain one or more characteristic of this exploitative model. Characteristics may include lack of peer review or quality control, quick publishing turnaround, mimicking the name and/or website of an established journal, lack of transparency about author fees, and nonexistent or fake editorial boards (Beaubien & Eckard, 2014, Berger, 2017, DOAJ, 2018. At the core of predatory publishing is the combination of the negative characteristics listed above with the use of article processing charges (APCs) to generate revenue. Bjork, with colleagues, has studied the development of APCs in OA publishing and found that the number of publishers charging APCs has increased over time. APCs currently range from $8 to $3,900 (Laakso & Bjork, 2012, Solomon & Bjork, 2012. In Xia's (2015) study on non-credible journals and publishing charges, the author found that 72% of the journals analyzed had a fee scheme with charges ranging from $8 to $950 (p. 70). Researchers may pay for APCs either from their own pockets or through grant or institutional funding; thus, publishing in non-credible journals may have a financial impact for researchers. Further, publishing in suspect journals may also make it more difficult for institutions to justify the coverage of APCs on behalf of their researchers. As Berger and Cirasella (2015) indicate, charging a fee does not necessarily make a journal non-credible, since many reputable open access and subscription-based journals charge APCs; "however, predatory journals are primarily fee-collecting operations-they exist for that purpose and only incidentally publish articles, generally without rigorous peer review, despite claims to the contrary" (p. 132).
With questions and concerns regarding credibility, quality, and potential financial impact, both researchers and librarians have sought effective mechanisms to determine the credibility of an open access publication venue.

Solutions Developed in Academic Libraries
Libraries have devised various solutions to educate constituents on predatory publishing and the common characteristics of non-credible journals. Activities include creating blacklists, listing predatory characteristics on websites and LibGuides, forming indicators, and spearheading events and publications to generate awareness in our respective academic environments. While Beall's lists have been controversial among some researchers, they represent an early attempt to identify journals with questionable practices (Beall, 2015). Berger and Cirasella (2015) acknowledge Beall's contribution of "amassing considerable knowledge and greatly increasing awareness of predatory publishing" while also pointing out some concerns with Beall's lists, including lack of contextualizing low-quality publishing as something that predates OA and focusing overly on predatory journals rather than the broader scholarly communication environment (p. 133).
Beyond Beall, other librarians and scholarly institutions have put forth efforts to create awareness and prevent researchers from publishing in non-credible journals. Due to rising concerns and questions from faculty, Beaubien and Eckard (2014) developed a set of Open Access Journal Quality Indicators (rather than create a LibGuide with questions or criteria to help in evaluating journals or journal white or black lists) to identify high-quality open access publications (p. 4). The authors created negative and positive indicators that can be applied to open access journals in any discipline. The set of indicators encourages researchers to look for ethical and unethical practices in areas such as scope of audience, primary audience, reputation of editorial board, societal or institutional affiliations, and fees for publishing and being indexed in either an open access or traditional directory (Beaubien and Eckard, 2014, p. 5).
Megan Wacha and the CUNY Office of Library Services created an activity titled Is It Predatory? Checklist for Evaluating Journals based on Beaubien and Eckard's indicators (Wacha, n.d.). The activity in checklist form guides researchers to evaluate a specific journal at the levels of journal, content, editorial, and publisher. At each level, researchers are encouraged to check off either positive or negative indicators. Indicators range from journal scope to locating an article's DOI to reviewing the editorial board. The last step of the activity, "Is it Predatory? Yes, No, or Unclear," promotes autonomy, encouraging researchers to determine the quality of the journal.
The Duke University Medical Center Library and Archives (2017) created the Be iN-FORMEd: Checklist to help researchers assess the quality of a journal or publisher. iN-FORMEd provides a framework for a set of criteria: number, fee, ownership, review, membership, and editorial. For each criterion, the checklist provides signals indicating whether the journal is legitimate or potentially predatory.
Not all efforts can be mentioned and acknowledged, yet it is important to state that academic libraries across the United States have created resources and services such as Lib-Guides, websites, trainings, and events as effective mechanisms to help faculty identify suitable publication venues.

Solutions Developed in the Wider Scholarly Community
The Association of College & Research Libraries (ACRL) launched the Scholarly Communication Toolkit in 2005. According to Karen Williams (2005), the toolkit's creator, "A primary goal of the toolkit is to summarize key issues and content in order to give readers quick, basic information on scholarly communication topics" (p. 199). The toolkit underwent various iterations with massive revisions (as well as transition to LibGuides) in 2016. (ACRL, 2016). The toolkit provides a myriad of resources for administrators, faculty, and librarians on scholarly communication topics. Using the section "Evaluating Journals," li-brarians can locate a list of resources on evaluating OA journals that can be shared with their respective academic community.
In Principles of Transparency and Best Practice in Scholarly Publishing, four scholarly institutions collaborated to create a list of guiding principles of transparency and criteria used to evaluate potential Directory of Open Access Journals (DOAJ) members (DOAJ, 2018). The driving force of the principles and best practices is to verify the legitimacy of publications in order to curate, maintain, and develop a directory of reliable open access journals. There are a total of sixteen principles, which include evaluating the peer review process, author fees, revenue sources, and archiving.
The campaign and online checklist Think. Check. Submit (2019) was launched by multiple founding and contributing organizations, including the four founding institutions of the Principles of Transparency and Best Practice in Scholarly Publishing. The checklist aims to help researchers assess and identify credible journals in which to publish their research. At each level (think, check, submit), researchers are asked a series of questions to help them evaluate the quality of a journal. Think. Check. Submit. is interdisciplinary and international in scope, as it is available in over thirty languages.

Rationale for New Tool
In locating and evaluating these resources, the authors believed there was an opportunity for the development of another tool, specifically a resource that would facilitate the verification of the quality of a journal in the journal evaluation process. The Journal Evaluation Tool aims to build upon the existing resources, in particular the list of criteria provided by the DOAJ's Principles of Transparency and Best Practice in Scholarly Publishing, taking it one step further by allowing researchers to assign a numerical weight (score) to a journal. The Journal Evaluation Tool was purposefully created as a rubric to add clarification and standardization to the journal evaluation criteria by providing researchers with a comprehensive list of credibility criteria, a rationale, and the ability to place journals in a good, fair, or poor category. The Journal Evaluation Tool fills a gap in the current landscape of tools by giving researchers guidance and autonomy in the evaluation and selection of open access journals. Ultimately the decision about where to publish is up to the researcher, and this tool is designed to assist them by providing an objective measure of credibility.

DESCRIPTION OF PROJECT
In fall 2014, an associate dean in the College of Science and Engineering at Loyola Marymount University (LMU) approached the serials and electronic resources librarian seeking guidance in evaluating publication venues on behalf of that college's faculty. The associate dean expressed concern that faculty in his college might not be confident in their identification of non-credible journals, especially open access journals, and that the quality of publications from the college might suffer as a result.
The serials and electronic resources librarian convened a small working group, comprised of herself, the reference and instruction librarian for business, and the digital program librarian. Informally named the Credible Journal Criteria Working Group, the charge of the group was to identify a mechanism to assist the college's faculty (and the wider Loyola Marymount University (LMU) publishing community) in selecting credible journals in which to publish. The Working Group began its work in spring 2015.

Development of Evaluation Criteria and a Checklist
Since the associate dean requested help specifically for identifying credible open access journals, the working group focused its development efforts there. The group undertook a literature review as one of its first activities, to understand the current landscape of open access publishing. The group was interested in learning how other academic libraries were helping their faculty evaluate open access journals. During the literature review, the group found the criteria listed on the websites Principles of Transparency and Best Practice in Scholarly Publishing and Open Access Journal Quality Indicators to be especially helpful. The review also highlighted gaps in existing tools, which the group planned to address in its work.
The initial work of the group was to develop a checklist of evaluation criteria, organized by topic/theme that could be used to evaluate the website of an online journal. The group identified seventeen criteria; as each criterion was reviewed, the checklist was marked with a color to identify that the review uncovered no problem (green), a possible problem (yellow), or an obvious problem (red). The members of the group tested the checklist by working independently to review three online journals. Upon regrouping for discussion of the testing, the group found discrepancies in how the checklist was applied. The group determined that the checklist did not provide enough specific guidance in the review of each criterion, which accounted for the variation in how the checklist was applied.
The group sought the assistance of the university's director of assessment, to request ideas for how to improve the checklist. The director of assessment suggested that the group transform the checklist into a rubric, which would make the wording unambiguous as well as offer a clearer process for conducting the evaluation. The group took the advice, and began work on transitioning from a checklist to a rubric.

Development of Open Access Journal Evaluation Rubric and Scoring Sheet
The group designed the rubric to include a two-step process in the evaluation, first examining the journal itself (fourteen evaluation criteria) and second the publisher (two evaluation criteria). 1 The group assigned three levels of quality to each criterion-good, fair, and poorand provided a clear description for each. Of the sixteen criteria, copyright information is the only one that has two levels of quality, good or poor.
The first criterion on the rubric, in the journal evaluation section, is "web search for the journal." The rubric prompts the reviewer to consider whether the results of the web search are good, fair, or poor. The results of a "good" web search will show that "the journal is within the top 5 entries on the first page of search results and there are no scam alert postings." The results of a "fair" web search will show that "the journal is on the first page of search results but not within the top 5 entries and there are no scam alert postings." The results of a "poor" web search will show that "the journal is not on the first page of search results or there is at least one scam alert post about the journal." For each of the criteria on the rubric, there are three guiding considerations to help the reviewer make their determination. The reader may find the full rubric as Appendix A in this manuscript, and a downloadable version at https:// digitalcommons.lmu.edu/librarian_pubs/40/.
To use alongside the rubric, the group designed a scoring sheet. The scoring sheet consists of the rationale for each criterion, space to assign scores for each, and a guide to interpret the final score. Scores are to be assigned as 3, 2, and 1, corresponding to the relevant quality levels of good, fair, and poor. On the scoring sheet for the first criterion, "web search for the journal," the rationale noted is that "we want the popular reputation of the journal to be credible." The reader may find the full scoring sheet as Appendix B in this manuscript, and a downloadable version at the same web address as the rubric.
At the end of the evaluation of the journal and its publisher, the reviewer tallies the scores for each criterion, resulting in a total score from 16 (lowest possible score) to 48 (highest possible score). The scoring sheet then provides the following guidance for how to interpret the total score. 48-38 Good: Within this range the journal meets many of the evaluation criteria defined for credibility. At the higher end of the range the journal would have the fewest credibility concerns. 37-27 Fair: Within this range the journal meets some of the evaluation criteria defined for credibility. For a score in this range, with some credibility concerns, the researcher would need to decide whether or not to publish in the journal. 26-16 Poor: Within this range the journal meets the fewest of the evaluation criteria defined for credibility.
Please note that there is no weighting of scores for this tool, and the good, fair, and poor categories are mathematically even. Yet researchers are allowed to give weight to any criteria that may be important to them in their academic area, if they see fit to do so.

Pilot Tests and Lessons Learned
The working group identified two audiences to pilot test the rubric and scoring sheet (now collectively called the Journal Evaluation Tool). The group first tested the Journal Evaluation Tool within the library among librarians, since they would be potential first points of contact with faculty as liaisons with their respective campus departments. The group conducted this first pilot test in summer 2016, developing step-by-step instructions for using the tool which were emailed, along with a link to a journal to be evaluated. The group selected a journal for this evaluation that it had identified as suspicious, from a publisher unfamiliar to the group. It is important to mention here that the group could have selected any journal for this testing, because though it was interested in discovering the credibility of the journal, the focus of the pilot was to test the mechanics of using the tool. After the librarians had completed their evaluations, the group facilitated an informal feedback session. The group prompted discussion about the clarity of language used on the rubric and scoring sheet, the perceived burden of evaluating all of the criteria, and sense-making of the final score. The librarians had concerns about which criteria were the most important, and wondered if any should be weighted. They questioned how many categories needed to have a score of 1 in order to make the decision to eliminate publishing in a given journal. They requested that more instructions be included about using the rubric and the scoring sheet. They appreciated that the rubric was helpful in identifying what to look for and noted that locating information to verify journal-and publisher-level information without the rubric would have been challenging. They shared that when conducting an online search for the selected journal, the search results showed several other journals with similar names but with unique acronyms. One of the librarians identified a mathematical error in the scoring ranges.
During the feedback session, the group reported how the scores of the evaluations of each of the eleven pilot testers compared to each other. The average score was calculated for each criterion by adding the individual librarian scores and dividing the sum by the total number of raters. The average score clarified which individual criteria were rated good, fair, or poor. The interrater reliability (IRR) was calculated to measure the degree of agreement among librarian raters using the intraclass correlation coefficient in the statistical software SPSS. As shown in Table 1, the overall interrater reliability was 88.4%, or very strong agreement.
The group conducted a second pilot test with members of the faculty for whom the tool was developed, the College of Science and Engineering. The second test was conducted in fall 2016. The associate dean of faculty development and graduate studies from the College of Science and Engineering identified faculty whom the group could invite to participate in the test. Of the ten faculty contacted, six agreed to participate. The group shared instructions and the evaluation tool via email and requested that they complete the task of evaluating the same journal publisher that the librarians had evaluated, in a three-week timeframe. The group invited those six faculty to participate in a discussion session about their experiences using the Journal Evaluation Tool, hosted by the group over lunch, with four completing the in-person discussion. The discussion covered topics related to the mechanics of using the tool, like length of time to complete the evaluation, questions about the criteria themselves, and scoring. The discussion segued into the faculty's experiences and attitudes about publishing using an open access model, and how all of this related to their departments' promotion and tenure processes. Again, the average scores and interrater reliability were calculated, for an overall interrater score of 69.7%, or moderate agreement. 2 The scores per criterion are noted in Table 2.
Once we concluded the focus groups and analyzed the findings, the group incorporated the suggestions provided by the faculty members by further clarifying instructions for using the Journal Evaluation Tool on the instructions sheet, noting that the tool should be considered a "guide" and not the "final score."

Promotion and Outreach
Once the pilot tests and revisions were completed, the group shared the final version locally with the library's departmental liaisons to promote the tool with their respective departments. The group posted the tool to the institutional repository (https://digitalcommons. 2 The group received only four of the six faculty scoring sheets. lmu.edu/librarian_pubs/40/) with a Creative Commons CC-BY license, so that it could be further disseminated and revised by others. The URL for the tool was shared via social media platforms to reach a wide audience. Since the tool was uploaded to the repository in December 2016, 203 institutions from across the world have visited or downloaded the tool, predominantly in the United States and Canada. Since the repository platform is search engine optimized, in reviewing the referral data, the majority of the users were referred from digital commons with over 5,000 downloads.

Limitations and Future Research
The authors note several limitations to this work and suggest future directions for the tool. A clear limitation is in the population used to assist in the development of the journal tool, faculty from a single university college and librarians from that same university. The project was begun to respond to a stated need from the university's College of Science and Engineering, and from that perspective the completion of the tool was satisfactory. The faculty representatives participating in the focus group and testing of the tool represented different academic areas within the college. Those faculty, however, had all achieved tenure. It would have benefitted the discussion had the group included tenure-track faculty, as the demands of those faculty would likely have brought to light different concerns regarding the use of the tool. The group did not expand its testing of the tool beyond the initial college, so it did not include user experiences and perspectives of faculty from other disciplines and schools. Time and resources were also a barrier for the group to conduct further studies with other schools, since one of the librarians in the group left the institution.
A final limitation is that the group was only able to achieve over 80% interrater reliability with one of the rater groups (librarians) on the scoring rubric. This highlights the subjective nature of journal evaluation and echoes Olivarez, Bales, Sare, and vanDuinkerken's (2018) finding that "bias is an inherent aspect of interpretive research and that conscious reflection throughout the research process is a method for mitigating such bias" (p. 56). Workshops and training for librarians and faculty could also serve as a process for mitigating such bias.

NEXT STEPS
The group released the tool with a Creative Commons CC-BY license in order to enable broad dissemination, use, and enhancement of it by anyone interested in developing the tool further. It will be valuable to understand the adapted use cases of the tool and learn about experiences from other librarians using this tool at their institutions; the group is monitoring the number of downloads of the tool from the institutional repository, as well as mentions of its use on social media.
As the scholarly publishing landscape continues to evolve, the ability to critically evaluate traditional and open access journals will remain a core practice. Researchers and librarians must be alert to changes in the digital environment and keep tools (such as the Journal Evaluation Tool) as up to date and relevant as possible to correspond with the current publishing landscape. This article has presented one way in which researchers and librarians have done just that. Step 1. Journal evaluation

Web search for the journal
The journal is within the top 5 entries on the first page of search results and there are no scam alert postings.
The journal is on the first page of search results but not within the top 5 entries and there are no scam alert postings.
The journal is not on the first page of search results or there is at least one scam alert post about the journal.

Journal name
The journal name cannot be confused with another journal.
The journal being evaluated has a name similar to another journal but is able to be distinguished between the two.
The journal being evaluated is unable to be distinguished from another with a similar name.

Editorial board
The editorial board is listed with their full names and institutional affiliation.
The editorial board is listed with their full names only (no affiliation).
There is no editorial board listed.

Review process
The journal states whether it is peer reviewed/ edited and has a review policy listed.
The journal states whether it is peer reviewed/ edited and has no review policy listed.
The journal does not state whether it is peer reviewed/edited and has no review policy listed.

Conflicts of interest
The journal thoroughly and clearly states a conflicts of interest policy, including how it will handle potential conflicts of interest of editors, authors, and reviewers.
The journal states a conflicts of interest policy, but the description of how conflicts will be handled is unclear.
The journal does not state a conflicts of interest policy.

Journal website
The journal website is competently designed and functional. (examples: no broken links, easy navigation, no missing information) The journal website is adequately designed with passable functionality. (examples: adequate navigation, few broken links, some missing information) The journal is poorly designed and is not functional. (examples: broken links, poor navigation, missing information)

Revenue sources
The journal clearly states its business model. This includes any revenue sources, like author fees, subscriptions, advertising, reprints, institutional support, and organizational support.
The journal's business model lacks clarity when stating its revenue sources, like author fees, subscriptions, advertising, reprints, institutional support, and organizational support.
The journal does not state its business model.

Journal archive
The journal website contains an archive of its past issues with links to full text articles.
The journal website contains an archive but it may be incomplete or does not contain links to full text articles.
The journal does not have an archive of its past issues.

Publishing schedule
The journal clearly states how often its issues will be published each year and this agrees with the archive.
The journal does not state how often its issues will be published but it can be determined from the archive.
The journal does not state how often its issues will be published each year and it cannot be determined from the archive.

Author fees
The journal clearly states the amount of money an author will pay to have each article published.
The journal states that an author fee is required but does not note how much it is.