A qualitative content analysis of watchlists vs safelists: How do they address the issue of predatory publishing?

Predatory journals and publishers are a growing concern in the scholarly publishing arena. As one type of attempt to address this increasingly important issue, numerous individuals, associations, and companies have begun curating journal watchlists or journal safelists. This study uses a qualitative content analysis to explore the inclusion/exclusion criteria stated by scholarly publishing journal watchlists and safelists to better understand the content of these lists, as well as the larger controversies that continue to surround the phenomenon that has come to be known as predatory publishing. Four watchlists and ten safelists were analyzed through an examination of their published mission statements and inclusion/exclusion criteria. Notable differences that emerged include the remaining influence of librarian Jeffrey Beall in the watchlists, and the explicit disavowal of his methods for the safelists, along with a growing recognition that the “list” approach may not fully address systemic aspects of predatory publishing that go beyond the individual author's ethical decision-making agency.


Introduction
Predatory publishers and journals first caught the scientific community's attention in 2008 when Jeffrey Beall, a librarian at University of Colorado Denver, coined the term to characterize a small number of open access journals and publishers he included on a list published on his website.Along with its many benefits, the open access movement had also ushered in new practices, such as "gold" open access, meaning that the cost of an article's publication is shifted to authors to enable free access through the journal's website.Beall (2012) used the word predatory to characterize journals and publishers he believed were exploiting this model to accept more articles, purely for the sake of increasing profits, often without adequate peer review.He started his list to help authors make informed decisions in the context of this rapidly changing publishing landscape.
When it first appeared in 2008, Beall's list included a handful of journals and publishers that he identified as predatory.But as the list grew, so did the controversy surrounding it.The controversy flared when Beall announced in an October 18, 2015, tweet that he had added a journal called Frontiers to his list of predatory journals.A Frontiers associate editor immediately tweeted back: "Frontiers being added to Beall's list reveals the big weakness of Beall's list: it's not based on solid data, but on Beall's intuition" (Bloudoff-Indelicato, 2015, para. 7; see also Crawford, 2014;Teixeira da Silva, 2017).In 2017, Beall took his list offline, offering the following explanation: "In January 2017, facing intense pressure from my employer, the University of Colorado Denver, and fearing for my job, I shut down the blog and removed all its content from the blog platform" (Beall, 2017, p. 273).
Beall's original list, as well as the array of initiatives and controversies that have ensued since its 2017 demise, highlight the ethical complexities that this article addresses.Amidst these controversies, most seem to agree with Beall's main premise.In fact, a 2017 report by the National Academies of Sciences, Engineering, and Medicine identified predatory journals as one of the "new forms of detrimental research practices" that currently threaten research integrity (NASEM, 2017, p. 2).However, the situation has been fraught with disagreement over who is authorized to define predatory journals and what is the best approach (Teixeira da Silva & Tsigaris, 2018;Van Noorden, 2014).Some have taken up Beall's cause, publishing updated lists based on Beall's work.For example, Cabells Scholarly Analytics (CSA) now offers a fee-based "Predatory Reports" list (CSA, 2020).Although Cabells International was founded in 1978, well before the term "predatory" was used in the context of scholarly publishing, their efforts to monitor journal quality have gained prominence in the wake of Beall's list and its demise.Cabells International also publishes "Journalytics," a list that takes the opposite approach: cataloguing journals that meet their stated criteria, rather than those that do not.Other entities have followed suit, advocating the more positive approach that is adopted in "Journalytics," rather than the negative approach of Beall's list (Neylon, 2017;Roberts, 2017).
Until recently, these two types of lists have often been referred to as "blacklists" and "whitelists."However, in this article, we follow the lead of Cabells International, as indicated in the announcement of their 2020 decision to stop using these terms because of the "symbolism inextricably tied to the idea of blacklists and whitelists" and "in support of, and in solidarity with, the fight against systemic racism that our country is facing" (Bisaccio, 2020, para. 1) Thus, rather than using the terms "blacklist" and "whitelist," in this article, we use the term "watchlist" to refer to lists that aim to identify predatory journals or publishers and "safelist" to refer to lists that aim to identify legitimate journals or publishers.
Through qualitative content analysis of the major watchlists and safelists currently available, this article offers a comprehensive and current guide to these tools.Through this analysis, we also use these lists to illuminate the controversy itself.Indeed, a growing body of literature suggests that predatory publishing is too complex to be addressed by any individual list; instead, experts are demanding a more nuanced approach.For instance, rather than distinguishing between good and bad journals, some suggest identifying types of unethical behaviors that can occur at any journal (Zernes, 2018).Furthermore, in the face of declining public research funding, the pressure to publish attention-grabbing results is intensified and can lead researchers to cut corners (Davies, 2019).Another factor is mandatory publishing quotas placed on scientists by national governments.This has been recently documented as a factor in, for example, India (Dehal et al., 2018) and Serbia (Djuric, 2015).However, selecting the right journal is not only difficult for authors in "peripheral" nations, who face limited options and increased pressure to publish in international journals (Nobes, 2017;Omobowale et al., 2014), but also for authors in the United States and other "center" (i.e., non-peripheral) nations, who have more options but still can succumb to predatory practices for a variety of reasons (Pyne, 2017). 1n short, experts disagree on the inherent value of open access publishing as a sustainable publishing model and even on how to define predatory (Roberts, 2017;Teixeira da Silva et al., 2019).Some groups are developing consensus around agreed-upon definitions (e.g., COPE Council, 2019;Grudniewicz et al., 2019) whereas others have argued for abandoning the term altogether and replacing it with another term such as deceptive (Anderson, 2019).Attempting to help authors navigate this increasingly complex publishing terrain, one recent article (Strinzel et al., 2019) analyzed two major safelists (the Directory of Open Access Journals and Cabells Whitelist, now known as "Journalytics") and two major watchlists (Beall's List and Cabells Blacklist, now known as "Predatory Reports") and identified four broad concepts across these criteria: transparency, ethics, professional standards, peer review and other services.Additionally, they found that watchlists focused heavily on professional standards and ethics, whereas safelists gave transparency the most weight.They also found that although both types of lists noted peer review as an important factor, they did not emphasize the quality of the peer review.Most importantly, they found an overlap of journals or publishers that were found on both list types and disagreement on the weighting and evaluation of criteria.Teixeira da Silva and Tsigaris (2020) pushed the argument against the validity of watchlists even further with their recent study noting the high prevalence of false discovery rates of predatory behavior when using Beall's criteria.
Our qualitative content analysis extends this line of inquiry, offering a more comprehensive assessment of currently available lists intended to help screen for predatory publishers and journals.Specifically, our content analysis is guided by the following research questions: • What are the lists currently available to help authors identify and avoid predatory journals?
• What are the similarities and differences in the criteria that these lists employ to identify predatory journals?
• Who are the major players behind these lists, and how are the as- sumptions, motivations, and unstated beliefs of these players reflected in the list's criteria?
• What are the points of tension and disagreement among these players, and what are the points of overlap and collaboration?
• What are the limitations of the "list" approach to addressing pre- datory publishing, and where is this approach situated in relation to the larger predatory publishing controversy?
Through addressing questions such as these, our analysis considers the various lists not only as lists, each with its own criteria and membership rules, but also as a window into a larger controversy that has been unfolding for many years.

Methodology
To acquire an appropriate sample of watchlists and safelists, Google searches were conducted between September 11 and October 29, 2019.Keywords for the searches included: "predatory publishers whitelist," "scholarly publishing organizations," "predatory publisher blacklists," "lists of reputable academic journals," "scholarly journal whitelists," "journal whitelist" and "journal blacklist."(The terminology searched for in our methodology reflects the time period when "blacklists" and "whitelists" were still the dominant terminology for these lists.)Secondary searches were also conducted through links or references from primary sources.This process yielded four watchlists and ten safelists, as shown in Table 1.
We conducted a "conventional content analysis" (Hsieh & Shannon, 2005, p. 1279), meaning that textual data were analyzed without imposing predefined categories; rather, patterns, codes, and themes were allowed to emerge from the textual data.Conventional content analysis seeks deeper understanding of concepts or phenomena but stops short of theory-building.Thus, the purpose of this study is not to evaluate the lists' usefulness or accuracy, but rather to highlight the commonalities and differences among the criteria in these lists and use these to understand the broad contours of the controversies that underlie them.To implement this analysis, criteria from all the identified sources were found and imported into NVivo qualitative analysis software.The first author developed an initial coding scheme reflecting the major categories that appear in these criteria.Then a second round of coding was conducted jointly by the first author and second author to collapse these codes into more salient groupings.The entire research team then convened to agree on the coding structure reflected in our codebook (Appendices A and B).

Results
Our results are organized around the two list types included in our analysis.After an overview of the data in each category, we summarize key features from the coding of criteria.The similarities and differences among criteria are contextualized in relation to the larger controversies that surround predatory publishing.

Watchlists
Beall still wields great influence among watchlist developers.In fact, even though Beall's official version was taken offline in January 2017, all four watchlists we analyzed express some relationship to Beall's list.The most direct relationship appears in "Beall's List of Predatory Journals and Publishers," a site maintained by an anonymous researcher.The site links to an archived version of Beall's list, dated January 15, 2017, and the site author publishes updated versions that include newly discovered journals and publishers that the researcher deems predatory.The site's "Disclaimer" clearly states that the author is not Jeffrey Beall, and "I prefer my identity to be anonymous, largely for the reasons that Beall mentioned in his recent article" (Beall's List,n.d.,para. 3).At the WordPress site, readers will also find a link to "Beall's Criteria," and the anonymous author claims to still use these criteria to determine which journals and publishers to include on the updated watchlists.
Another example is "Stop Predatory Journals" (SPJ), which states that "The kernel for this list was extracted from the archive of Beall's list at web.archive.org"(SPJ,n.d.,para. 1).This site runs on GitHub, a platform that allows anyone to make anonymous contributions, thus offering a "collaborative approach" (SPJ,n.d.,para. 5).The authors' anonymity, they claim, "allows for them not to be harassed" (SPJ,n.d.,para. 4).
The third watchlist is "Dolos List."As this site's author states, "There are other lists from which this list is based, including which is based on the Beall's List" (Georges,n.d.,para. 14).Although similar in origin, this list author emphasizes his personal identity.In fact, it seems that the site's author wants the list to be identified with his scholarly credentials.Thus, for example, the site includes an "About Me" page that links to the CV, diploma, and other pertinent documents to establish the legitimacy and expertise of its creator, who identifies himself as "Professeur Alexandre GEORGES, Theoretical Physicist" (Georges, n.d.).Furthermore, he believes that a list should be compiled by a named individual, rather than anonymous, while also suggesting that his list "provides additional elements and follows stricter rules for addition and withdrawal" than Beall's, even though his initial list was based on Beall's (Georges,n.d.,para. 13).
Of the four watchlists in our analysis, Cabells Scholarly Analytics' (CSA) watchlist (now known as "Predatory Reports") expresses the least explicit connection to Beall's list.Other than a quotation from Jeffrey Beall that appears on the landing page, the site makes no direct reference to Beall, nor does it suggest that its content was based on Beall's List.In fact, the site's language implies that one of the motivations of Cabells is to provide a list that is superior to Beall's, and to other comparable lists, because of the criteria and systematic analysis on which the list is based.For instance, the site claims that "Each journal is examined against 60+ behavioral indicators …" and "Our criteriabased methodology, informed by industry expertise and community support, objectively evaluates whether a journal is predatory" (CSA, 2020, para. 5).
Given that all four watchlists make some mention of Beall's list, it is not surprising that Beall's criteria influence the criteria published on each of these sites.Furthermore, of these four watchlists, all except one, Stop Predatory Journals, are still being actively updated and edited.As shown in Appendix A, our coding identified four major categories of criteria in these watchlists: Publication practices, false branding, role of authors, and list procedures.

Publication practices
This category encompasses an array of criteria related to the publication process.Two of the second-level criteria under publication practices were present in all four watchlists: paper quality and faulty peer review.In discussing paper quality, both Cabells and Beall's List emphasize the form of published articles, suggesting that if a journal routinely publishes a lot of non-academic pieces, such as essays by nonacademic authors, or "polemical editorials," (Beall, 2015, p. 4) as stated in the Beall's list criteria, these are signs of a predatory journal.The criteria for Dolos List and SPJ, by contrast, place more emphasis on article content, such as "false scientific data or pseudoscientific information" (Georges,n.d.,para. 10) and "acceptance of low-quality papers" (SPJ,n.d.,para. 3).
Faulty peer review is the criterion that receives most attention across the four lists.All four lists include language to indicate that if no actual peer review is being done, even though the journal claims to be peer reviewed, this indicates a predatory journal.Of the four watchlists, Cabells devotes the most space to specific practices that might indicate faulty peer review.For example, they include lack of information, or faulty information about an editorial board or reviewers on the journal's website, as well as lack of a peer review policy.Three of the four watchlists (Dolos list, Beall's list, and Cabells list) mention the possibility that peer review is being done, but not rigorously.For example, Beall's list mentions assigning author-recommended reviewers without proper vetting.Dolos list and Cabells provide less specific descriptions of lack of rigor in peer review.
A third type of criteria that emerged under publication practices, access, copyright, and intellectual property violation, was also present, with some variation, across the four watchlists.In many cases, this theme encompasses more idiosyncratic features of journals-features that may be difficult for the casual observer to detect.For instance, both Beall's List and Cabells include the practice of not allowing web crawlers to search journal content as a criterion that marks a journal as predatory.In some cases, there is not much explanation for this criterion, and, in fact, this has been a criticism against the watchlist approach to predatory publishing (e.g.Misra et al., 2017).

False branding
False branding refers to a journal's public presentation.Criteria in this category are present in all four watchlists.Three of the four watchlists designate e-mail spamming as a predatory business practice.For example, the criteria for Cabells, SPJ, and Beall's list include the practice of sending e-mail solicitations for authors and potential editorial board membership.In fact, Dolos list is the only one of the four watchlists in our analysis that does not name e-mail spamming of authors as an indicator of predatory publishing.This omission is surprising given that this is one of the more visible, well-known, and easily identifiable features of predatory publishing across the scholarly community (Beall, 2015;Memon, 2018).
Other criteria prevalent under false branding are misleading business practices, which encompasses items such as transparency of author fees and misleading staff and partnership claims which relates to false or misleading claims about editorial staff or relationships with other entities.In short, this category entails a claim that would lead potential authors or readers to believe a journal is legitimate when it is not.Variation exists in the kind of claim that is made and in how difficult it would be for a casual observer to recognize the fraudulent nature of the claim.

Role of authors
Only two of the four watchlists (Dolos List and Beall's List) made mention of the role of authors in avoiding predatory publications.In the case of Dolos List, there is an assumption of innocence and an expressed desire to protect junior scholars from persecution for inadvertently publishing in a predatory publication.Beall's list, on the other hand, lists several actions authors should take to ensure they are properly vetting a publication, including reading peer comments, looking at publication history, and assessing direct contact with the publisher.This is a stark contrast with the author agency suggested in the safelist approach, as will be discussed in the subsequent section.

List procedures
Of the four watchlists examined, only one (Dolos List) publicly stated the procedures for adding or removing a publication from the list.This lack of transparency is one of the major critiques of the watchlist approach to combatting predatory publishing, and again, is in stark contrast to the more transparent methodology adopted in the safelist approach.

Safelists
Safelist advocates believe safelists are more feasible and reliable than watchlists because "they are a list of all those agents that have been certified as meeting a certain level of quality assurance" (Neylon, 2017, para. 6).Watchlists, by contrast, are "technically infeasible" because they are "never complete" (Neylon, 2017, para. 5).Our analysis identified ten safelists.These include three discipline-specific lists: Australian Business Deans Council Journal Quality List (ABDC, specific to business journals), Berlin Institute of Health Open Access Journal Positive List (BIH, specific to health professions), and International Committee of Medical Journal Editors (ICMJE, specific to medicine).They also include one list focused on journals published in the Global South: Journal Publishing Practices and Standards (JPPS, which publishes several sub-lists, each focusing on journals of a particular geographic region).
The other safelists included in our analysis adopt a more comprehensive focus.One of these, Cabells Journalytics (a counterpart to Cabells Predatory Reports, and run by Cabells Scholarly Analytics), is the only safelist that requires a fee for access.Another list is Publons, which is not explicitly identified as a safelist but emerged in our search because the website includes a Journal List, with a list of criteria that publishers must meet before they receive a green checkmark.
In contrast to the watchlists included in our analysis, each of which is clearly authored and published by an independent individual or organization, analysis of the safelists reveals a great deal of overlap.This reflects the extensive collaboration that has occurred among the organizations that have developed safelists.Three organizations in particular are leaders in this regard: Directory of Open Access Journals (DOAJ), Open Access Scholarly Publishers Association (OASPA), and Committee on Publication Ethics (COPE).These organizations have a distinct presence across the global effort to address predatory publishing, and because of their collaboration and leadership, our analysis reveals largely similar content across the safelists' criteria.In fact, so much collaboration has occurred among these lists that it is often difficult to distinguish the criteria of one safelist from another.In many cases, the collaboration is made apparent on the organization's homepage.For example, the Berlin Institute of Health explains that its safelist (now termed "Positive List") is an aggregate of journals included both in DOAJ and in PubMed.Rather than developing its own criteria, the Berlin Institute relies on DOAJ's safelist criteria (BIH, 2020).Even while acknowledging the significant overlap in the content of safelists, and criteria, that has resulted from this collaboration, we still treat these as separate items because, as will become apparent in the analysis below, each list has its own process for determining journal inclusion.
Whereas the watchlists we analyzed all indicate some connection to Beall's list, the ten safelists hardly mention Beall, and their mission statements reveal a wider variety of purpose.In fact, predatory publishing is mentioned explicitly in the rationale for only two safelists, and for both of these, it is mentioned in passing.Specifically, the ABDC list includes "Not appear to be a predatory journal" (ABDC, 2019, pg.7) as one of its criteria for inclusion in the list, but it does not specify the factors or characteristics that might cause a journal to be assessed as predatory.ICMJE also explicitly mentions predatory publishing: "… a growing number of entities advertise themselves as scholarly, peer-review medical journals yet do not function as such" ("predatory" or "pseudo-journals") (ICMJE,n.d.,para. 4).Other than these two safelists, for which the mission statements specifically mention predatory publishing, most others included in our analysis make little or no explicit mention of predatory publishing.This distinction suggests a possible difference in motivation between the two categories of list: Watchlist authors seem interested in upholding Beall's legacy, whereas safelist authors seem more interested in detaching themselves from the controversial aspects of Beall's list.
For example, one safelist, JournalGuide, links to "Beall's famous list of criteria" in a page that explains this site's rationale.But after this brief mention, the site authors go on to distinguish their own goals from Beall's, noting that their goal in adopting the safelist strategy is "to remain inclusive" (JournalGuide, 2014, para.3).JournalGuide is a free service that allows users to search for a particular journal by title; it then provides identifying information, including a green "Verified" designation for those journals that meet the stated criteria for safelist inclusion.
Without mentioning predatory publishing or Jeffrey Beall, some lists indirectly suggest that eliminating predatory publishing is one of their motives by stating criteria that involve peer review or other quality markers.For example, Journal Publishing Practices and Standards (JPPS) states its purpose is to assess "journals against a detailed and transparent set of criteria," and that it "is intended to give researchers a greater feeling of trust that they are submitting their work to quality publications and will, hopefully, encourage them to submit their work to regional journals" (JPPS, 2017a, para.3).Furthermore, JPPS also houses multiple other geography-specific lists that are also safelists, but each has a geographic focus, and thus, they express a different motive than the more global safelists.The regions/countries included in these geographic lists include Africa (AJOL), Bangladesh (BanglaJOL), Cambodia (CamJOL), Mongolia (MongoliaJOL), Nepal (NepalJOL), and Sri Lanka (SLJOL) (JPPS, 2017a).These geography-specific lists express a much broader motivation beyond just the elimination of predatory publishing.For example, as stated on the AJOL homepage, "At the same time as online academic resources from the developed Global North are made available to Africa (such as HINARI, AGORA and OARE), there needs to be corresponding online availability of information from Africa" (AJOL, n.d., para.1).Thus, the main motivation is to ensure that lesser-known journals published in Africa will be accessible to wider audiences.As stated later in the site's introductory page, "Valuable information has not reached the people who need it" (AJOL,n.d.,para. 3).Later in the homepage is a list of evaluation criteria that would ostensibly rule out predatory journals, but this is never stated explicitly.In fact, this site reveals another important dimension of this controversy, as an important motivation of the AJOL site is to ensure that African journals do not automatically get perceived as predatory, or low quality, even while acknowledging the importance of maintaining high standards for journals in the AJOL list.Similar motivations are also expressed in other geography-specific safelists housed under JPPS.
Cabells also does not mention predatory publishing in its criteria, and in fact, its "About Us" page claims that its journal monitoring services originated "almost 40 years ago" (CSA, n.d., para.3), long before the rise of open access or predatory publishing.Similarly, Directory of Open Access Journals (DOAJ) does not mention predatory publishing; rather, its mission is "is to increase the visibility, accessibility, reputation, usage and impact of quality, peer-reviewed, open access scholarly research journals globally, regardless of discipline, geography or language" (DOAJ, n.d., para.5).This safelist is community based, and journals must apply for membership.Along similar lines, Open Access Scholarly Publishers Association (OASPA) states its mission as follows: "Representing a community of scholarly publishers and related organisations, OASPA is committed to developing and disseminating solutions that advance open access, preserve the integrity of scholarship, and promote best practice" (OASPA, n.d., para.1).Committee on Publication Ethics (COPE) also works closely with DOAJ and OASPA, although COPE has a broader scope, not limited just to open access publishing.The three organizations, along with World Association of Medical Editors (WAME) have collaborated, as stated on the COPE website, "to identify principles of transparency and best practices for scholarly publications and to clarify that these principles form part of the criteria on which membership applications will be evaluated" (COPE, n.d., para.1).
As shown in Appendix B, coding of the safelist criteria was more complex, revealing five top-level categories: Business practices, publication practices, truth in branding, how to use safelist, and multiple independent assessments.Some of these obviously correspond with, and contrast to, top-level categories in the watchlist codebook, but some key differences emerge, reflecting the different motivations of the authors of the two types of list.First, in coding the watchlist criteria, a separate category for business practices was not warranted because the criteria displayed so much overlap between business practices and publication practices that it was impossible to distinguish.By contrast, in coding the safelist criteria, a clear separation emerged, as these criteria reflected extensive attention to clearly identifiable publication practices (e.g., editorial guidelines), separate from clearly identifiable business practices (e.g., funding and ownership).Also, some differences emerge in the codebook because watchlists have a different starting goal than safelists.Thus, false branding is a criterion for watchlist inclusion, whereas truth in branding is a criterion for safelist inclusion.However, these criteria refer to similar aspects of a journal's public presentation.Because of the extensive collaboration among organizations that have developed safelists, for many categories of criteria, our analysis reveals overlaps, as will be shown in the following discussions.

Business practices
For example, under business practices, several sub-categories include criteria that are virtually identical across Cabells, COPE, DOAJ, and OASPA.In particular, these four lists use largely the same language to articulate the criteria coded as communication policies and funding and ownership transparency.Communication policies include coded text that refers to a journal's advertising policies.In this regard, the shared concern of these four groups is to exclude journals in which paid sponsors of advertising have undue influence on the content published in the journal.The key language in all four of these safelists is "Advertisements should not be related in any way to editorial decision making and shall be kept separate from the published content" (COPE/ DOAJ/OASPA/WAME, 2018, p. 2).On a related note, also coded under communication policies, is the following language on direct marketing, which appears in the criteria for all four safelists: "Any direct marketing activities, including solicitation of manuscripts that are conducted on behalf of the journal, shall be appropriate, well targeted, and unobtrusive" (COPE/DOAJ/OASPA/WAME, 2018, p. 2).And for funding and ownership transparency, the concern is transparency and accuracy of information about the journal's ownership or management, as captured in the following language in the stated criteria of all four safelists: Information about the ownership and/or management of a journal shall be clearly indicated on the journal's website.Publishers shall not use organizational or journal names that would mislead potential authors and editors about the nature of the journal's owner.(COPE/DOAJ/OASPA/WAME, 2018, p. 2) The significant overlap among the stated criteria of these four safelists reflects the extensive collaboration among these organizations, in contrast to the independent approach that characterizes watchlists.Furthermore, across the content coded as business practices, the four major collaborating organizations (i.e., Cabells, COPE, DOAJ, and OASPA) are predominant; little content from other safelists' criteria was coded in this category.This tells us something else about the safelist movement as a whole: that this movement appears to be divided into a few different segments, with the four major organizations being key players and collaborators, but their scope has limitations in that other major players have not been influenced by them and are instead advocating different kinds of criteria.This overlap among the four key players, and sincere desire to collaborate, may also suggest that these four key players are motivated by their desire to consolidate power by achieving consensus on the aspects of watchlist criteria that have previously been controversial.

Publication practices
Significant overlap among these major players also appears under publication practices.For example, in the editorial guidelines sub-category, the following language regarding peer review is used by COPE, DOAJ, and OASPA: Journal content must be clearly marked as whether peer reviewed or not.Peer review is defined as obtaining advice on individual manuscripts from reviewers expert in the field who are not part of the journal's editorial staff.This process, as well as any policies related to the journal's peer review procedures, shall be clearly described on the journal's Web site, including the method of peer review used.Journal websites should not guarantee manuscript acceptance or very short peer review times.(COPE/DOAJ/OASPA/ WAME, 2018, p. 1) Interestingly, Cabells includes some similar language, but includes additional details defining the process and key terms, such as 'rigorous.'Three other safelist developers (ABDC, ICMJE, and JPPS) mention peer review.ABDC and JPPS devote brief text to peer review, whereas ICMJE has a lengthier passage that specifically refers to predatory publishing: A growing number of entities are advertising themselves as "scholarly medical journals" yet do not function as such.These journals ("predatory" or "pseudo-journals") accept and publish almost all submissions and charge article processing (or publication) fees, often informing authors about this after a paper's acceptance for publication.They often claim to perform peer review but do not … (ICMJE, 2019 p. 4) In short, peer review is obviously important for determining inclusion in both watchlists and safelists.Although the language varies, the concern is to rule out journals that do not require peer review, or do not require peer review of adequate rigor.
In the quality of scholarship sub-category under publication practices we also see a difference between watchlist and safelist approaches.For watchlists, the emphasis was on quality of published content, whereas for safelists, there is more content coded under this sub-category, and the emphasis is broader, to include more detail on the practices that may lead to poor quality content.Some of these practices would be hard to detect from reviewing the published content of a journal, and some of the organizations that publish safelists spell out these considerations in great detail.

Truth in branding
In the truth in branding top-level category, we also see overlap among the four major players.As the codebook indicates, an overarching concern in this category is language that refers to a journal's public presentation.For several sub-codes under this category, there are overlaps among most or all the key players.For example, in content coded as website quality, three of the four major players (OASPA, COPE, and DOAJ) use the same language: A journal's website, including the text that it contains, shall demonstrate that care has been taken to ensure high ethical and professional standards.It must not contain information that might mislead readers or authors, including any attempt to mimic another journal/publisher's site.An 'Aims & Scope' statement should be included on the website and the readership clearly defined.There should be a statement on what a journal will consider for publication including authorship criteria (e.g., not considering multiple submissions, redundant publications) to be included.ISSNs should be clearly displayed (separate for print and electronic).
(COPE/DOAJ/OASPA/WAME, 2018) Interestingly, Cabells does not use this language, but simply says, "All journals shall have a clear and functioning web site.Additionally, a journal's web site shall be free of: grammatical errors, prominent misspellings and/or non-functioning links" (CSA, n.d., para.10).Notably, this particular criterion has been somewhat controversial, in regard to watchlists, because some newer journals, especially those in peripheral nations, may not be able to afford the design of a sophisticated website, but this does not mean they are predatory journals (Memon & Waqas, 2018).Relatedly, "hijacked" journals will often produce a sophisticated website that resembles that of the prestigious journal the hijackers are trying to imitate.This further problematizes the use of a website's appearance as an indicator (Asadi et al., 2017;Shahri et al., 2018).

How to use safelist and multiple independent assessments
For the two remaining top-level categories, how to use safelist and multiple independent assessments, we see more variation among the safelists.The former includes various types of descriptive language indicating the list's intended audience, as well as other matters such as procedures for adding or removing journals.The latter category includes an assortment of text that informs readers how this particular list relates to other safelists and, in some cases, encourages readers to rely on multiple sources, rather than just a single source, to assess the quality and legitimacy of a journal.In these two codes, we see more variation among the lists, even among the major collaborating organizations.This is because even though there is significant overlap among the criteria, the different safelists stipulate different processes for determining journal inclusion, and related questions such as how a journal can get removed from the list; these differences may reflect different motivations of the organizations that sponsor the lists.For example, OASPA relies on criteria established by DOAJ and COPE.But for OASPA, list membership requires a fee, and to be included in the OASPA list, a publisher must already have one journal that is a member of DOAJ (OASPA, n.d.).COPE also requires a membership fee from organizations, although of course, these organizations also must meet the published criteria-again, criteria that are largely shared with OASPA and DOAJ (COPE, n.d.; COPE, n.d.).By contrast, JPPS and DOAJ do not require a membership fee.Rather, any journal that the organization identifies as meeting the specified criteria that they specify will be added to their published list (DOAJ, n.d.;JPPS, 2017b).
In short, watchlist and safelist advocates share the motivation of ensuring that journals maintain a public presence that is accurate and ethically responsible and that they uphold accepted standards of academic integrity.Although we see great overlap among the content of the criteria publicized for safelists, we see variation in procedures for inclusion in, or removal from, the lists.We also see, regarding some of the criteria for these safelists, some nuanced differences in the language used and in the amount of attention devoted to various criteria.In contrast to the watchlists in our analysis, the safelists seem to operate with more of a shared sense of purpose and collaborative spirit, possibly suggesting that they are motivated to consolidate power by working together to achieve more prominence than any individual or single organization could ever achieve.

Implications and future challenges
Qualitative content analysis of currently available watchlists and safelists suggests some key differences and similarities, both in their motivations and in their criteria.On the former point, the motivations of those who have developed watchlists remain the most closely tied to Jeffrey Beall's initial attempt to help authors avoid predatory publishers and journals.In contrast, safelist authors express little or no connection to Beall, and, in some cases, seem to publicly disavow any connection with him due to the stigma that may ensue.Our analysis also highlights the extensive collaboration that has occurred among those who are developing safelists, in contrast to the watchlist initiatives, which tend to follow in the tradition of Beall's list, offering more independent, soloauthored perspectives.
Perhaps the most important conclusion that emerges from our comprehensive analysis of currently available watchlists and safelists is that any attempt to create a list of this nature will be incomplete and out-of-date from the moment it is published.This is because of the extent to which predatory publishing is changing and growing every day.We are also seeing endless new variations.For example, in "hijacked journals," cybercriminals literally "hijack" a prestigious academic journal, taking its name, claiming to be editors, starting a false website, and then sending spam e-mails to authors encouraging them to submit and pay an author's fee (Asadi et al., 2017;Shahri et al., 2018).Such hijacked journals would not be detected by any of the screening mechanisms identified in our analysis.
A related problem is that, in some cases, suspect publishing practices occur at a journal that does not appear to meet any of the criteria that would mark it as predatory.Thus, for instance, in 2017, a journal called Tumor Biology retracted 107 articles.The journal at that time was indexed in Social Science Citation Index, published by Springer, and was considered a reputable journal by every measure.The retractions were "due to fake reviews," likely including author-nominated reviewers without adequate vetting (Hu et al., 2019; see also Shopovski et al., 2019).
In the context of larger controversies that still surround predatory publishing, a clear limitation of the list approach is to place the responsibility on individual people rather than systems.By contrast, other approaches encourage a more multi-faceted understanding of predatory publishing, distributing responsibility elsewhere.For instance, one suggested approach is for universities to establish "journalology" centers to support their researchers in deciding where to publish (Krishan & Kanchan, 2019).Saurin (2016) also advocates a more complex approach, understanding research misconduct "as emergent phenomena arising from interactions between agents involved in the broader research and publication system" (p.1850).
To address these limitations of the "list" approach-limitations that have been discussed in previous literature but come into clearer view through our analysis-more empirical research is needed to illuminate the lived realities of stakeholders impacted by, and implicated in, predatory publishing.Such research needs to go beyond collecting mere anecdotes, using rigorous qualitative and quantitative methods to gain valid and reliable social science data that offers insights into the shortcomings of currently available watchlists and safelists, from the perspectives of those who use them.Along these lines, some researchers are beginning to explore the efficacy of information campaigns on raising awareness of predatory publishing (e.g., AlRyalat et al., 2019), whereas others have asked researchers broader questions about their experiences and knowledge.For example, Davies (2019) interviewed scientists and asked them about their daily experiences of research integrity.In so doing, she identified a significant gap between research integrity as defined in current ethical codes and research integrity as defined by scientists.Interviews with various stakeholders-authors, librarians, editors, and others-involved in predatory publishing may reveal similar disparities between how predatory publishing is conceived in the lists we have analyzed and how it is experienced in stakeholders' lived realities.
Amidst the quickly evolving landscape is today's scholarly publishing industry, it is tempting to cling to a term such as predatory that suggests authors face an obvious choice in determining which venues are trustworthy and which are not.Such nomenclature implies a deontological ethics of individual choice in the face of clearly defined distinctions between right and wrong, and the important work that has been done to develop various types of journal lists appears to be, at least in part, motivated by this desire for certain answers.In truth, however, our analysis has contributed to the efforts of those who emphasize that such decision-making is not merely a binary choice between predatory and not predatory.As we build on the important work that has been done to develop journal lists, we need to acknowledge the important accomplishments of these lists while pursuing more empirical research to understand further the complexities of predatory publishing and the decisions that today's scholarly authors face in order to enable and equip them to make informed publishing decisions.

Funding
This project is part of a National Science Foundation Grant (Grant # 1926348), entitled "Standard: STEM Training in Ethics of Publication Practices (STEPP)," and is subject to the provisions of NSF 18-532 [Cultivating Cultures for Ethical STEM].The sponsors had no direct participation or influence on the actual data collection and analysis.

Communication and contact practices
Criteria related to the journal's communication practices (i.e.spamming), the ability to contact them, falsifying physical locations, and the quality/navigability of the journal/publisher website "the website does not identify a physical address or the publisher gives a fake address" and "a 'contact us' page that only includes a web form or an email address" and "targeting scholars through mass-email spamming in attempts to get them to publish or serve on editorial boards"

Misleading staff and partnership claims
False, misleading, or missing claims about editorial staff (names and/or qualifications) and relationships with other entities (universities, corporations, etc.) "falsely claims universities or other organizations as partners or sponsors" and "listing fake scholars as members of editorial boards…" and "the journal does not identify a formal editorial/review board"

Misleading Business Practices
Criteria related to how the journal conducts business, including claims of non/forprofit status, clarity/transparency of fees, and general transparency about business operations "the journal or publisher claims to be a non-profit when it is actually a for-profit company" and "lack of financial transparency, particularly in terms of the cost of publications for authors" and "Notifying author of fees only after acceptance" Role of Authors Criteria related to the role/responsibility of authors in determining if a journal/ publisher is legitimate List Procedures Criteria describing the process through which journals are added or removed from the blacklist

Codebook
Emergent themes and subthemes

Business practices Criteria that pertain to oversight of the business processes Communication policies
Criteria related to how the journal/publisher communicates with the public through their website (availability to directly contact editors, staff, etc.), advertising and direct marketing, and how (or whether) their policies for such communication are clearly stated "Advertisements should not be related in any way to editorial decision making and shall be kept separate from the published content." Funding and ownership transparency Criteria pertaining to clarity of business status (for-or non-profit), revenue sources (partnerships, fees, etc.) and ownership and/or management "Information about the ownership and/or management of a journal shall be clearly indicated on the journal's website.Publishers shall not use organizational or journal names that would mislead potential authors and editors about the nature of the journal's owner."

Publication practices
Criteria that pertain to oversite of the publication process Access, copyright, and intellectual property guidelines Criteria pertaining to the presence of clearly stated guidelines related to how journals/publishers manage issues surrounding access to already published manuscripts (e.g.archiving, electronic access), copyright statements, and guidelines surrounding issues of intellectual property "A comprehensive written copyright and licensing policy must be clearly stated on the journal's website."

Author instructions & policies
Criteria that discuss instructions and policies directed at authors and pertaining to topics including misconduct, corrections/retractions, and defining authorship "Clear and detailed Instructions for Authors are present and linked to from the homepage."Editorial guidelines Criteria pertaining to the presence of clearly stated guidelines related to how the editorial process is conducted, including the peer review process, guidelines for dealing with plagiarism "Journal content must be clearly marked as whether peer reviewed or not…This process, as well as any policies related to the journal's peer review procedures, shall be clearly described on the journal's Web site…Journal websites should not guarantee manuscript acceptance or very short peer review times."Publication history Criteria related to the publication history and track record of the journal/ publisher "The journal has an established publishing track record." Quality of scholarship Criteria that refer to the quality of the content published, including article selection guidelines and the actual content of the published articles relating to objectivity, substantive merit, utility and importance "Editorial decisions should be based on the manuscript's originality, quality, and contribution to evidence about important questions."

Truth in branding
Criteria related to how the journal/publisher portrays itself to the public and its intended audience Clarity of journal title Criteria discussing the naming convention of the journal and whether or not it encourages confusion with other, more well-established journals "The Journal name shall be unique and not be one that is easily confused with another journal or that might mislead potential authors and readers about the Journal's origin or association with other journals."Editorial board Criteria related to the makeup, function, and credibility of the editorial board "The full names and affiliations of the journal's editors must be provided on the journal's website as well as contact information for the editorial office."Ethical guidelines Criteria pertaining to journals/publishers having clearly stated ethical guidelines on their interface (e.g.website) for authors "A journal shall also have policies on publishing ethics." Fees Criteria related to how journals/publishers display information related to their fee structures "Any fees for publishing must be clearly displayed in a place that is easy to find."Scope of journal Criteria discussing how journals/publishers align themselves regarding to their scope of coverage "Be relevant to the discipline areas."

Website quality
Criteria describing quality indicators related to journal/publisher websites "A journal's website, including the text that it contains, shall demonstrate that care has been taken to ensure high ethical and professional standards."How to use safelist Content related to target audience(s) for the Safelist, how the information should be used, how journals/publishers can be added to the list, and procedures or guidelines discussing how and who should be evaluating potential additions to the lists Multiple, independent assessments Criteria related to encouraging safelist users to utilize multiple sources in order to vet a journal/publisher, including examining metrics and indexes, professional membership, as well as using criteria from other organizations and peer advice Metrics and indexes accuracy Criteria related to accurate use of ISSNs, inclusion in external and credible metrics, and accurate display of relevant citation and impact metrics "The journal has a registered ISSN and eISSN." Professional membership Criteria pertaining to a journal/publisher's membership in a professional organization that is widely acknowledged to be credible "COPE, WAME, ICMJE, OASPA membership and/or other organizations regarding ethics in research and publication will be considered."Reliance on other organizations' evaluations Criteria pointing to other professional organization's vetting process and journal/ inclusion in that organization as a mark of credibility "We do not check the quality of the listed journals ourselves, but incorporate journals listed in the DOAJ, which undertakes a quality assessment of the journals."
a Previously called Cabells Blacklist.b Previously called Berlin Institute of Health-Open Access Journal Whitelist.c Previously called Cabells Whitelist.