Retracted Publications in Mental Health Literature: Discovery across Bibliographic Platforms

INTRODUCTION Retractions are a mechanism by which science corrects itself, withdrawing statements or claims that have proven to be erroneous. However, this requires that such corrections be displayed clearly and consistently. This paper considers how retracted publications in the mental health literature are represented across different platforms. METHODS Using Retraction Watch, we identified 144 retracted articles in the mental health field. We looked across seven platforms to determine the consistency and clarity of the retracted status of these publications. RESULTS Of the 812 records for retracted publications, 40.0% (n=325) did not indicate that the paper had been retracted. Of available PDFs, 26.3% (53/201) did not indicate that the paper had been retracted. Of the 144 articles studied, only 10 were represented as being retracted across all resources through which they were available. DISCUSSION Retracted publications in this sample were inconsistently represented across library resources. While technical solutions, such as Crossmark by Crossref, may help mitigate these challenges, the inconsistent display of retractions has implications for education and outreach. CONCLUSION Our study found that the retractions in our sample were not clearly and consistently represented across sources. Libraries, which provide access to and training in these resources, have a responsibility to raise awareness of these inconsistencies and to advocate for more timely and accurate metadata.


INTRODUCTION
Articles may be retracted for a wide range of reasons, from self-acknowledged error to data falsification. In and of themselves, retractions are not problematic; retraction provides a mechanism through which science corrects itself. However, to be effective, retraction notices must be clearly and consistently represented wherever retracted material may be found. In the case of the mental health field, a multidisciplinary area encompassing both basic and clinical health and social sciences, individuals may access literature through a variety of different disciplinary and multidisciplinary resources. An inconsistent representation of retractions in these resources poses challenges in advancing research and presents potential threats to patient care.
Adhering to best practices in the representation of retractions is an ethical issue in publishing that requires the attention of all individuals involved in the production and distribution of scholarly journals. This article considers the extent to which guidelines established by the Committee on Publication Ethics (COPE) are being implemented or reflected in the mental health literature and the bibliographic databases through which users access these resources.

LITERATURE REVIEW
Retracted publications are considered a rarity, making up only an estimated 0.02% of the biomedical literature (Grieneisen & Zhang, 2012). However, the rate of retractions has risen notably in recent years, with retraction notices in MEDLINE increasing 41% between 2014 and 2015 compared to the number of notices in 2013 and 2014 (U.S. National Li-brary of Medicine, 2016). Papers can be retracted for a range of reasons. Most commonly, papers are retracted due to misconduct, including fraud and plagiarism, or error (Fang, Steen, & Cadadevall, 2012). Only 9.8% of retractions are due to fraud; however, this percentage has increased tenfold since 1975 (Fang et al., 2012).
Retracted publications may call into question the validity of research based upon their contents. However, as Steen (2011) notes, "harm from a retracted study could also arise by influencing the ways in which patients are treated" (p. 688). Although the information -seeking behaviors and attitudes of physicians are well documented (Dawes & Sampson, 2003;Mikalef, Kourouthanassis, & Pateli, 2017;Schuers et al., 2016), less research is available on the information practices of mental health professionals. Mental health is not a single discipline, but rather refers to a multidisciplinary area that encompasses bench research, such as pharmacology and neuroscience, as well as clinical and social science research, such as social and educational psychology. Mental health scholarship was chosen for the present study not only for its breadth, but also for its implications for research and practice. Students and researchers working in health care professions, psychology, public administration, and social services all interact with mental health literature in their academic and professional lives. The U.S. Department of Education (2016) estimates that between 1970 and 2015, over 7 million university degrees-approximately 18.5% of all degrees granted in the United States-were issued in these subject areas. Moreover, the work of these individuals may impact the 17.9% of Americans who deal with mental health issues annually (Center for Behavioral Health Statistics and Quality, 2016).
In the predigital era, librarians were more engaged in educating users about retracted articles. In a 1998 survey, 41% of surveyed medical libraries in Canada and the United States reported a practice for calling attention to retracted publications, including annotating the front page of retracted articles, annotating the first page of the issue in which the article was published, and keeping a list of retracted articles at the circulation desk (Hughes, 1998). Similarly, a study conducted by Duggar et al. (1995) addressed library practices for identifying and raising awareness of retracted articles. However, this area of practice has become less prominent in recent years. Libraries remain committed to facilitating broad access, but established practices could not be adapted for digital resources and new practices did not emerge to take their place.
While practices for handling retractions of electronic resources may not have emerged among librarians, best practices have been developed for publishers. In 2009, the Committee on Publication Ethics (COPE) established guidelines for the publication of retractions, corrections, and notices of concern (Committee on Publication Ethics, 2009). COPE advocates for a consistent experience, arguing that retraction notices should be published both in print and electronically and that retracted articles be clearly identified both in the journal and any bibliographic databases (Committee on Publication Ethics, 2009). Clarity of retractions is also of concern, as COPE notes that retraction notices should be clearly identified and linked to the retracted publication in all electronic versions (Committee on Publication Ethics, 2009).
• Retraction notice characteristics. Retraction notices should include identifying information from the retracted article (e.g., title, authors). Retractions should be clearly identified and distinct from other status changes (e.g., corrections) and "editors are responsible for ensuring that retractions are labelled in such a way that they are identified by bibliographic databases" (p. 2).
• Retraction notice accessibility. Retraction notices should be indicated on all electronic sources such as "on the journal website and any bibliographic databases" (p. 2). Notices should link to the retracted article and "should appear on all electronic searches for the retracted publication" (p. 2).
• Retraction notice timeliness. Retraction notices should be "published promptly to minimize harmful effects from misleading publications" (p. 1).

Figure 1. Summary of Retraction Guidelines from the Committee on Publication Ethics
Despite these guidelines, the use of retracted literature remains an issue. Chen, Hu, Milbank, and Schultz (2013) found that "many retracted articles are highly cited with hundreds of citations and are often part of active areas of research," and note that articles may unknowingly "extend a citation trail originating from a retracted article" (p. 252). This statement supports previous findings that "fewer than 5% of the citing papers indicated any awareness that the cited article was retracted or named in a finding of misconduct" (Neale, Dailey, & Abrams, 2010, p. 261).
Enhancing the transparency of the scholarly communication system-"the system through which research and other scholarly writings are created, evaluated for quality, disseminated to the scholarly community, and preserved for future use" (Association of College & Research Libraries, 2003)-requires critique of that system, particularly when considering the practical implications for information behavior.

METHODS
This study draws on consistency inspection, a well-established method in human-computer interaction, in which one or more individuals review a single product or series of products to ensure a consistent experience for the end user, and to identify both the presence of usability issues and the severity of those issues (Wixon, Jones, Tse, & Casaday, 1994). In usability testing, consistency inspection is largely focused on interface design and functionality. We instead utilize this method to examine the consistency of the data being represented in these systems by examining the same records across multiple interfaces. In this research, we consider the extent to which COPE guidelines are being consistently applied across multiple library resources when accessing mental health literature.
We identified a sample of publications by mining Retraction Watch, a website run by the nonprofit Center for Scientific Integrity that publishes information on retracted publications, using a set of key terms that were developed through a review of the website (Center for Scientific Integrity, n.d.). These terms included standardized terms established by Retraction Watch (psychology, psychiatry, neuroscience, neuroscience retractions, neurology retractions, drug design, sexual and marital therapy, and sociology), as well as searching across the Retraction Watch website for the terms "mental health," "counsel*," "social work," "psycholog*," and "psychiatr*." Results were deduplicated; and corrections, notices of duplicate publication, and expressions of concern were removed. The aim of the search was to develop a broad sample of retracted publications in the area of mental health research rather than to identify all retracted publications or create a comprehensive list.
Our search strategy yielded 144 retracted journal articles, published between 1996 and 2015.
On average, articles were retracted 3.75 years after publication, with retraction dates ranging from 2010 to 2016. These articles were produced by 29 different publishers and published in 76 different journals. These journals spanned a variety of disciplines and represented 31 different categories in Journal Citation Reports, to use one familiar index, ranging from Mathematical & Computational Biology to Family Studies to Business (Clarivate Analytics, 2016).
Between June 27, 2016, and July 8, 2016, we conducted known-item searches for retracted articles and any related retraction notices in the following resources: (1) publisher sites, (2) MEDLINE via Ovid, (3) PsycINFO via Ovid, (4) EBSCO databases, (5) Scopus, (6) Web of Science, and (7) PubMed. Resources included both individual databases and aggregators in a range of disciplines to represent the multidisciplinary nature of mental health research. Because our study was focused on publisher practices around retraction transparency and discovery, we excluded unofficial sources of these articles (e.g., academia.edu and ResearchGate) from the data-gathering process.
We searched for each of the 144 articles and recorded 1,013 observations during the twoweek period. We then excluded observations from resources other than the aforementioned seven, as well as observations recording the absence of the article in the resource. Our analysis included 812 records. Using the themes we identified in the COPE guidelines, as shown in Figure 1, our specific questions are outlined in Figure 2.

Retraction Notice Characteristics
Is the fact that the article was retracted shown in the record? If so, where (e.g., in the title, in the abstract)? Is the retracted status indicated in any available PDFs? If so, where?
Reported in Rows 1-4 and "Clarity of Retracted Status" section

Retraction Notice Accessibility
If there is a separate retraction notice available, is that notice linked to the original article? Is it indexed with subject headings to ensure discoverability?
Reported in Rows 5-6 and "Presence of Retraction Notices" section

Retraction Notice Timeliness
As specific guidance is not offered surrounding what would constitute prompt publication, it was not possible to measure consistent application of this guideline.

Presence of Retraction Status
Of the 812 records studied, 487 (60.0%) indicated that the article had been retracted.

Clarity of Retraction Status
Of the 487 records which showed a retraction, 311 (63.9%) indicated the retraction in only one place. Where the retraction was indicated once, it was most commonly found in the title (n=139, 44.7%, see Fig. 4) or the abstract (n=124, 39.9%, see Fig. 5). In the 176 instances in which the retraction was indicated in more than one place, publication type (n=136, 77.3%, see Fig. 6) was the most commonly used field, followed by title (n=105, 59.7%), a note or banner (n=80, 45.5%, see Fig. 3), and abstract (n=37). Of available PDFs, 73.6% (148/201) show that the articles have been retracted. In 79.7% of cases (118/148), the retraction is noted in one place on the PDF, and in 20.2% (30/148) of cases, the retraction is noted twice. Where the retraction was noted once, this was most commonly done by writing "Retracted" across all PDF pages (n=82, 69.5%, see Fig. 7). This was also the most common strategy when the retraction was noted more than once (n= 30, 100.0%). Other strategies included adding the retraction notice to the beginning (n=28) or the end of the article (n=19). In 17 instances, the original article was removed and replaced with the retraction notice.

Presence of Retraction Notice
Of the 812 records included from these seven platforms, 672 (82.8%) were associated with separate retraction notices in the same resource. This ranged from 66.4% (77/116) in EB-SCO resources to 86.8% (125/144) of articles available through publisher sites. Of the 672 article records associated with retraction notices, 233 (34.7%) of those article records did not indicate that there had been a retraction. Of articles with separate retraction notices, 54.8% (n=368) of notices were not indexed with subject headings or linked to the original article (see Fig. 8).

DISCUSSION
Our findings show inconsistent representation of our sample of publications across platforms. Of the 144 articles studied, only 10 were represented as being retracted across all resources through which they were available. There was no platform that consistently met or failed to meet all of COPE's guidelines. This inconsistency poses challenges for end users accessing the content, as their experiences may vary widely depending on their point of access.
Given the significant overlap in content between some of these resources, the discrepancies found within this sample are surprising. PubMed and MEDLINE via Ovid provide access to the same underlying publication data, the vast majority of which is also integrated into Scopus (De Groote, 2000;Falagas, Pitsouni, Malietzis, & Pappas, 2007). This suggests there are potential limits to interoperability and the workflows that update article records. The challenge posed by outdated metadata and documents is exacerbated by the availability of multiple access points and copies.
Databases that presented article records with retraction notices did not do so consistently. Little guidance is available on how retractions should be represented in the original article and how those records should be modified, whether from COPE or another source. However, literature on human computer interaction may provide some insight. Ohno (2004) found that title information tended to attract the most attention from users of digital documents. This supports previous research regarding library catalogs which found that titles, abstracts, and keywords were the metadata elements most frequently used by patrons in determining relevance of the item (Janes, 1991;Marcus, Kugel, & Benenfeld, 1978;Wang & Soergel, 1998). While this is not specific to retractions, the finding that titles attract the greatest attention, followed by abstracts and keywords, may provide some indication of how retracted publications should be annotated.
Both COPE and the International Committee of Medical Journal Editors (ICJME) indicate that the retraction notice should include the title of the original article in the title of the retraction notice, should be discoverable in all electronic searches for the item, and should include information on who has retracted the paper and for what reason (Committee on Publication Ethics, 2009; International Committee of Medical Journal Editors, n.d.). Detailed information not only facilitates transparency, but encourages the self-correcting nature of science, as research has shown that insufficient detail in correcting misinformation may be associated with an ongoing belief in that misinformation (Johnson & Seifert, 1994;Wilkes & Leatherbarrow, 1988). ICJME also guides editors to place retraction notices in prominent sections of the journals and to include them in tables of contents. While our study did not consider where retraction notices were found within the context of the jour-nal issue, research regarding the circulation of inaccurate news reports has found that "the retraction will need to be circulated with equal vigor [to the original report], or else continued influence will persist at high levels" (Ecker, Lewandowsky, Swire, & Chang, 2011, p. 577).
Facilitating the discovery of retractions may not simply be matter of further developing users' search skills. Previous research has noted that "users of bibliographic databases and online journals should be aware that there is variability between the databases in timeliness of annotating retractions and between journals in clarity of annotations" (Wright & McDaid, 2011, p. 166). Given that there were challenges in retrieving retraction notices even when conducting known-item searches, it would seem unlikely that users would discover these retractions during their information-seeking processes. The information-seeking context of the health professional is a necessary consideration, as research has shown that clinicians pursue answers to approximately 51% of their questions and on average spend less than 3 minutes searching for the answer to each question (Del Fiol, Workman, & Gorman, 2014). Additional search steps to ensure accuracy are likely outside of the average health professional's workflow.
For mental health professionals, the ability to locate, appraise, and synthesize evidence to be applied in patient care is a core competency required for certification and licensing (American Board of Counseling Psychology, 2016; American Board of Psychiatry & Neurology Psychiatry, 2011;Council on Social Work Education, 2015). Intersections between scholarly communication and information literacy have been well articulated in previous literature (Duckett & Warren, 2013;Gelfand & Palmer, 2013;Gilman, 2013). However, the representation or misrepresentation of retracted publications creates opportunities to frame conversations around critical appraisal, discovery, and information ethics.
Inconsistency of platforms in identifying retractions could be connected to the themes of ACRL's Framework for Information Literacy for Higher Education (Association of College & Research Libraries, 2015). The ACRL Framework is based on threshold concepts, or ideas that pose significant challenges to the learner and act as a portal to a greater or more nuanced understanding of a particular practice or discipline. In this context, the discipline could be considered scholarly publishing. Encouraging skepticism of the peer review and publication process could be a means of conferring authority upon the researcher and his or her work, and reinforcing the necessity of "question[ing] traditional notions of granting authority" (Association of College & Research Libraries, 2015, p. 4). Although our study examines the representation of retractions across resources, library instruction in scholarly communication could be extended to explore the rationale behind retractions, including the challenges of the publish-or-perish incentive system in the academy (Baker, 2016), best practices for peer review, guidelines such as those from COPE and ICJME, and the range of terminology associated with retractions, from expressions of concern to errata.
A COPE Working Group has proposed eliminating the word retraction and instead operating under an "amendment model," which considers scholarly publications to be living documents that may have multiple post-publication versions, which would be denoted as either "insubstantial," "substantial," or "wholesale/complete" (Barbour, Bloom, Lin, & Moylan, 2017). The aim behind such work is to destigmatize post-publication corrections and enhance transparency by outlining the extent of and reasons for the amendment. Encouraging transparency in the scientific process is admirable. However, the existence of multiple versions of an article, each of which would have its own DOI or URL, could pose significant challenges for these downstream access points, particularly those that are already inconsistently representing retractions and retraction notices.
While awareness-raising activities are of value, technical means of addressing the challenge of retractions have also emerged. As new tools are created to identify retractions, the academic library community will need to continuously assess the benefits and limitations. Several tools and resources have emerged that show potential in helping users to identify retracted articles: • Rutgers University hosts a Retractions Database, which allows end users to search by author, journal, and a variety of other fields (Tallau, 2011). When searching for a particular item that has been retracted, two results will appear: one for the original article, and one for the retraction notice. However, as documentation regarding this resource is minimal, it is not possible to confirm where the data are drawn from and how the retractions are identified. Perhaps most important, the database does not appear to have been updated since 2011.
• The Center for Scientific Integrity is developing a database of retractions. The goal is to "develop the technical infrastructure needed to support existing and new Retraction Watch content" while enhancing discoverability in a searchable database (McCook, 2015). In December 2016, Retraction Watch shared a link to the Retraction Watch Retraction Database (beta), which allows users to search in many fields including author, DOI, and journal (Oransky, 2016). In the current iteration search results display basic bibliographic information such as the authors, article title, and journal title as well as information specific to the retraction, including the reason(s) for the retraction, original paper DOI, and retraction notice DOI. At the time of writing, a thorough evaluation of the tool is not possible, as it is still in beta.
• Open source solutions are in development. One example is Open Retractions (openretractions. com), which allows users to search for retracted articles by DOI. This tool is only effective when status changes are reported in publisher metadata to PubMed or Crossref. Open Retractions is "an API and web interface to check whether a paper has been retracted" (Smith-Unna & Smith-Unna, 2017). This tool and others need improvements to adequately capture retractions and other status changes. During test searches for two retracted publications in August 2017, DOI searches did not reveal the retracted status of these items, despite the fact that PubMed records for the articles were marked as retracted.
While the development of these databases indicates both an interest in and a desire to enhance the discoverability of retractions, databases such as these have been criticized as "impractical" in that they require additional searching on the part of researchers in addition to the initial identification of articles (Grieneisen & Zhang, 2012). The nonprofit Digital Object Identifier Registry Crossref has introduced a more proactive approach. Crossmark, a service of Crossref, is a tool that provides a logo revealing if a document is the current version or if an updated version exists. An update would include a retraction notice as well as other status updates such as corrigenda or corrections. For the user, Crossmark logos on documents introduce inconsistency, because a publisher would need to be participating in Crossref in order to display the Crossmark logos within search results. Also required in order to display a Crossmark logo is a Crossref DOI for each article (or other material type). Complicating matters further, a Crossmark logo will be visible on some publisher pages but not in other commonly used databases and aggregators like EBSCO, because Crossref membership requires that an organization publishes content (Crossref, 2017a). Several publishers represented in our study are members of Crossmark, including Elsevier, Springer, and Oxford University Press. Publishers participating in Crossmark must pay per-item fees and edit content to reflect status updates, and a subject or multidisciplinary platform may not be eligible based on Crossref 's criteria (Crossref, 2017b).
Our findings represent a snapshot of the representation of retractions within the mental health field. COPE does not provide an exact time frame for the dissemination of retraction notices, instead recommending that retractions "be published promptly to minimize harmful effects from misleading publications" (Committee on Publication Ethics, 2009, p. 1). Since data collection, further enhancements to records may have been made. Future research should attempt to determine the time frames in which retractions are being made available and develop more precise recommendations for the prompt correction of research. Moreover, although our sample was multidisciplinary, it is specific to the field of mental health and the findings cannot be extrapolated to all disciplines. Future research should consider other disciplines and disciplinary resources.
As a large, research-intensive institution, our level of access is greater than that of many institutions worldwide. As a result, we were able to triangulate and determine whether an article has been retracted in a way that an institution or individual with more limited access could not. Our experiences will differ significantly from those of individuals at other institutions. Future research should consider consistency of access in smaller institutions and outside of academia.

CONCLUSION
Our study found that the retracted publications in our sample were not clearly and consistently represented across sources and that the COPE retraction guidelines do not appear to have been adopted consistently in these cases. As information professionals we are uniquely positioned to understand the complexities of this environment and communicate these challenges to both consumers and producers of scholarly content, as the inconsistent representation of retractions may have significant implications both for research and practice. Librarians involved in content production might consider how their own systems and practices reflect COPE guidelines, both in the initial publication and in downstream resources, as the ways in which these systems are transmitting information may be problematic. Librarians should educate their users about the complexities of retractions as a component of both information literacy and scholarly communication outreach activities.