Into the gray: A modified approach to citation analysis to better understand research impact

Academic authors and funders often want to know the “impact” of their publications, and this impact is generally judged by how and where the paper is cited in other academic works. This limited appraisal has been expanded in recent years as many are beginning to argue that nonacademic publishing venues should be included in assessing the impact of academic publications. This is an issue of particular concern with the growing emphasis on “knowledge translation” from the scientific literature to policy and practice applications 1–3 and to sources other than the traditional peer-reviewed and indexed venues, in other words, translation into the “gray literature” 4. 
 
In this comment and opinion piece, the authors describe the process of developing and applying a “modified citation analysis” that builds on existing methods of examining a research paper's impact in two key ways: (1) by deliberately including gray literature in the citation analysis search process, and (2) by including quantitative and qualitative methods of analysis to gain a better understanding of how a research paper was used. By broadening the search and deepening the level of analysis, we suggest this new approach can better assess the impact of a given research paper—both within and outside of traditional peer-reviewed venues. We begin with a review of gray literature and then describe current methods for analyzing the impact of a research paper. Finally, we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation.


Introduction
Academic authors and funders often want to know the ''impact'' of their publications, and this impact is generally judged by how and where the paper is cited in other academic works. This limited appraisal has been expanded in recent years as many are beginning to argue that nonacademic publishing venues should be included in assessing the impact of academic publications. This is an issue of particular concern with the growing emphasis on ''knowledge translation'' from the scientific literature to policy and practice applications [1][2][3] and to sources other than the traditional peer-reviewed and indexed venues, in other words, translation into the ''gray literature'' [4].
In this comment and opinion piece, the authors describe the process of developing and applying a ''modified citation analysis'' that builds on existing methods of examining a research paper's impact in two key ways: (1) by deliberately including gray literature in the citation analysis search process, and (2) by including quantitative and qualitative methods of analysis to gain a better understanding of how a research paper was used. By broadening the search and deepening the level of analysis, we suggest this new approach can better assess the impact of a given research paper-both within and outside of traditional peer-reviewed venues. We begin with a review of gray literature and then describe current methods for analyzing the impact of a research paper. Finally, we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation.

Gray literature
Gray literature is an important aspect of scientific evidence as it ''produc[es] and distribut[es] the seeds of new knowledge'' [5]. Current definitions of gray literature are overly broad and lack clear lines of distinction between different producers, users, and dissemination techniques (Table 1). A commonly cited definition is: ''Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing; i.e., where publishing is not the primary activity of the producing body'' [6]. Another definition proposed by the International Journal on Grey Literature is: ''the information and resources that do not categorically fall into what is available via standard, traditional or commercial publishing channels'' [7]. Even these well-respected definitions provide only a vague understanding of what is classified as gray literature.
The distribution of gray literature in a multitude of mediums by individual scholars and scientists, research institutes, and community and nonprofit organizations has become widespread since the advent of online repositories and other open access venues [8][9][10]. Organizations produce policy briefs, issue briefs, or technical reports on specific content areas (e.g., health, environment, and poverty) meant to inform decision making that may not enter formal publication venues. This can be attributed to a bias toward publishing larger studies with specific results (i.e., positive, novel, and generalizable), the considerable time lag between research production and publication [11], or the production of gray literature for internal organization purposes.
Gray literature as a key dissemination strategy. Gray literature remains underused and undervalued. However, its importance and relevance is becoming apparent in areas such as policy development and research [12][13][14]. In the medical field, the well-regarded Cochrane systematic evidence review process requires the inclusion of any relevant gray literature as part of the evidence base [ [4] argue that the exclusion of gray literature can influence the results of meta-analyses. In one example, by extending a search to include literature outside ''mainstream sources,'' an estimated 29.2% more sources were found [17]. Gray literature is also important in many fields of research in both understanding the theoretical underpinnings of findings [18] and identifying research gaps that produce research questions [19]. This is especially true in health research, where findings published in the gray literature often inform the development of clinical trials [16].
Gray literature is extremely valuable in providing timely communication on complex issues-often using simple, actionable, and comprehensive language-making it a key resource for stakeholders outside of academia [20,21]. These works often include important information on context, policy decisions, and public interest that are of particular value to decision makers [22]. It is apparent that while the amount and influence of gray literature is dramatically increasing, quality assurance, search methods, and impact measures for gray literature have not kept pace.
Quality of gray literature. The sheer volume and accessibility of gray literature also raises the issue of the quality of these resources [23]. With the general absence of peer review, the only way to ensure the quality of gray literature is through critical assessment. Although the producers of gray literature may be experienced and knowledgeable in their field, there is essentially no formal quality control. There have been suggestions to improve the quality of the more ''traditional'' types of gray literature such as technical and evaluation reports, strategic plans, policy briefs, and others. An example is to include standard information in documents such as: (1) the process of creating the product, (2) the review process (if any), and (3) any conflicts of interest [24]. The Grey Literature International Steering Committee provides guidelines on what scientific and technical reports should look like. They should include a peer-review process, ethical considerations, and publishing and editorial recommendations [25]. However, given the unregulated space that gray literature inhabits, it is unrealistic to expect all who create it to conform to such standards, leaving the end user responsible for assessing its quality.
Access to gray literature. Searching for gray literature is challenging and can be time consuming. There is currently no accepted strategy or protocol for doing so. While some gray literature becomes incorporated in the published literature and vice versa, much gray literature remains ''fugitive'' [26] and difficult to capture. Since gray literature falls under the broad definition of ''materials not published in a traditional way,'' it can be found in numerous places and requires many different search strategies to collect it. Al-though some of the commercially available databases, such as Web of Science and Scopus, contain gray literature (e.g., dissertations, conference proceedings), this is not their primary function, and they likely capture only a fraction of existing material. Google Scholar is a good adjunct to these but similarly does not have a primary focus on gray literature. As its search algorithms are unknown, it is difficult to understand and assess what has been included and excluded from searching Google Scholar.
Another challenge with gray literature is the lack of bibliographic control or classification [27]. Unlike scholarly works such as journal articles, most gray literature is not indexed in databases. Although some databases do include gray literature, poor or inconsistent choice of titles or keywords can make such documents difficult to source, catalog, and store [28]. The ''scattered'' and disorganized nature of gray literature presents particular difficulty for those wishing to incorporate it into a systematic and comprehensive search strategy, as is required for many types of literature review.

Current methods for analyzing citation impact
Several methods of analyzing individual study citations are currently used to determine impact in terms of penetration into the broader academic literature. These include methods such as citation analysis, co-citation analysis, bibliographic coupling, and a co-citation-bibliographic coupling hybrid. In general, the purpose of these types of analysis is to track citations of publications and, in some methods, group similar papers together. Although these methods of impact analysis are widely used, they fail to capture gray literature, and all are based solely on counts or quantitative comparisons of citations. That is, none delve deeper to explore how (as opposed to simply how often) a research paper has been used.
A relatively new method of citation analysis is the Becker Medical Library Model for Assessment of Research (Becker Model), which analyzes research impact through indicators such as ''contribution to the knowledgebase'' and ''change in practice'' by tracking diffusion of research output and activities [29]. The Becker Model suggests locating gray literature materials such as non-peerreviewed journal articles, trade publications, and supplemental materials but does not assist in locating these items, and it still relies solely on quantitative counts of research outputs, albeit of more types. (For more information on this approach, visit https://becker .wustl.edu/impact-assessment.) Modified citation analysis. We were interested in tracking the impact of a specific research article published in a high-impact, peer-reviewed journal [30]. This was the first randomized controlled trial (RCT) of universal screening in health care settings for exposure to intimate partner .) After reviewing current methods for citation analysis (as above), we determined that none of these could effectively track the influence of a single paper through both commercially published and gray literature. For this reason, we developed a ''modified citation analysis'' method that would capture both traditional and gray literature and allow both quantitative and qualitative interpretive analysis on extracted data. Such an approach was important because (1) we knew that the content of the paper was of interest and contentious in the academic realm and beyond; therefore, limiting our search to only commercially published sources would not provide an accurate portrayal of how the findings were used, and (2) we were interested in how the paper was interpreted and used; therefore, citation counts alone would be insufficient.
We searched the scholarly and academic and gray literature for sources with high potential to influence policy, practice, or further research. Since we were primarily interested in the various uses of the study findings, we did not assess source quality or perform any formal results synthesis of the included materials. Preliminary scans of the literature helped determine the most effective subject headings and keywords or phrases to use for these searches. These terms were then applied using the appropriate database or other online search tools to retrieve articles, documents, media reports, or other items that specifically cited or discussed the MacMillan et al. paper [30]. All retrieved results that met inclusion criteria were stored in a reference management program (Figure 1).
The gray literature was searched using multiple resources, such as multidisciplinary databases (Scopus), websites (MedlinePlus), and point-of-care tools (MDConsult, Up-ToDate). Major health care associations and professional organizations likely to include related content were identified, and their websites were individually searched. News media reports were searched by using news databases, including Factiva, Lexis Nexis, Google News, and Proquest Canadian Major Dailies.
''Traditional'' citation analysis statistics were computed using SPSS 20.0. These included the number of sources citing the MacMillan et al. paper [30] (by search method) and the number of MacMillan et al. [30] citations per source. Data analysis took place in three steps according to the specific research questions guiding the analysis: (1) analyzing content of text extractions, (2) categorizing sources regarding their position on IPV screening, and (3) coding sources regarding how they defined IPV screening. Table 2 illustrates our coding process.
Adapting this approach. This method is theoretically adaptable to other contexts where tracing the uptake and interpretation of new knowledge would benefit from more than quantitative citation counts and inter-linkages. The gray literature search strategy would need to be specifically tailored to the area of interest and could involve more extensive searching depending on the discipline and topic area. For this reason, we recommend that experts in the field be involved in the process and that the searches be undertaken by trained information professionals.
In addition to a tailored search strategy, the quantitative and qualitative coding of the text citing the original document will also depend on the specific questions being addressed. Some codes may apply to any paper; for example, anyone might code whether the paper is cited for a methodological or theoretical point, or might examine whether the citation appears in the ''Introduction,'' ''Method,'' or ''Discussion'' sections. Most codes, however, will be highly specific to the goals of the analysis.

Discussion
Our modified citation analysis has several advantages over more mainstream methods of citation analysis. By expanding our search strategy, we were able to find more citing sources. Specifically, using search techniques designed to track citations in the commercially published academic literature (e.g., Google Scholar's ''cited by'' tool and Web of Science), we found eighty sources citing the MacMillan et al. paper [30]. The gray literature searches (e.g., Google, news databases, and association websites) found an additional twenty-nine sources. Second, by moving beyond basic citation counts and coding how the paper was cited, we were able to provide a richer understanding of its impact on subsequent research, as well as policy and practice guidance regarding screening for intimate partner violence. It also provided a way to assess the evolution of the main debates in this area (for example, confusion caused by varying definitions of ''screening'').
This method requires additional resources. The much broader range of potential search venues demands more time and expertise. Delving into gray literature is a challenging task and requires planning and coordination, including consideration-specific inclusion/ exclusion searching. Unlike database searching, common nomenclatures rarely exist for searching diverse gray literature sources; therefore, the concept of consistency in search terms across sources is difficult to achieve. It is important to document all search strategies used in each source to attempt reproducibility in the ever-changing nature of online resources. Similarly, the addition of a qualitative analysis component necessitates multiple researchers to undertake coding, analysis, and interpretation that is rigorous and adheres to qualitative methodology standards [32]. More research using this method is required to assess its validity and utility across different applications.

Conclusion
The modified citation analysis approach appears well suited to report how new research evidence is taken up, interpreted, and used in published material, a quality that is valuable to those attempting to assess the effectiveness of knowledge translation strategies [3,33]. We believe it is a useful extension of traditional citation analysis methods and addresses some of the gaps in the proposed Becker Model [29]. Since there are no validated methods for conducting this method of citation identification, we created a new search and analysis strategy, building on the growing importance and acknowledgment of searching gray literature [34,35].
This new approach adds to the ongoing discussion regarding the All extractions were read independently by 2 researchers. Coding framework with 9 thematic codes and 26 subcodes a. All extractions coded using coding framework All extractions were read and coded independently by 2 researchers. b. All extractions referencing ''no harm '' or ''no benefit'' finding were further coded.
When extractions were coded as having cited the trial for the ''no harm'' or ''no benefit'' finding, we further coded to see if explained by the authors.

Consolidated coding
Researchers worked together to create one set of coded work. Disagreements resolved through discussion.
One set of coded data to be inputted in SPSS for frequency analysis II Coded source as either focused or not on intimate partner violence (IPV) screening Each source was examined on the whole to see if the focus of the source was IPV screening.
Sources that did have a focus on IPV screening were counted Coded sources support screening or not Each source was categorized into one of four categories: supports screening, does not support screening, unclear, or no stance. III Explore explicit definitions of screening Each source was coded as containing explicit definition or not; explicit definitions extracted.
A list of explicit definitions for screening Implicit coding of screening definitions Each definition was read and coded inductively (coding list was created based on data).
inclusion and analysis of gray literature resources in searching for different types of ''evidence'' to include in both academic research, practice, and policy contexts. We hope that it encourages a more holistic approach to assessing ''impact'' of new research evidence.