Increasing the efficiency of medical research article searching for the public and experts by ‘‘shepardizing’’: a lesson from legal research databases

A number of factors distinguish online medical journal data (such as MEDLINE/PubMed and Scopus) from legal research databases (LexisNexis and Westlaw); however, one major difference that could substantially improve the experience of users of medical journal databases is the adoption of a technique from legal research databases known as “shepardizing” 1–4.

A number of factors distinguish online medical journal data (such as MEDLINE/PubMed and Scopus) from legal research databases (Lexis Nexis and Westlaw); however, one major difference that could substantially improve the experience of users of medical journal databases is the adoption of a technique from legal research databases known as ''shepardizing'' [1][2][3][4].
Courts are the source of most information in legal research databases. Because courts challenge and overrule the views of prior legal cases frequently, these databases must keep track of which views of the courts are current and which ones have been changed or overruled. As a result, legal information has a hierarchy, which is reflected in legal research databases. The process that is used to track these constant changes is generally referred to as ''shepardizing'' [5]. It is named after Frank Shepard, who began publishing lists of cases that were cited by courts in the late 1800s for various jurisdictions [6]. The unique aspect of Shepard's lists were that not only did they provide citations for every case in different jurisdictions, but the lists also included an indication of how the citation was being affected (negatively, positively, or not at all) by the case that used it as a reference. In other words, a lawyer conducting research on a particular issue can use Shepard's citation list to determine if any references the lawyer wants to rely on are good law or not. Over time, words that Shepard used to describe how citations were affected by the evolving opinions of courts became easy-to-understand symbols in electronic databases.
Today, LexisNexis controls the Shepard'sH Citations Service, and their main competitor, Westlaw, offers a similar service called Key-CiteH [7][8][9][10]. Both services are called ''citators.'' Their purpose remains to determine whether a citation is good law or has been overruled [11]. In this manner, the relevance of this citation information for legal research and scholarship has not changed in over 100 years. Today, legal research databases allow anyone searching legal cases to easily determine if a case is good law or if it has been overruled partially or completely. This approach to managing the validity of cases allows people to easily identify which cases represent current legal thinking and which do not. Instead of engaging in a lengthy search for information that requires reading every case that a person finds on a research topic, that person is able to quickly focus on the most relevant cases and more importantly exclude those that are no longer useful.
Despite the number of years that this approach has been used in the legal field and the efficiencies that arise from this approach, medical journal databases have not adopted a similar tool. In fact, the author searched PubMed for ''shepardizing,'' ''KeyCite,'' and ''citator'' and found no relevant articles. Hence, the use of these tools in the field of medicine has been overlooked by scholars and journal database companies. The remainder of this comment and opinion piece discusses the potential benefits for the public, researchers, clinicians, and graduate or professional students if this technique were incorporated into medical journal databases.
In contrast to the organized approach of managing the evolving landscape of legal knowledge, medical journal databases treat all articles as though they are equivalent. This approach is not efficient because of the enormous amount of medical research information being produced monthly by the medical research enterprise worldwide and the likelihood that some of this information will not be credible [12]. For example, a simple search of ''drug and diabetes'' on PubMed will generate a list of more than 100,000 articles, many of which are outdated. However, there is no indication in the database of which studies have reliable, current knowledge and which ones do not. Instead, members of the public and even researchers have to read each article's abstract and, in some cases, the entire article to figure out whether it represents a current viewpoint or not. The date of the article can help, but the date alone does not mean that an article reflects the current, best information on a topic. For nonexpert members of the public, trying to figure out what is useful is nearly impossible, especially since the information is written in highly technical language. Nonexperts may assume all information available on the database is still valid today. The process poses similar challenges for students researching topics. Even for experts who are exploring new topics, this approach can be time consuming and highly inefficient.
The stated difference between legal and medical research databases is a critical one because medical information also has a hierarchy of evidence, but the hierarchy is not reflected in journal databases. Researchers and clinicians update their knowledge throughout their careers, but this updating process is not reflected in the databases they use to gather information, unlike legal research databases, which reflect the evolution of the field. Also, despite the fact that experts update their knowledge regularly, no one can keep up with research in every field. Thus, searching medical research information is taxing when experts explore a new topic area because there is no clear indication of which research articles represent the best information on any given topic. The time-consuming nature of finding good articles can be a hindrance to researchers developing innovative, interdisciplinary ideas for research projects. This information can be discovered by reading an article and its relevant citations, but this is quite time consuming. This inefficiency that occurs during the initial phase of research (idea formation) can slow down the process of medical innovation. ''Medical shepardizing'' could speed up the process of acquiring new information on a medical research topic and thereby enhance the speed of this initial stage of medical innovation.
A medical journal citator could use a similar, but not the same, approach as a legal citator. A medical journal citator could have two major components: (1) an assessment of the quality of the evidence and (2) a statement indicating whether the article is reliable or has been challenged or ''overruled'' (partially or completely) by another article. In addition, the medical journal citator should provide links to any articles that have challenged or overturned (partially or completely) the citation, an approach used in legal information databases. These features would make searching medical article databases much more efficient than the status quo approach. Numerous standards for assessing the quality of medical evidence can be used, such as the Institute for Clinical Systems Improvement's evidence grading system [13,14]. Simplifying this system even further so that members of the public can easily understand the ratings would make it more valuable. For example, if the evidence grading approach could be translated into a familiar ''A, B, C'' grading system, it would be more accessible to the public. The information on whether an article has been challenged could also be translated into an easy-to-understand rating system such as: (1) red (do not use, new data have proved this article to be invalid); (2) yellow (article challenged); or (3) green (not challenged, reflects current thinking). Although researchers and clinicians are trained to compare studies, patients are not, and to the extent that these databases are supposed to be useful for patients and their families, an easy method of comparing studies and figuring out whether a study is still valid would be a marked improvement from the current databases. In addition, user friendly approaches would aid professional and graduate students in health care fields who are learning to assess what research evidence can be relied on.
While there is no medical journal citator currently in use, there are ongoing efforts to identify the best medical research on certain topics. Some of the commonly used approaches include Cochrane Reviews, the Database of Abstracts of Reviews of Effects (DARE), clinical guidelines, updated clinical information on UpToDate, and systematic reviews conducted by research teams throughout the country [15][16][17]. Also, clinical filters on medical journal databases (such as PubMed) can help researchers narrow down article searches to certain types of research, such as randomized controlled trials [18]. While these approaches are all extremely useful for the medical research enterprise and clinical medicine, they do not provide the same information that a medical citator would. Systematic reviews are static and limited in scope compared to medical citators. They are only good for a limited period of time and are often only conducted every few years for a given topic. When new medical research arises, it is not incorporated into systematic reviews because they are so labor intensive. They require much more information to be generated before another review can take place. Also, they usually only cover topics in which a robust body of evidence exists, which limits their utility to certain questions. By contrast, a medical citator would be updated continually because it tracks the evolution of medical information and would cover all research topics. Clinical tools such as UpToDate come closer to medical citators in that they are updated regularly. However, they focus on providing information to clinicians and often do not contain much of the research that is being produced on various topics. Also, they do not identify which studies reflect current information and which ones do not. They only provide a few of the best studies on any given topic. UpToDate is also not easily accessible to the public and not used by most biomedical researchers who draw their current information from academic journals. Finally, filters that are used for medical journal databases return a list of articles without any indication of how useful the articles are. Even with a filtered list of research articles, searching is much more efficient when labels are attached to articles indicating if the information has been challenged or opposed by some other article. A citator would provide this information, whereas a filtered search would not.
A key concern for a medical citator would be who determines which articles are being challenged or ''overruled.'' I suggest that this should be a collaborative approach involving all the key stakeholders impacted by this process. As stated, standards already exist for determining the quality of the evidence. Standards for determining whether an article is challenged or partially or completely invalidated by another article could be developed by entities that maintain medical research article databases, such as the National Library of Medicine or private companies. Journal editors and article reviewers could follow these standards, and the staff of entities that maintain the databases could assess whether the standards are being applied appropriately. Authors of articles, journal editors, and article reviewers could jointly determine how older articles are being impacted by new articles that reference the old articles. For example, authors could clearly point out how their articles challenge evidence provided by older articles when they submit their papers for publication. After review by entities that maintain the databases, these older articles could then receive a designation that indicated in a userfriendly way how they should be viewed (red, yellow, or green). Obviously, there would be technical hurdles to address in order to implement a medical citator system.

Comment and opinion
However, the use of this approach in the legal field for years demonstrates that it is technically feasible.
The incorporation of a medical citator into medical journal databases could make identifying the most useful medical articles more efficient for the public, researchers, clinicians, and professional or graduate students, just as it makes identifying the most useful legal cases more efficient. Over time, this approach could further enhance the value of these already well-utilized databases and allow researchers to spend more time innovating instead of reading through irrelevant articles. Incorporating a medical citator into medical journal databases is one of the many ways that existing research knowledge can be better leveraged to improve America's health.