Download PDF
Appl Clin Inform 2015; 06(02): 334-344
DOI: 10.4338/ACI-2015-01-RA-0010
Research Article
Schattauer GmbH

Validation of a Crowdsourcing Methodology for Developing a Knowledge Base of Related Problem-Medication Pairs

A. B. McCoy
1   Center for Applied Health Services Research, Ochsner Health System
2   Department of Biostatistics and Bioinformatics, Tulane University School of Public Health and Tropical Medicine, New Orleans LA
,
A. Wright
3   Department of Medicine, Brigham and Women’s Hospital
4   Department of Clinical Informatics Research and Development, Partners HealthCare
5   Harvard Medical School, Boston, MA
,
M. Krousel-Wood
1   Center for Applied Health Services Research, Ochsner Health System
6   Department of Medicine, Tulane University School of Medicine
7   Department of Epidemiology, Tulane University School of Public Health and Tropical Medicine, New Orleans, LA
,
E. J. Thomas
8   Department of Internal Medicine, The University of Texas Medical School at Houston
9   The University of Texas at Houston-Memorial Hermann Center for Healthcare Quality and Safety, Houston, TX
,
J. A. McCoy
10   Department of Urology, Ochsner Health System, New Orleans, LA
,
D. F. Sittig
11   The University of Texas School of Biomedical Informatics at Houston, Houston, TX
› Author Affiliations
Further Information

Publication History

received: 21 January 2015

accepted: 05 April 2015

Publication Date:
19 December 2017 (online)

   Permissions and Reprints

Summary

Background: Clinical knowledge bases of problem-medication pairs are necessary for many informatics solutions that improve patient safety, such as clinical summarization. However, developing these knowledge bases can be challenging.

Objective: We sought to validate a previously developed crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large, non-university health care system with a widely used, commercially available electronic health record.

Methods: We first retrieved medications and problems entered in the electronic health record by clinicians during routine care during a six month study period. Following the previously published approach, we calculated the link frequency and link ratio for each pair then identified a threshold cutoff for estimated problem-medication pair appropriateness through clinician review; problem-medication pairs meeting the threshold were included in the resulting knowledge base. We selected 50 medications and their gold standard indications to compare the resulting knowledge base to the pilot knowledge base developed previously and determine its recall and precision.

Results: The resulting knowledge base contained 26,912 pairs, had a recall of 62.3% and a precision of 87.5%, and outperformed the pilot knowledge base containing 11,167 pairs from the previous study, which had a recall of 46.9% and a precision of 83.3%.

Conclusions: We validated the crowdsourcing approach for generating a knowledge base of problem-medication pairs in a large non-university health care system with a widely used, commercially available electronic health record, indicating that the approach may be generalizable across health-care settings and clinical systems. Further research is necessary to better evaluate the knowledge, to compare crowdsourcing with other approaches, and to evaluate if incorporating the knowledge into electronic health records improves patient outcomes.

Citation: McCoy AB, Wright A, Krousel-Wood M, Thomas EJ, McCoy JA, Sittig DF. Validation of a crowdsourcing methodology for developing a knowledge base of related problem-medication pairs. Appl Clin Inf 2015; 6: 334–344

http://dx.doi.org/10.4338/ACI-2015-01-RA-0010

Keywords

Crowdsourcing - electronic health records - knowledge bases - problem-oriented medical records - computer-assisted drug therapy - validation studies
 

  • References

  • 1 Feblowitz JC, Wright A, Singh H, Samal L, Sittig DF. Summarization of clinical information: a conceptual model. J Biomed Inform 2011; 44 (04) 688-699.
    CrossrefPubMedGoogle Scholar
  • 2 McCoy AB, Wright A, Laxmisan A, Ottosen MJ, McCoy JA, Butten D, Sittig DF. Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications. J Am Med Inform Assoc JAMIA 2012; 19 (05) 713-718.
    CrossrefPubMedGoogle Scholar
  • 3 Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA J Am Med Assoc 1995; 274 (01) 29-34.
    CrossrefPubMedGoogle Scholar
  • 4 Blumenthal D, Tavenner M. The “Meaningful Use” Regulation for Electronic Health Records. N Engl J Med 2010; 363 (06) 501-504.
    CrossrefPubMedGoogle Scholar
  • 5 Laxmisan A, McCoy AB, Wright A, Sittig DF. Clinical Summarization Capabilities of Commercially-available and Internally-developed Electronic Health Records. Appl Clin Inform 2012; 3 (01) 80-93.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 6 Health IT and Patient Safety: Building Safer Systems for Better Care –Institute of Medicine. Available from: http://iom.edu/Reports/2011/Health-IT-and-Patient-Safety-Building-Safer-Systems-for-Better-Care.aspx
    PubMed
  • 7 Han YY, Carcillo JA, Venkataraman ST, Clark RSB, Watson RS, Nguyen TC, Bayir H, Orr RA. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
    CrossrefPubMedGoogle Scholar
  • 8 Horsky J, Kuperman GJ, Patel VL. Comprehensive analysis of a medication dosing error related to CPOE. J Am Med Inform Assoc JAMIA 2005; 12 (04) 377-382.
    CrossrefPubMedGoogle Scholar
  • 9 Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA J Am Med Assoc 2005; 293 (10) 1197-1203.
    CrossrefPubMedGoogle Scholar
  • 10 McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, Peterson JF. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc JAMIA 2012; 19 (03) 346-352.
    CrossrefPubMedGoogle Scholar
  • 11 Ash JS, Berg M, Coiera E. Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors. J Am Med Inform Assoc 2004; 11 (02) 104-112.
    PubMedGoogle Scholar
  • 12 Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc JAMIA 2007; 14 (04) 415-423.
    CrossrefPubMedGoogle Scholar
  • 13 Sittig DF, Singh H. Legal, ethical, and financial dilemmas in electronic health record adoption and use. Pediatrics 2011; 127 (04) e1042-e1047.
    CrossrefPubMedGoogle Scholar
  • 14 Sittig DF, Teich JM, Osheroff JA, Singh H. Improving clinical quality indicators through electronic health records: it takes more than just a reminder. Pediatrics 2009; 124 (01) 375-377.
    CrossrefPubMedGoogle Scholar
  • 15 Sittig DF, Singh H. Rights and responsibilities of users of electronic health records. CMAJ Can Med Assoc J J Assoc Medicale Can. 2012; 184 (13) 1479-1483.
    PubMedGoogle Scholar
  • 16 Carter JS, Brown SH, Erlbaum MS, Gregg W, Elkin PL, Speroff T, Tuttle MS. Initializing the VA medication reference terminology using UMLS metathesaurus co-occurrences. Proc AMIA Symp 2002; 116-120.
    PubMedGoogle Scholar
  • 17 Elkin PL, Carter JS, Nabar M, Tuttle M, Lincoln M, Brown SH. Drug knowledge expressed as computable semantic triples. Stud Health Technol Inform 2011; 166: 38-47.
    PubMedGoogle Scholar
  • 18 Wright A, Chen ES, Maloney FL. An automated technique for identifying associations between medications, laboratory results and problems. J Biomed Inform 2010; 43 (06) 891-901.
    CrossrefPubMedGoogle Scholar
  • 19 Brown SH, Miller RA, Camp HN, Guise DA, Walker HK. Empirical derivation of an electronic clinically useful problem statement system. Ann Intern Med 1999; 131 (02) 117-126.
    CrossrefPubMedGoogle Scholar
  • 20 Zeng Q, Cimino JJ, Zou KH. Providing concept-oriented views for clinical data using a knowledge-based system: an evaluation. J Am Med Inform Assoc JAMIA 2002; 9 (03) 294-305.
    CrossrefPubMedGoogle Scholar
  • 21 Chen ES, Hripcsak G, Xu H, Markatou M, Friedman C. Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc JAMIA 2008; 15 (01) 87-98.
    CrossrefPubMedGoogle Scholar
  • 22 Kilicoglu H, Fiszman M, Rodriguez A, Shin D, Ripple A, Rindflesch TC. Semantic MEDLINE: A Web Application for Managing the Results of PubMed Searches. 2008: 69-76.
    PubMedGoogle Scholar
  • 23 Duke JD, Friedlin J. ADESSA: A Real-Time Decision Support Service for Delivery of Semantically Coded Adverse Drug Event Data. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2010; 2010: 177-181.
    PubMedGoogle Scholar
  • 24 Wu Y, Wright A, Xu H, McCoy AB, Sittig DF. Development of a Unified Computable Problem-Medication Knowledge Base. AMIA Annu Symp Proc AMIA Symp AMIA Symp 2014; 2014.
    PubMedGoogle Scholar
  • 25 Berk RA. An introduction to ensemble methods for data analysis. Sociol Methods Res 2006; 34 (03) 263-295.
    CrossrefPubMedGoogle Scholar
  • 26 Tapscott D. Wikinomics : how mass collaboration changes everything. New York: Portfolio; 2006
    Google Scholar
  • 27 Howe J. The rise of crowdsourcing. Wired Mag 2006; 14 (06) 1-4.
    PubMedGoogle Scholar
  • 28 Giles J. Internet encyclopaedias go head to head. Nature 2005; 438 7070 900-901.
    CrossrefPubMedGoogle Scholar
  • 29 Ekins S, Williams AJ. Reaching out to collaborators: crowdsourcing for pharmaceutical research. Pharm Res 2010; 27 (03) 393-395.
    CrossrefPubMedGoogle Scholar
  • 30 Hughes S, Cohen D. Can Online Consumers Contribute to Drug Knowledge? A Mixed-Methods Comparison of Consumer-Generated and Professionally Controlled Psychotropic Medication Information on the Internet. J Med Internet Res. 2011 13. 03
    PubMedGoogle Scholar
  • 31 Brownstein CA, Brownstein JS, Williams DS, 3rd, Wicks P, Heywood JA. The power of social networking in medicine. Nat Biotechnol 2009; 27 (10) 888-890.
    CrossrefPubMedGoogle Scholar
  • 32 Parry DT, Tsung-Chun Tsai. Crowdsourcing techniques to create a fuzzy subset of SNOMED CT for semantic tagging of medical documents. 2010 IEEE International Conference on Fuzzy Systems (FUZZ). IEEE. 2010: 1-8.
    PubMedGoogle Scholar
  • 33 Wagholikar KB, MacLaughlin KL, Kastner TM, Casey PM, Henry M, Greenes RA, Liu H, Chaudhry R. Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening. J Am Med Inform Assoc. 2013 Apr 5; amiajnl –2013-001613.
    PubMedGoogle Scholar
  • 34 Good BM, Su AI. Crowdsourcing for. Bioinformatics 2013; btt333.
    PubMedGoogle Scholar
  • 35 Sweidan M, Williamson M, Reeve JF, Harvey K, O’Neill JA, Schattner P, Snowdon T. Evaluation of features to support safety and quality in general practice clinical software. BMC Med Inform Decis Mak 2011; 11 (01) 27.
    CrossrefPubMedGoogle Scholar
  • 36 Hersh W. Evaluation of biomedical text-mining systems: lessons learned from information retrieval. Brief Bioinform 2005; 6 (04) 344-356.
    CrossrefPubMedGoogle Scholar