Skip to main content
SpringerLink
Log in

Empirical Performance of LGPS and LEOPARD: Lessons for Developing a Risk Identification and Analysis System

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Background

The availability of large-scale observational healthcare data allows for the active monitoring of safety of drugs, but research is needed to determine which statistical methods are best suited for this task. Recently, the Longitudinal Gamma Poisson Shrinker (LGPS) and Longitudinal Evaluation of Observational Profiles of Adverse events Related to Drugs (LEOPARD) methods were developed specifically for this task. LGPS applies Bayesian shrinkage to an estimated incidence rate ratio, and LEOPARD aims to detect and discard associations due to protopathic bias. The operating characteristics of these methods still need to be determined.

Objective

Establish the operating characteristics of LGPS and LEOPARD for large scale observational analysis in drug safety.

Research Design

We empirically evaluated LGPS and LEOPARD in five real observational healthcare databases and six simulated datasets. We retrospectively studied the predictive accuracy of the methods when applied to a collection of 165 positive control and 234 negative control drug-outcome pairs across four outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding.

Results

In contrast to earlier findings, we found that LGPS and LEOPARD provide weak discrimination between positive and negative controls, although the use of LEOPARD does lead to higher performance in this respect. Furthermore, the methods produce biased estimates and confidence intervals that have poor coverage properties.

Conclusions

For the four outcomes we examined, LGPS and LEOPARD may not be the designs of choice for risk identification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ahmad SR. Adverse drug event monitoring at the Food and Drug Administration. J Gen Intern Med. 2003;18(1):57–60.

    Article  PubMed  Google Scholar 

  2. Olsson S. The role of the WHO programme on International Drug Monitoring in coordinating worldwide drug safety efforts. Drug Saf. 1998;19(1):1–10.

    Article  PubMed  CAS  Google Scholar 

  3. Avorn J. Evaluating drug effects in the post-Vioxx world: there must be a better way. Circulation. 2006;113(18):2173–6.

    Article  PubMed  Google Scholar 

  4. Public Law 110-85: Food and Drug Administration Amendments Act of 2007. 2007.

  5. Woodcock J, Behrman RE, Dal Pan GJ. Role of postmarketing surveillance in contemporary medicine. Ann Rev Med. 2011;62:1–10.

    Google Scholar 

  6. Schuemie MJ, Coloma PM, Straatman H, Herings RM, Trifirò G, Matthews JN, et al. Using electronic health care records for drug safety signal detection: a comparative evaluation of statistical methods. Med Care. 2012;50(10):890–7.

    Article  PubMed  Google Scholar 

  7. Schuemie MJ. Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARD. Pharmacoepidemiol Drug Saf. 2011;20(3):292–9.

    Article  PubMed  CAS  Google Scholar 

  8. DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Stat. 1999;53(3):190–6.

    Google Scholar 

  9. Ryan PB, Schuemie MJ. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0110-2.

  10. Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for evidence based epidemiology. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0102-2.

  11. Ryan PB, Schuemie MJ, Welebob E, Duke J, Valentine S, Hartzema AG. Defining a reference set to support methodological research in drug safety. Drug Saf (in this supplement issue). doi:10.1007/s40264-013-0097-8.

  12. Armstrong B. A simple estimator of minimum detectable relative risk, sample size, or power in cohort studies. Am J Epidemiol. 1987;126(2):356–8.

    Article  PubMed  CAS  Google Scholar 

  13. Cantor SB, Kattan MW. Determining the area under the ROC curve for a binary diagnostic test. Med Decis Mak. 2000;20(4):468–70.

    Google Scholar 

  14. Masso Gonzalez EL, Patrignani P, Tacconelli S, Garcia Rodriguez LA. Variability among nonsteroidal antiinflammatory drugs in risk of upper gastrointestinal bleeding. Arthritis Rheum. 2010;62(6):1592–601.

    Google Scholar 

  15. Ashworth NL, Peloso PM, Muhajarine N, Stang M. Risk of hospitalization with peptic ulcer disease or gastrointestinal hemorrhage associated with nabumetone, Arthrotec, diclofenac, and naproxen in a population based cohort study. J Rheumatol. 2005;32(11):2212–7.

    PubMed  CAS  Google Scholar 

  16. Gralla RJ, Osoba D, Kris MG, Kirkbride P, Hesketh PJ, Chinnery LW, et al. Recommendations for the use of antiemetics: evidence-based, clinical practice guidelines. American Society of Clinical Oncology. J Clin Oncol. 1999;17(9):2971–94.

    PubMed  CAS  Google Scholar 

  17. Richards BL, Whittle SL, Buchbinder R. Muscle relaxants for pain management in rheumatoid arthritis. Cochrane Database Syst Rev. 2012;1:CD008922.

  18. Coloma PM, Avillach P, Salvo F, Schuemie MJ, Ferrajolo C, Pariente A, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Saf. 2013;36(1):13–23.

    Article  PubMed  Google Scholar 

  19. Freeman PR. The performance of the two-stage analysis of two-treatment, two-period crossover trials. Stat Med. 1989;8(12):1421–32.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgements

The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Drs. Schuemie and Ryan are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration. Dr. Madigan has no conflicts of interest to declare.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project.from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.

Author information

Authors and Affiliations

  1. Department of Medical Informatics, Erasmus University Medical Center Rotterdam, P.O. Box 2040, 3000 CA, Rotterdam, The Netherlands

    Martijn J. Schuemie

  2. Department of Statistics, Columbia University, New York, NY, USA

    David Madigan

  3. Janssen Research and Development LLC, Titusville, NJ, USA

    Patrick B. Ryan

  4. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, USA

    Martijn J. Schuemie, David Madigan & Patrick B. Ryan

Authors
  1. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Madigan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Patrick B. Ryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martijn J. Schuemie.

Additional information

The OMOP research used data from Truven Health Analytics (formerly the Health Business of Thomson Reuters), and includes MarketScan® Research Databases, represented with MarketScan Lab Supplemental (MSLR, 1.2 m persons), MarketScan Medicare Supplemental Beneficiaries (MDCR, 4.6 m persons), MarketScan Multi-State Medicaid (MDCD, 10.8 m persons), MarketScan Commercial Claims and Encounters (CCAE, 46.5 m persons). Data also provided by Quintiles® Practice Research Database (formerly General Electric’s Electronic Health Record, 11.2 m persons) database. GE is an electronic health record database while the other four databases contain administrative claims data.

Appendix

Appendix

figure a

LGPS and LEOPARD estimates for all test cases, by database. MSLR MarketScan Lab Supplemental, MDCD MarketScan Multi-state Medicaid, MDCR MarketScan Medicare Supplemental Beneficiaries, CCAE MarketScan Commercial Claims and Encounters, GE GE Centricity; Blue negative controls; Orange positive controls; Red crosses indicate flagged by LEOPARD as protopathic bias. Open circles indicate a drug-outcome pair with not enough power to detect a minimal detectable rate of 1.25. Each line represents point estimate and 95 % confidence interval for the drug-outcome pair in a particular database

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schuemie, M.J., Madigan, D. & Ryan, P.B. Empirical Performance of LGPS and LEOPARD: Lessons for Developing a Risk Identification and Analysis System. Drug Saf 36 (Suppl 1), 133–142 (2013). https://doi.org/10.1007/s40264-013-0107-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-013-0107-x

Keywords

Search

Navigation