Skip to main content

Advertisement

SpringerLink
Log in

Modeling error in experimental assays using the bootstrap principle: understanding discrepancies between assays using different dispensing technologies

  • SPECIAL SERIES: STATISTICS IN MOLECULAR MODELING
  • Guest Editor: Anthony Nicholls
  • Published:
Journal of Computer-Aided Molecular Design Aims and scope Submit manuscript

Abstract

All experimental assay data contains error, but the magnitude, type, and primary origin of this error is often not obvious. Here, we describe a simple set of assay modeling techniques based on the bootstrap principle that allow sources of error and bias to be simulated and propagated into assay results. We demonstrate how deceptively simple operations—such as the creation of a dilution series with a robotic liquid handler—can significantly amplify imprecision and even contribute substantially to bias. To illustrate these techniques, we review an example of how the choice of dispensing technology can impact assay measurements, and show how large contributions to discrepancies between assays can be easily understood and potentially corrected for. These simple modeling techniques—illustrated with an accompanying IPython notebook—can allow modelers to understand the expected error and bias in experimental datasets, and even help experimentalists design assays to more effectively reach accuracy and imprecision goals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Care must be taken to distinguish between fully independent replicates and partial replicates that only repeat part of the experiment (for example, repeated measurements performed using the same stock solutions), since partial measurements can often underestimate true error by orders of magnitude [16].

  2. The companion IPython notebook is available online at: http://github.com/choderalab/dispensing-errors-manuscript.

  3. Volumes, masses, and concentrations must all be positive, so it is more appropriate in principle to use a lognormal distribution to model these processes to prevent negative values. In practice, however, if the relative imprecision is relatively small and negative numbers do not cause large problems for the functions, a normal distribution is sufficient.

  4. While manufacturer-provided specifications for imprecision and inaccuracy are often presented as the maximum-allowable values, we find these are a reasonable starting point for this kind of modeling.

  5. A surprising amount of effort is required to ensure thorough mixing of two solutions, especially in the preparation of dilution series [30–32]. We have chosen not to explicitly include this effect in our model, but it could similarly be added within this framework given some elementary data quantifying the bias induced by incomplete mixing.

  6. Note that a 1:2 dilution refers to combining one part solute solution with one part diluent.

  7. The published protocol [20, 21] does not specify how many dilutions were used, so for illustrative purposes, we selected \(n_{\text {dilutions}} = 8\).

  8. We note that real assays may encounter solubility issues with such high compound concentrations, and that the nonideal nature of water:DMSO solutions means that serial dilution of DMSO stocks will not always guarantee all dilutions will readily keep compound soluble. Here, we also presume the DMSO and EDTA control wells are not used in fitting to obtain \(p\hbox {IC}_{50}\) values.

References

  1. Kozikowski BA (2003) J Biomol Screen 8:210

    Article  CAS  Google Scholar 

  2. Kozikowski BA (2003) J Biomol Screen 8:205

    Article  CAS  Google Scholar 

  3. Cheng X, Hochlowski J, Tang H, Hepp D, Beckner C, Kantor S, Schmitt R (2003) J Biomol Screen 8:292

    Article  CAS  Google Scholar 

  4. Waybright TJ, Britt JR, McCloud TG (2009) J Biomol Screen 14:708

    Article  CAS  Google Scholar 

  5. Harris D, Olechno J, Datwani S, Ellson R (2010) J Biomol Screen 15:86

    Article  CAS  Google Scholar 

  6. Grant RJ, Roberts K, Pointon C, Hodgson C, Womersley L, Jones DC, Tang E (2009) J Biomol Screen 14:452

    Article  CAS  Google Scholar 

  7. Tjernberg A (2005) J Biomol Screen 11:131

    Article  Google Scholar 

  8. Simeonov A, Jadhav A, Thomas CJ, Wang Y, Huang R, Southall NT, Shinn P, Smith J, Austin CP, Auld DS, Inglese J (2008) J Med Chem 51:2363

    Article  CAS  Google Scholar 

  9. Baell JB, Holloway GA (2010) J Med Chem 53:2719

    Article  CAS  Google Scholar 

  10. Di L, Kerns EH (2006) Drug Discov Today 11:446

    Article  CAS  Google Scholar 

  11. McGovern SL, Caselli E, Grigorieff N, Shoichet BK (2002) J Med Chem 45:1712

    Article  CAS  Google Scholar 

  12. McGovern SL, Shoichet BK (2003) J Med Chem 46:1478

    Article  CAS  Google Scholar 

  13. Feng BY, Shelat A, Doman TN, Guy RK, Shoichet BK (2005) Nat Chem Biol 1:146

    Article  CAS  Google Scholar 

  14. Feng BY, Shoichet BK (2006) J Med Chem 49:2151

    Article  CAS  Google Scholar 

  15. Busch M, Thorma HB, Kober I (2015) J Biomol Screen 18:744

    Article  Google Scholar 

  16. Chodera JD, Mobley DL (2013) Annu Rev Biophys 42:121

    Article  CAS  Google Scholar 

  17. Kramer C, Kalliokoski T, Gedeck P, Vulpetti A (2012) J Med Chem 55:5165

    Article  CAS  Google Scholar 

  18. Kalliokoski T, Kramer C, Vulpetti A, Gedeck P (2013) PLoS ONE 8:e61007

    Article  CAS  Google Scholar 

  19. Ekins S, Olechno J, Williams AJ (2013) PLoS ONE 8:e62325

    Article  CAS  Google Scholar 

  20. Barlaam BC, Ducray R (2009) Novel pyrimidine derivatives 965, uS20090054428 A1

  21. Barlaam BC, Ducray R, Kettle JG (2010) Pyrimidine derivatives for inhibiting Eph receptors, uS7718653 B2

  22. Xia G, Kumar SR, Masood R, Zhu S, Reddy R, Krasnoperov V, Quinn DI, Henshall SM, Sutherland RL, Pinski JK et al (2005) Cancer Res 65:4623

    Article  CAS  Google Scholar 

  23. Bardelle C, Cross D, Davenport S, Kettle JG, Ko EJ, Leach AG, Mortlock A, Read J, Roberts NJ, Robins P, Williams EJ (2008) Bioorg Med Chem Lett 18:2776

    Article  CAS  Google Scholar 

  24. Lowe D (2015) Drug assay numbers, all over the place. In the Pipeline: http://blogs.sciencemag.org/pipeline/archives/2013/05/03/drug_assay_numbers_all_over_the_place

  25. Evanko D (2013) Serial dilution woes. http://blogs.nature.com/methagora/2013/05/serial-dilution-woes.html

  26. Ekins S (2013) What it took to get the paper out. http://www.collabchem.com/2013/05/03/what-it-took-to-get-the-paper-out/

  27. Palmgren J, Monkkonen J, Korjamo T, Hassinen A, Auriola S (2006) Eur J Pharm Biopharm 64:369

    Article  CAS  Google Scholar 

  28. Taylor JR (1997) An introduction to error analysis: the study of uncertainties in physical measurements. University Science Books, Mill Valley

    Google Scholar 

  29. Shalizi C Simple simulation methods for quantifying uncertainty. https://www.grc.org/programs.aspx?year=2013&program=cadd, https://openeye.app.box.com/s/ebafecebpplobcdy1yxg, http://lanyrd.com/2013/grc-cadd-2013/slides

  30. Walling L, Carramanzana N, Schulz C, Romig T, Johnson M (2007) ASSAY Drug Dev Technol 5:265

    Article  CAS  Google Scholar 

  31. Weiss S, John G, Klimant I, Heinzle E (2002) Biotechnol Prog 18:821

    Article  CAS  Google Scholar 

  32. Mitre E, Schulze M, Cumme GA, Rossler F, Rausch T, Rhode H (2007) J Biomol Screen 12:361

    Article  CAS  Google Scholar 

  33. Tecan Genesis Operating Manual (2001). http://www.tecan.com

  34. Echo 5XX Specifications (2011). http://www.labcyte.com/inc/lib/download_sheets?pkid=11

  35. Dong H, Ouyang Z, Liu J, Jemal M (2006) J Assoc Lab Autom 11:60

    Article  Google Scholar 

  36. Gu H, Deng Y (2007) J Assoc Lab Autom 12:355

    Article  CAS  Google Scholar 

  37. Jones G (2015) J Comput Aided Mol Des 29:1

    Article  CAS  Google Scholar 

  38. Wingfield J (2012) Impact of acoustic dispensing on data quality in HTS and hit confirmation. Drug Discovery 2012. Manchester

  39. Olechno J, Ekins S, Williams AJ, Fischer-Colbrie M (2013) Direct improvement with direct dilution. http://americanlaboratory.com/914-Application-Notes/142860-Direct-Improvement-With-Direct-Dilution/

  40. Olechno J, Ekins S, Williams AJ (2013) Sound dilutions. https://theanalyticalscientist.com/issues/0713/sound-dilutions/

  41. Olechno J, Shieh J, Ellson R (2006) J Assoc Lab Autom 11:240

    Article  Google Scholar 

  42. Jones RE, Zheng W, McKew JC, Chen CZ (1094) J Lab Autom 221106821349:2013

    Google Scholar 

Download references

Acknowledgments

The authors are grateful to Anthony Nichols (OpenEye) and Martin Stahl (Roche) for organizing the excellent 2013 Computer-Aided Drug Discovery Gordon Research Conference on the topic of “The Statistics of Drug Discovery”, as well as Terry Stouch for both his infinite patience and inspiring many of the ideas in this work. The authors are especially grateful to Cosma Shalizi for presenting a clear and lucid overview of the bootstrap principle to this audience, and we hope this contribution can further aid readers in the community in employing these principles in their work. The authors further acknowledge Adrienne Chow and Anthony Lozada of Tecan US for a great deal of assistance in understanding the nature of operation and origin of errors in automated liquid handling equipment. The authors thank Paul Czodrowski (Merck Darmstadt) for introducing us to IPython notebooks as a means of interactive knowledge transfer. JDC and SMH acknowledge support from the Sloan Kettering Institute, a Louis V. Gerstner Young Investigator Award, and NIH Grant P30 CA008748. SE acknowledges Joe Olechno and Antony Williams for extensive discussions on the topic, as well as the many scientists that responded to the various blog posts mentioned herein.

Author information

Authors and Affiliations

  1. Computational Biology Program, Sloan Kettering Institute, Memorial Sloan Kettering Cancer Center, New York, NY, 10065, USA

    Sonya M. Hanson & John D. Chodera

  2. Collaborations in Chemistry, Fuquay-Varina, NC, 27526, USA

    Sean Ekins

Authors
  1. Sonya M. Hanson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sean Ekins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John D. Chodera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John D. Chodera.

Ethics declarations

Conflict of interest

The authors acknowledge no conflicts of interest, but wish to disclose that JDC is on the Scientific Advisory Board of Schrödinger and SE is an employee of Collaborative Drug Discovery.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (IPYNB 473 KB)

Supplementary material 2 (HTML 806 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hanson, S.M., Ekins, S. & Chodera, J.D. Modeling error in experimental assays using the bootstrap principle: understanding discrepancies between assays using different dispensing technologies. J Comput Aided Mol Des 29, 1073–1086 (2015). https://doi.org/10.1007/s10822-015-9888-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10822-015-9888-6

Keywords

Search

Navigation