Abstract
An initial screening experiment may lead to ambiguous conclusions regarding the factors which are active in explaining the variation of an outcome variable: thus, adding follow-up runs becomes necessary. To better account for model uncertainty, we propose an objective Bayesian approach to follow-up designs, using prior distributions suitably tailored to model selection. To select the best follow-up runs, we adopt a model discrimination criterion based on a weighted average of Kullback–Leibler divergences between predictive distributions for all possible pairs of models. Our procedure should appeal to practitioners because it does not require prior specifications, being fully automatic. When applied to real data, it produces follow-up runs which better discriminate among factors relative to current methodology.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abramowitz M, Stegun IA (1964) Handbook of mathematical functions with formulas, graphs, and mathematical tables. National Bureau of Standards Applied Mathematics Series, vol 55. U.S Government Printing Office, Washington, D.C
Barrios Zamudio E (2013) Using the BsMD package for Bayesian screening and model discrimination. http://cran.r-project.org/web/packages/BsMD/vignettes/BsMD.pdf
Bayarri MJ, Garcia-Donato G (2007) Extending conventional priors for testing general hypotheses in linear models. Biometrika 94(1):135–152
Bayarri MJ, Berger JO, Forte A, García-Donato G (2012) Criteria for Bayesian model choice with application to variable selection. Ann Stat 40(3):1550–1577
Berger J (2006) The case for objective Bayesian analysis. Bayesian Anal 1(3):385–402
Berger JO, Pericchi L (1996) The intrinsic Bayes factor for model selection and prediction. J Am Stat Assoc 91(433):109–122
Berger JO, Pericchi LR (2001) Objective Bayesian methods for model selection: introduction and comparison. In: Model selection. IMS lecture notes monograph series, vol 38. Institute of Mathematical Statistics, Beachwood, pp 135–207
Bingham DR, Chipman HA (2007) Incorporating prior information in optimal design for model selection. Technometrics 49(2):155–163
Box GEP, Hill WJ (1967) Discrimination among mechanistic models. Technometrics 9(1):57–71
Box GEP, Hunter WG, Hunter JS (1978) Statistics for experimenters. An introduction to design, data analysis, and model building. Wiley, New York
Casella G, Moreno E (2006) Objective Bayesian variable selection. J Am Stat Assoc 101(473):157–167
Castillo I, van der Vaart A (2012) Needles and straw in a haystack: posterior concentration for possibly sparse sequences. Ann Stat 40(4):2069–2101
Chipman H, Hamada M (1996) Discussion: factor-based or effect-based modeling? Implications for design. Technometrics 38(4):317–320
Clyde MA, Ghosh J, Littman ML (2011) Bayesian adaptive sampling for variable selection and model averaging. J Comput Graph Stat 20(1):80–101
Han C, Chaloner K (2004) Bayesian experimental design for nonlinear mixed-effects models with application to HIV dynamics. Biometrics 60(1):25–33
Jeffreys H (1961) Theory of probability, 3rd edn. Clarendon Press, Oxford
Liang F, Paulo R, Molina G, Clyde MA, Berger JO (2008) Mixtures of \(g\) priors for Bayesian variable selection. J Am Stat Assoc 103(481):410–423
Maruyama Y, George EI (2011) Fully Bayes factors with a generalized \(g\)-prior. Ann Stat 39(5):2740–2765
Meyer D (1996) mdopt: Fortran programs to generate md-optimal screening and follow-up designs, and analysis of data. http://lib.stat.cmu.edu/
Meyer RD, Steinberg D, Box GEP (1996) Follow-up designs to resolve confounding in fractional factorials. Technometrics 38(4):303–313
Montgomery DC (2006) Design and analysis of experiments. Wiley, New York
O’Hagan A (1995) Fractional Bayes factors for model comparison. J R Stat Soc Ser B 57(1):99–138. (With discussion and a reply by the author)
O’Hagan A, Forster J (2004) Kendall’s advanced theory of statistics, vol 2b: Bayesian inference, 2nd edn. Arnold, London
Perez JM, Berger JO (2002) Expected posterior prior distributions for model selection. Biometrika 89(3):491–512
Pericchi LR (2005) Model selection and hypothesis testing based on objective probabilities and Bayes factors. In: Dey D, Rao CR (eds) Bayesian thinking: modeling and computation. Handbook of statistics, vol 25. Elsevier/North-Holland, Amsterdam, pp 115–149
Scott JG, Berger JO (2010) Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem. Ann Stat 38(5):2587–2619
Tsutakawa RK (1972) Design of experiment for bioassay. J Am Stat Assoc 67(339):584–590
Wolters MA, Bingham DR (2011) Simulated annealing model search for subset selection in screening experiments. Technometrics 53(3):225–237
Zellner A, Siow A (1980) Posterior odds ratios for selected regression hypotheses. In: Bernardo JM, DeGroot MH, Lindley DV, Smith AFM (eds) Bayesian statistics: proceedings of the first international meeting held in Valencia (Spain), University of Valencia, pp 585–603
Acknowledgments
We thank the Editor and the Reviewers for useful comments that contributed to a better presentation of the paper. The R-code to find the optimal follow-up runs was developed by Marta Nai Ruscone, Dipartimento di Scienze Statistiche, Università Cattolica del Sacro Cuore, Milan. We are indebted to the participants in the workshop on Model Oriented Design and Analysis (MODA 10) and O-Bayes 2013 conference for useful comments on preliminary versions of this paper. In particular, we thank Veronika Rǒcková for a detailed discussion of our work, including priors on model space and the derivation of the model discrimination criterion, as well as Gonzalo García-Donato for pointing out the relationship between the posterior under the hierarchical g-prior and that based on the reference prior.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
11749_2015_461_MOESM1_ESM.pdf
Appendix A: Derivation of KL-divergence between the predictive distributions for the follow-up runs under two models. Appendix B: Relationship between the posterior distributions under the hierarchical g-prior and the reference prior. Tables and Figures: A collection of Tables and Figures complementing those in the main text.
Rights and permissions
About this article
Cite this article
Consonni, G., Deldossi, L. Objective Bayesian model discrimination in follow-up experimental designs. TEST 25, 397–412 (2016). https://doi.org/10.1007/s11749-015-0461-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11749-015-0461-3
Keywords
- Bayesian design of experiments
- Bayesian model selection
- Kullback–Leibler divergence
- Non-informative prior
- Screening experiment