Abstract
The key contribution of this paper is a theoretical framework to analyse humans’ decision-making strategies under uncertainty, and more specifically how human subjects manage the trade-off between information gathering (exploration) and reward seeking (exploitation) in particular active learning in a black-box optimization task. Humans’ decisions making according to these two objectives can be modelled in terms of Pareto rationality. If a decision set contains a Pareto efficient (dominant) strategy, a rational decision maker should always select the dominant strategy over its dominated alternatives. A distance from the Pareto frontier determines whether a choice is (Pareto) rational. The key element in the proposed analytical framework is the representation of behavioural patterns of human learners as a discrete probability distribution, specifically a histogram considered as a non-parametric estimate of discrete probability density function on the real line. Thus, the similarity between users can be captured by a distance between their associated histograms. This maps the problem of the characterization of humans’ behaviour into a space, whose elements are probability distributions, structured by a distance between histograms, namely the optimal transport-based Wasserstein distance. The distributional analysis gives new insights into human behaviour in search tasks and their deviations from Pareto rationality. Since the uncertainty is one of the two objectives defining the Pareto frontier, the analysis has been performed for three different uncertainty quantification measures to identify which better explains the Pareto compliant behavioural patterns. Beside the analysis of individual patterns Wasserstein has also enabled a global analysis computing the WST barycenters and performing k-means Wasserstein clustering.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Data availability
Data are those already used in [9] and available at the following link: https://github.com/acandelieri/humans_strategies_analysis.
References
Wilson, R.C., Bonawitz, E., Costa, V.D., Ebitz, R.B.: Balancing exploration and exploitation with information and randomization. Curr. Opin. Behav. Sci. 38, 49–56 (2020)
Wilson, R.C., Geana, A., White, J.M., Ludvig, E.A., Cohen, J.D.: Humans use directed and random exploration to solve the explore–exploit dilemma. J. Exp. Psychol. Gen. 143(6), 2074 (2014)
Gershman, S.J.: Deconstructing the human algorithms for exploration. Cognition. 173, 34–42 (2018)
Schulz, E., Gershman, S.J.: The algorithmic architecture of exploration in the human brain. Curr. Opin. Neurobiol. 55, 7–14 (2019)
Schulz, E., Tenenbaum, J.B., Reshef, D.N., Speekenbrink, M., Gershman, S.: Assessing the Perceived Predictability of Functions. In: CogSci, vol. 6 (2015, November)
Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. Springer International Publishing (2019)
Frazier, P.I.: Bayesian optimization. In: Recent Advances in Optimization and Modeling of Contemporary Problems, pp. 255–278. INFORMS (2018)
Borji, A., Itti, L.: Bayesian Optimization Explains Human Active Search. Adv. Neural Inform. Process. Syst. 26, 55–63 (2013)
Candelieri, A., Perego, R., Giordani, I., Ponti, A., Archetti, F.: Modelling human active search in optimizing black-box functions. Soft. Comput. 24, 17771–17785 (2020). https://doi.org/10.1007/s00500-020-05398-2
Griffiths, T.L., Kemp, C., Tenenbaum, J.B.: Bayesian models of cognition. In: Sun, R. (ed.) Cambridge Handbook of Computational Cognitive Modelling. Cambridge University Press, Cambridge (2008)
Kruschke, J.K.: Bayesian approaches to associative learning: from passive to active learning. Learn. Behav. 36(3), 210–226 (2008)
Wilson, A.G., Dann, C., Lucas, C., Xing, E.P.: The human kernel. Adv. Neural Inform. Process. Syst. 28, 2854-2862 (2015)
Gershman, S.J.: Uncertainty and exploration. Decision. 6(3), 277 (2019)
Bock, H.H., Diday, E. (eds.): Analysis of Symbolic Data: Exploratory Methods for Extracting Statistical Information from Complex Data. Springer Science & Business Media (2012)
Monge, G.: Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci. 666–704 (1781)
Kantorovich, L.: On the Transfer of Masses (in Russian). In: Doklady Akademii Nauk. pp. 227–229 (1942)
Villani, C.: Optimal Transport: Old and New, vol. 338, p. 23. Springer, Berlin (2009)
Solomon, J., Rustamov, R., Guibas, L., & Butscher, A.: Wasserstein propagation for semi-supervised learning. In: International Conference on Machine Learning, pp. 306–314. PMLR (2014)
Peyré, G., Cuturi, M.: Computational optimal transport: with applica-tions to data science. Foundations and trends®. Mach. Learn. 11(5–6), 355–607 (2019)
Applegate, D., Dasu, T., Krishnan, S., Urbanek, S.: Unsupervised clustering of multidimensional distributions using earth mover distance. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 636–644. (2011, August)
Cabanes, G., Bennani, Y., Verde, R., Irpino, A.: On the use of Wasserstein metric in topological clustering of distributional data. arXiv preprint arXiv:2109.04301 (2021)
Puccetti, G., Rüschendorf, L., Vanduffel, S.: On the computation of Wasserstein barycenters. J. Multivar. Anal. 176, 104581 (2020)
Ye, J., Wu, P., Wang, J.Z., Li, J.: Fast discrete distribution clustering using Wasserstein barycenter with sparse support. IEEE Trans. Signal Process. 65(9), 2317–2332 (2017)
Verdinelli, I., Wasserman, L.: Hybrid Wasserstein distance and fast distribution clustering. Electron. J. Stat. 13(2), 5088–5119 (2019)
Cohen, J.D., McClure, S.M., Yu, A.J.: Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. B Biol. Sci. 362(1481), 933–942 (2007)
Gershman, S.J., Uchida, N.: Believing in dopamine. Nat. Rev. Neurosci. 20(11), 703–714 (2019)
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2), 235–256 (2002)
Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.W.: Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Trans. Inf. Theory. 58(5), 3250–3265 (2012)
Žilinskas, A., Calvin, J.: Bi-objective decision making in global optimization based on statistical models. J. Glob. Optim. 74(4), 599–609 (2019)
De Ath, G., Everson, R.M., Rahat, A.A., Fieldsend, J.E.: Greed is good: exploration and exploitation trade-offs in Bayesian optimisation. arXiv preprint arXiv:1911.12809 (2019)
De Ath, G., Everson, R.M., Fieldsend, J.E., Rahat, A.A.: ϵ-shotgun: ϵ-greedy batch bayesian optimisation. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 787–795. (2020)
Sandholtz, N. Modeling Human Decision-Making in Spatial and Temporal Systems (Doctoral Dissertation, Science: Department of Statistics and Actuarial Science) (2020)
Kahneman, D.: Thinking, Fast and Slow. Farrar, Straus and Giroux, New York (2011)
Tversky, A., Kahneman, D.: Rational Choice and the Framing of Decisions. In Multiple Criteria Decision Making and Risk Analysis Using Microcomputers, pp. 81–126. Springer, Berlin, Heidelberg (1989)
Kourouxous, T., Bauer, T.: Violations of dominance in decision-making. Bus. Res. 12(1), 209–239 (2019)
Peters, O.: The ergodicity problem in economics. Nat. Phys. 15(12), 1216–1221 (2019)
Williams, C. K., & Rasmussen, C. E. (2006). Gaussian Processes for Machine Learning (Vol. 2, No. 3, p. 4). Cambridge: MIT Press
Gramacy, R. B.: Surrogates: Gaussian process modeling, design, and optimization for the applied sciences. Chapman and Hall/CRC (2020)
Močkus, J. On Bayesian Methods for Seeking the Extremum. In: Optimization Techniques IFIP Technical Conference (pp. 400–404). Springer, Berlin, Heidelberg (1975)
Bemporad, A.: Global optimization via inverse distance weighting and radial basis functions. Comput. Optim. Appl. 77(2), 571–595 (2020)
Candelieri, A., Ponti, A., Archetti, F.: Uncertainty quantification and exploration–exploitation trade-off in humans. J. Ambient. Intell. Humaniz. Comput. 1–34 (2021)
Bonneel, N., Peyré, G., Cuturi, M.: Wasserstein barycentric coordinates: histogram regression using optimal transport. ACM Trans. Graph. 35(4), 71–71 (2016)
Kusner, M., Sun, Y., Kolkin, N., Weinberger, K. From Word Embeddings to Document Distances. In: International conference on machine learning (pp. 957–966). PMLR (2015, June)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International conference on machine learning, pp. 214–223. PMLR (2017, July)
Kandasamy, K., Neiswanger, W., Schneider, J., Poczos, B., Xing, E.: Neural architecture search with bayesian optimisation and optimal transport. arXiv preprint arXiv:1802.07191 (2018)
Bachoc, F.: Advances in Gaussian Process. (2019)
De Plaen, H., Fanuel, M., Suykens, J.A.: Wasserstein Exponential Kernels. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2020, July)
Le, T., Yamada, M., Fukumizu, K., Cuturi, M.: Tree-sliced variants of wasserstein distances. arXiv preprint arXiv:1902.00342 (2019)
Oh, J.H., Pouryahya, M., Iyer, A., Apte, A.P., Tannenbaum, A., Deasy, J.O.: Kernel wasserstein distance. arXiv preprint arXiv:1905.09314 (2019)
Bachoc, F., Gamboa, F., Loubes, J.M., Venet, N.: A Gaussian process regression model for distribution inputs. IEEE Trans. Inf. Theory. 64(10), 6620–6637 (2017)
Bachoc, F., Suvorikova, A., Ginsbourger, D., Loubes, J.M., Spokoiny, V.: Gaussian processes with multidimensional distribution inputs via optimal transport and Hilbertian embedding. Electron. J. Stat. 14(2), 2742–2772 (2020)
Mallasto, A., Gerolin, A., Minh, H.Q.: Entropy-regularized 2-Wasserstein distance between Gaussian measures. Inf. Geom. 1–35 (2021)
Balcan, M.F., Blum, A., Srebro, N.: A theory of learning with similarity functions. Mach. Learn. 72(1), 89–112 (2008)
Rakotomamonjy, A., Traoré, A., Berar, M., Flamary, R., Courty, N.: Distance measure machines. arXiv preprint arXiv:1803.00250 (2018)
Ponti, A., Candelieri, A., Archetti, F.: A new evolutionary approach to optimal sensor placement in water distribution networks. Water. 13(12), 1625 (2021a)
Ponti, A., Candelieri, A., Archetti, F.: A Wasserstein distance based multiobjective evolutionary algorithm for the risk aware optimization of sensor placement. Intell. Syst. Appl. 10, 200047 (2021b)
Acknowledgements
We greatly acknowledge the DEMS Data Science Lab, Department of Economics Management and Statistics (DEMS), for supporting this work by providing computational resources.
Appendix
The ten global optimization test functions used in this study, including their analytical formulations, search spaces and information about optimums and optimizers, can be found at the following link: https://www.sfu.ca/~ssurjano/optimization.html
Since they are minimization test functions, we have returned −f(x) as score in order to translate them into the maximization problems depicted in the following.
Funding
Open access funding provided by Università degli Studi di Milano - Bicocca within the CRUI-CARE Agreement.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Authors declare that they do not have any conflicts of interests or competing interests.
Ethics approval
Informed consent was given in accordance with the university’s procedure and the Helsinki declaration.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Candelieri, A., Ponti, A., Giordani, I. et al. On the use of Wasserstein distance in the distributional analysis of human decision making under uncertainty. Ann Math Artif Intell 91, 217–238 (2023). https://doi.org/10.1007/s10472-022-09807-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-022-09807-0