Skip to main content
SpringerLink
Log in
Book cover

Advances in Social Simulation pp 367–378Cite as

The Large-Scale, Systematic and Iterated Comparison of Agent-Based Policy Models

  • Conference paper
  • First Online:

Part of the book series: Springer Proceedings in Complexity ((SPCOM))

Abstract

Vital to the increased rigour (and hence reliability) of Agent-based modelling are various kinds of model comparison. The reproduction of simulations is an essential check that models are as they are described. Here we argue that we need to go further and carry out large-scale, systematic and persistent model comparison—where different models of the same phenomena are compared against standardised data sets and each other. Lessons for this programme can be gained from the Model Intercomparison Projects (MIP) in the Climate Community and elsewhere. The benefits, lessons and particular difficulties of implementing a similar project in social simulation are discussed, before sketching what such a project might look like. It is time we got our act together!

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Maxwell, S.E., Lau, M.Y., Howard, G.S.: Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? Am. Psychol. 70(6), 487–498 (2015). https://doi.org/10.1037/a0039400

    Article  Google Scholar 

  2. Axtell, R., Axelrod, R., Epstein, J. M., Cohen, M. D.: Aligning simulation models: a case study and results. Computat. Math. Organiz. Theory 1(2), 123–141 (1996). https://link.springer.com/article/10.1007%2FBF01299065

  3. Chattoe-Brown, E., Gilbert, N., Robertson, D. A., Watts, C. J.: Reproduction as a means of evaluating policy models: a case study of a COVID-19 simulation. medRxiv 01.29.21250743 (2021). https://doi.org/10.1101/2021.01.29.21250743

  4. Edmonds, B., Hales, D.: Replication, replication and replication: some hard lessons from model alignment. J.Artif. Societ. Soc. Simul. 6(4), 11 (2003). http://jasss.soc.surrey.ac.uk/6/4/11.html

  5. Riolo, R.L., Cohen, M.D., Axelrod, R.: Evolution of cooperation without reciprocity. Nature 414(6862), 441–443 (2001). https://doi.org/10.1038/35106555

    Article  ADS  Google Scholar 

  6. Hales, D.: Vision for a more rigorous “replication first” modelling journal. Rev. Artif. Societ. Soc. Simul, 5 Nov (2018). https://rofasss.org/2018/11/05/dh/

  7. Siebers, P.O., Achter, S., Bernardo, P., Cristiane, B., Melania and Chattoe-Brown, E.: First Steps Towards RAT: a protocol for documenting data use in the agent-based modeling process (Extended Abstract), in Ahrweiler, Petra and Neumann, Martin (eds.) Advances in Social Simulation: ESSA 2019, Springer Proceedings in Complexity (Cham: Springer), pp. 257–261 (2021). https://doi.org/10.1007/978-3-030-61503-1_24

  8. Hales, D., Rouchier, J., Edmonds, B.: Model-to-model analysis. J. Artif. Societ. Soc. Simul. 6(4), 5 (2003). http://jasss.soc.surrey.ac.uk/6/4/5.html

  9. Lafuerza, L.F., Dyson, L., Edmonds, B., McKane, A.J.: Staged models for interdisciplinary research. PLoS ONE 11(6), e0157261 (2016). https://doi.org/10.1371/journal.pone.0157261

    Article  Google Scholar 

  10. Tittensor, D. P., Eddy, T. D., Lotze, H. K., Galbraith, E. D., Cheung, W., Barange, M., Blanchard, J. L., Bopp, L., Bryndum-Buchholz, A., Büchner, M., Bulman, C., Carozza, D. A., Christensen, V., Coll, M., Dunne, J. P., Fernandes, J. A., Fulton, E. A., Hobday, A. J., Huber, V., … Walker, N. D.: A protocol for the intercomparison of marine fishery and ecosystem models: Fish-MIP v1.0. Geoscient. Model Dev. 11(4), 1421–1442 (2018). https://doi.org/10.5194/gmd-11-1421-2018

  11. Wei, Y., Liu, S., Huntzinger, D.N., Michalak, A.M., Viovy, N., Post, W.M., Schwalm, C.R., Schaefer, K., Jacobson, A.R., Lu, C., Tian, H., Ricciuto, D.M., Cook, R.B., Mao, J., Shi, X.: The north american carbon program multi-scale synthesis and terrestrial model intercomparison project - Part 2: environmental driver data. Geoscient. Model Dev. 7(6), 2875–2893 (2014). https://doi.org/10.5194/gmd-7-2875-2014

    Article  ADS  Google Scholar 

  12. Gates, W.L., Boyle, J.S., Covey, C., Dease, C.G., Doutriaux, C.M., Drach, R.S., Fiorino, M., Gleckler, P.J., Hnilo, J.J., Marlais, S.M., Phillips, T.J., Potter, G.L., Santer, B.D., Sperber, K.R., Taylor, K.E., Williams, D.N.: An overview of the results of the atmospheric model intercomparison project (AMIP I). In Bull. Am. Meteorol. Soc. 80(1), 29–55 (1999). https://doi.org/10.1175/1520-0477(1999)080%3c0029:AOOTRO%3e2.0.CO;2

    Article  ADS  Google Scholar 

  13. Climate model inter-comparison project 6. https://www.wcrp-climate.org/wgcm-cmip/wgcm-cmip6. Last Accessed 19 May 2021

  14. Eyring, V., Bony, S., Meehl, G.A., Senior, C.A., Stevens, B., Stouffer, R.J., Taylor, K.E.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geoscient. Model Dev. 9(5), 1937–1958 (2016). https://doi.org/10.5194/gmd-9-1937-2016

    Article  ADS  Google Scholar 

  15. Covid Forecast Hub. https://covid19forecasthub.org. Last Accessed 19 May 2021

  16. Moshontz, H. + 85 others. The psychological science accelerator: advancing psychology through a distributed collaborative network 1(4) 501–515 (2018). https://doi.org/10.1177/2515245918797607

  17. Psychological Science Accelerator. https://psysciacc.org/. Last Accessed 19 May 2021

  18. Jones, B.C., DeBruine, L.M., Flake, J.K., et al.: To which world regions does the valence–dominance model of social perception apply? Nat. Human Behav. 5, 159–169 (2021). https://doi.org/10.1038/s41562-020-01007-2

    Article  Google Scholar 

  19. Chattoe-Brown, E.: ‘Why questions like “Do Networks Matter?” Matter to methodology: how agent-based modelling makes it possible to answer them’, Int. J. Soc. Res. Methodol. (online first 2020). https://doi.org/10.1080/13645579.2020.1801602

  20. Aodha, L., Edmonds, B.: Some pitfalls to beware when applying models to issues of policy relevance. In Edmonds, B., Meyer, R. (eds.) Simulating Social Complexity—A handbook, 2nd edition. Springer, pp. 801–822 (2017). https://doi.org/10.1007/978-3-319-66948-9_29

  21. Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., Goss-Custard, J., Grand, T., Heinz, S.K., Huse, G., Huth, A., Jepsen, J.U., Jørgensen, C., Mooij, W.M., Müller, B., Pe’er, G., Piou, C., Railsback, S.F., Robbins, A.M., Robbins, M.M., Rossmanith, E., Rüger, N., Strand, E., Souissi, S., Stillman, R.A., Vabø, R., Visser, U., DeAngelis, D.L.: A standard protocol for describing individual-based and agent-based models. Ecol. Model. 198(1–2), 115–126 (2006). https://doi.org/10.1016/j.ecolmodel.2006.04.023

    Article  Google Scholar 

  22. Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S., ... Railsback, S. F.: Towards better modelling and decision support: documenting model development, testing, and analysis using TRACE. Ecol. Modell. 280, 129–139 (2014). https://doi.org/10.1016/j.ecolmodel.2014.01.018

  23. Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J,, Johnston, A.S.A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M-S., Stillman, R. A., Thiele, J. C., Ayllón, D. The ODD protocol for describing agent-based and other simulation models: a second update to improve clarity, replication, and structural realism. J. Artif. Societ. Soc. Simul. 23(2), 7 (2020). http://jasss.soc.surrey.ac.uk/23/2/7.html. https://doi.org/10.18564/jasss.4259

  24. Polhill, J. G., Edmonds, B.: Open access for social simulation. J. Artif. Societ. Soc. Simul. 10(3), 10 (2007). http://jasss.soc.surrey.ac.uk/10/3/10.html

  25. Squazzoni, F., Polhill, J. G., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., ... Gilbert, N.: Computational models that matter during a global pandemic outbreak: a call to action. J. Artif. Societ. Soc. Simul. 23(2), 10 (2020). http://jasss.soc.surrey.ac.uk/23/2/10.html. https://doi.org/10.18564/jasss.4298

  26. Edmonds, B., Le Page, C., Bithell, M., Chattoe-Brown, E., Grimm, V., Meyer, R., Montañola-Sales, C., Ormerod, P., Root, H., Squazzoni, F.: Different modelling purposes. J. Artif. Societ. Soc. Simul. 22(3), 6 (2019). http://jasss.soc.surrey.ac.uk/22/3/6.html. https://doi.org/10.18564/jasss.3993

  27. Patterson, M. M.: The coming influenza pandemic: lessons from the past for the future. J. Osteopathic Med. 105(11), 498–500 (2005). https://doi.org/10.7556/jom_2005_11.0001

  28. Bithell, M., Edmonds, B.: The systematic comparison of agent-based policy models—It’s time we got our act together! Review of Artificial Societies and Social Simulation, 11th May 2021 (2020). https://rofasss.org/2021/05/11/SystComp/

Download references

Acknowledgements

This paper is an expanded version of [28].

Author information

Authors and Affiliations

  1. Department of Geography, University of Cambridge, Cambridge, England

    Mike Bithell

  2. School of Media, Communication and Sociology, University of Leicester, Leicester, England

    Edmund Chattoe-Brown

  3. Centre for Policy Modelling, Manchester Metropolitan University, Manchester, England

    Bruce Edmonds

Authors
  1. Mike Bithell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edmund Chattoe-Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bruce Edmonds
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike Bithell .

Editor information

Editors and Affiliations

  1. Financial Markets, Cracow University of Economics, Kraków, Poland

    Marcin Czupryna

  2. SGH Warsaw School of Economics, Warsaw, Poland

    Bogumił Kamiński

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bithell, M., Chattoe-Brown, E., Edmonds, B. (2022). The Large-Scale, Systematic and Iterated Comparison of Agent-Based Policy Models. In: Czupryna, M., Kamiński, B. (eds) Advances in Social Simulation. Springer Proceedings in Complexity. Springer, Cham. https://doi.org/10.1007/978-3-030-92843-8_28

Download citation

Publish with us

Policies and ethics

Search

Navigation