Skip to main content
SpringerLink
Log in

Gathering big data for teamwork evaluation with microworlds

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

We identify some of the challenges related with conducting research into teamwork, addressing in particular the data gathering problem, where researchers face multiple tensions derived from different viewpoints regarding what data to gather and how to do it. To address this problem, we propose a microworld approach for conducting research into teamwork. We present the main requirements guiding the microworld development, and discuss a set of components that realise the requirements. Then, we discuss a study that used the developed microworld to evaluate a groupware tool, which was designed to support team activities related to infrastructure maintenance. The paper emphasises the range of data gathered with the microworld, and how it contributed to simultaneously evaluate team behaviour and tool design. The paper reflects on the major contributions brought by the microworld approach, emphasising in particular the capacity to gather diverse data, and to combine behaviour and design evaluations. This research contributes to consolidate the microworld approach in teamwork research. It also contributes to reduce the gap between behavioural-oriented and design-oriented research. The combination of the behaviour-oriented and design-oriented views is of particular importance to design science, since it is founded on iterative cycles of development and evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Salas, E., Dickinson, T., Converse, S., Tannenbaum, S.: Toward an understanding of team performance and training. In: Swezey, R.W., Salas, E. (eds.) Teams: Their Training and Performance. Ablex Publishing, Westport (1992)

    Google Scholar 

  2. Wilson, K., Salas, E., Priest, H., Andrews, D.: Errors in the heat of battle: taking a closer look at shared cognition breakdowns through teamwork. Hum. FactorsC 49(2), 243–256 (2007)

    Article  Google Scholar 

  3. Ellis, C., Gibbs, S., Rein, G.: Groupware: some issues and experiences. Commun. ACM 34(1), 39–58 (1991)

    Article  Google Scholar 

  4. Jacovi, M., Soroka, V., Gilboa-Freedman, G., Ur, S., Shahar, E., Marmasse, N.: The chasms of CSCW: a citation graph analysis of the CSCW conference. In: Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work 2006, pp. 289–298. ACM (2006)

  5. Grudin, J., Poltrock, S.: Taxonomy and theory in computer supported cooperative work. In: Kozlowski, S. (ed.) Handbook of Organizational Psychology, pp. 1323–1348. Oxford University Press, Oxford (2012)

    Google Scholar 

  6. Antunes, P., Herskovic, V., Ochoa, S., Pino, J.: Structuring dimensions for collaborative systems evaluation. ACM Comput. Surv. 44(2), 1–28 (2012)

    Article  Google Scholar 

  7. Antunes, P., Xiao, L., Pino, J.: Assessing the impact of educational differences in HCI design practice. Int. J. Technol. Des. Educ. 24(3), 317–335 (2014). doi:10.1007/s10798-013-9254-8

    Article  Google Scholar 

  8. Herskovic, V., Pino, J., Ochoa, S., Antunes, P.: Evaluation methods for groupware systems. In: Haake, J., Ochoa, S., Cechich, A. (eds.) Groupware: design, implementation, and use. 13th International Workshop, CRIWG 2007, Bariloche, Argentina, September 2007 Proceedings, vol. 4715. LNCS, pp. 328–336. Springer, Heidelberg (2007)

  9. Hamadache, K., Lancieri, L.: Strategies and taxonomy, tailoring your CSCW evaluation. In: International Conference on Collaboration and Technology, pp. 206–221. Springer, Heidelberg (2009)

  10. Grudin, J.: Why CSCW applications fail: problems in the design and evaluation of organizational interfaces. In: Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work, Portland, pp. 85–93. ACM (1988)

  11. Hughes, J., King, V., Rodden, T., Andersen, H.: Moving out from the control room: ethnography in system design. In: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, Chapel Hill, pp. 429–439. ACM Press (1994)

  12. Hevner, A., March, S., Park, J., Ram, S.: Design science in information systems research. Manag. Inf. Syst. Q. 28(1), 75–105 (2004)

    Article  Google Scholar 

  13. Gregor, S.: The nature of theory in information systems. MIS Q. 30(3), 611–642 (2006)

    Article  Google Scholar 

  14. Gregor, S., Jones, D.: The anatomy of a design theory. J. Assoc. Inf. Syst. 8(5), 312–335 (2007)

    Google Scholar 

  15. Fjermestad, J., Hiltz, S.: An assessment of group support systems experimental research: methodology and results. J. Manag. Inf. Syst. 15(3), 7–149 (1999)

    Article  Google Scholar 

  16. Klein, G.: Naturalistic decision making. Hum. Factors 50(3), 456–460 (2008)

    Article  Google Scholar 

  17. Hollan, J., Hutchins, E., Kirsh, D.: Distributed cognition: toward a new foundation for human–computer interaction research. ACM Trans. Comput. Hum. Interact. 7(2), 174–196 (2000)

    Article  Google Scholar 

  18. Suchman, L.: Plans and Situated Actions: The Problem of Human–Machine Communication. Cambridge University Press, Cambridge (1987)

    Google Scholar 

  19. Endsley, M., Garland, D.: Situation Awareness Analysis and Measurement. CRC Press, Boca Raton (2000)

    Google Scholar 

  20. Endsley, M., Jones, W.: Situation awareness. In: Lee, J.D., Kirlik, A. (eds.) The Oxford Handbook of Cognitive Engineering, pp. 88–108. Oxford University Press, New York (2013)

    Google Scholar 

  21. Antunes, P., Herskovic, V., Ochoa, S., Pino, J.: Reviewing the quality of awareness support in collaborative applications. J. Syst. Softw. 89, 146–169 (2014)

    Article  Google Scholar 

  22. Wickens, C.: Situation awareness: Review of Mica Endsley’s 1995 articles on situation awareness theory and measurement. Hum. Factors 50(3), 397–403 (2008)

    Article  Google Scholar 

  23. Salmon, P., Stanton, N., Walker, G., Jenkins, D., Ladva, D., Rafferty, L., Young, M.: Measuring situation awareness in complex systems: comparison of measures study. Int. J. Ind. Ergon. 39(3), 490–500 (2009)

    Article  Google Scholar 

  24. Collins, A., Joseph, D., Bielaczyc, K.: Design research: theoretical and methodological issues. J. Learn. Sci. 13(1), 15–42 (2004)

    Article  Google Scholar 

  25. Lew, R., Boring, R., Ulrich, T.: A prototyping environment for research on human-machine interfaces in process control use of Microsoft WPF for microworld and distributed control system development. In: 7th International Symposium on Resilient Control Systems. IEEE (2014)

  26. Jercic, P., Astor, P., Adam, M., Hilborn, O., Schaaff, K., Lindley, C., Sennersten, C., Eriksson, J.: A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: European Conference on Information Systems, p. 207 (2012)

  27. Naweed, A., Hockey, G., Clarke, S.: Designing simulator tools for rail research: the case study of a train driving microworld. Appl. Ergon. 44(3), 445–454 (2013)

    Article  Google Scholar 

  28. Kluge, A.: Performance assessments with microworlds and their difficulty. Appl. Psychol. Meas. 32(2), 156–180 (2008)

    Article  MathSciNet  Google Scholar 

  29. Mosier, C.: A critical examination of the concepts of face validity. Educ. Psychol. Meas. 7, 191–205 (1947)

    Article  Google Scholar 

  30. Wastell, D., Peckover, S., White, S., Broadhurst, K., Hall, C., Pithouse, A.: Social work in the laboratory: using microworlds for practice research. Br. J. Soc. Work 41, 744–760 (2011)

    Article  Google Scholar 

  31. Rolo, G., Diaz-Cabrera, D.: Decision-making processes evaluation using two methodologies: field and simulation techniques. Theor. Issues Ergon. Sci. 6(1), 35–48 (2005)

    Article  Google Scholar 

  32. Gray, W.: Simulated task environments: the role of high-fidelity simulations, scaled worlds, synthetic environments, and laboratory tasks in basic and applied cognitive research. Cogn. Sci. Q. 2(2), 205–207 (2002)

    MathSciNet  Google Scholar 

  33. Rigas, G., Carling, E., Brehmer, B.: Reliability and validity of performance measures in microworlds. Intelligence 30(5), 463–480 (2002)

    Article  Google Scholar 

  34. DiFonzo, N., Hantula, D., Bordia, P.: Microworlds for experimental research: Having your (control and collection) cake, and realism too. Behav. Res. Methods Instrum. Comput. 30(2), 278–286 (1998)

    Article  Google Scholar 

  35. Chen, C., Zhang, C.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275, 314–347 (2014)

    Article  Google Scholar 

  36. Popper, K.: Science: conjectures and refutations. In: Introductory Readings in the Philosophy of Science, pp. 38–47. Prometheus Books, Amherst (1998)

  37. Venkatesh, V., Brown, S., Bala, H.: Bridging the qualitative–quantitative divide: guidelines for conducting mixed methods research in information systems. MIS Q. 37(1), 21–54 (2013)

    Article  Google Scholar 

  38. Horn, D., Finholt, T., Birnholtz, J., Motwani, D., Jayaraman, S.: Six degrees of Jonathan Grudin: a social network analysis of the evolution and impact of CSCW research. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 582–591. ACM (2004)

  39. Pinelle, D., Gutwin, C.: A review of groupware evaluations. In: Proceedings of 9th IEEE WETICE Infrastructure for Collaborative Enterprises (2000)

  40. Grant, A., Wall, T.: The neglected science and art of quasi-experimentation: why-to, when-to, and how-to advice for organizational researchers. Org. Res. Methods 12(4), 653–686 (2008)

    Article  Google Scholar 

  41. Winter, R.: Design science research in Europe. Eur. J. Inf. Syst. 17(5), 470–475 (2008)

    Article  Google Scholar 

  42. Baskerville, R., Wood-Harper, A.: A critical perspective on action research as a method for information systems research. In: Willcocks, L.P., Sauer, C., Lacity, M.C. (eds.) Enacting Research Methods in Information Systems, pp. 169–170. Springer, Berlin (2016)

    Google Scholar 

  43. Jenkins, A.: Research methodologies and MIS research. In: Mumford, E., et al. (eds.) Research Methods in Information Systems, pp. 103–117. North-Holland Publishing Co. Amsterdam, The Netherlands (1985)

  44. Sinkovics, R., Ghauri, P.: Enhancing the trustworthiness of qualitative research in international business. Manag Int Rev 48(6), 689–714 (2008)

    Article  Google Scholar 

  45. Flach, J.: Mind the gap: a skeptical view of macrocognition. In: Schraagen, J., Militello, L., Ormerod, T., Lipshitz, R. (eds.) Naturalistic Decision Making and Macrocognition. Ashgate, Hampshire (2008)

    Google Scholar 

  46. Wallace, B., Ross, A. (eds.): Beyond Human Error—Taxonomies and Safety Science. CRC Taylor and Francis Group, New York (2006)

    Google Scholar 

  47. Dekker, S.: The Field Guide to Understanding Human Error. Ashgate, Hampshire (2006)

    Google Scholar 

  48. Klein, G.: A recognition-primed decision (RPD) model of rapid decision making. In: Klein, G., Orasanu, J., Calderwood, R., Zsambok, C. (eds.) Decision Making in Action: Models and Methods. Ablex, Westport, CT (1993)

  49. Salas, E., Fiore, S., Warner, N., Letsky, M.: Emerging multi-disciplinary theoretical perspectives in team cognition: an overview. Theor. Issues Ergon. Sci. 11(4), 245–249 (2010)

    Article  Google Scholar 

  50. Lipshitz, R., Klein, G., Orasanu, J., Salas, E.: Taking stock of naturalistic decision making. J. Behav. Decis. Mak. 14, 331–352 (2001)

    Article  Google Scholar 

  51. Cleven, A., Gubler, P., Hüner, K.: Design alternatives for the evaluation of design science research artifacts. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology, Philadelphia, pp. 1–8. ACM (2009)

  52. Patrick, J., James, N.: Process tracing of complex cognitive work tasks. J. Occup. Org. Psychol. 77(2), 259–280 (2004)

    Article  Google Scholar 

  53. Sá, M., Carriço, L., Antunes, P.: Ubiquitous psychotherapy. IEEE. Pervasive Comput. 6(1), 20–27 (2007)

    Article  Google Scholar 

  54. Piirainen, K., Gonzalez, R., Kolfschoten, G.: Quo Vadis, design science?—a survey of literature. In: Global Perspectives on Design Science Research, vol. 6105. Lecture Notes in Computer Science, pp. 93–108. Springer (2010)

  55. Briggs, R.: On theory-driven design and deployment of collaboration systems. Int. J. Hum Comput Stud. 64(7), 573–582 (2006)

    Article  Google Scholar 

  56. Card, S., Moran, T., Newell, A.: The Psychology of Human–Computer Interaction. Lawrance Elrbaum, Hillsdale (1983)

    Google Scholar 

  57. Reason, J.: Human Error. Cambridge University Press, Cambridge (1990)

    Book  Google Scholar 

  58. Cacciabue, P.: Guide to Applying Human Factors Methods. Springer, London (2004)

    Book  Google Scholar 

  59. Cooke, N., Gorman, J., Myers, C., Duran, J.: Interactive team cognition. Cogn. Sci. 37(2), 255–285 (2013)

    Article  Google Scholar 

  60. Gibson, J.: The Senses Considered as Perceptual Systems. Houhton Mifflin, Boston (1966)

    Google Scholar 

  61. Antunes, P., Zurita, G., Baloian, N.: Key indicators for assessing the design of geocollaborative applications. Int. J. Inf. Technol. Decis. Mak. 13(2), 361–385 (2014)

    Article  Google Scholar 

  62. Turvey, M., Shawn, R.: Toward an ecological physics and physical psychology. In: Solso, R., Massaro, S. (eds.) The Science of the Mind: 2001 and Beyond, pp. 144–169. Oxford University Press, New York (1995)

    Google Scholar 

  63. Salas, E., Sims, D., Burke, C.: Is there a “Big Five” in teamwork? Small Group Res. 36(5), 555–599 (2005)

    Article  Google Scholar 

  64. Davis, F.: A technology acceptance model for empirically testing new end-user information systems: theory and results. Ph.D. Thesis, Massachusetts Institute of Technology (1986)

  65. Read, A., Hullsiek, B., Briggs, R.: The seven layer model of collaboration: an exploratory study of process identification and improvement. In: 45th Hawaii International Conference on System Science, pp. 412–420. IEEE (2012)

  66. Barnard, P., May, J., Duke, D., Duce, D.: Systems, interactions, and macrotheory. ACM Trans. Comput. Hum. Interact. 7(2), 222–262 (2000)

    Article  Google Scholar 

  67. Klein, G., Ross, K., Moon, B., Klein, D., Hoffman, R., Hollnagel, E.: Macrocognition. IEEE Intell. Syst. 18(3), 81–85 (2003)

    Article  Google Scholar 

  68. Fiore, S., Smith-Jentsch, K., Salas, E., Warner, N., Letsky, M.: Towards an understanding of macrocognition in teams: developing and defining complex collaborative processes and products. Theor. Issues Ergon. Sci. 11(4), 250–271 (2010)

    Article  Google Scholar 

  69. Papert, S.: Microworlds: transforming education. Artif. Intell. Educ. 1, 79–94 (1987)

    Google Scholar 

  70. Brehmer, B., Dorner, D.: Experiments with computer-simulated microworlds: escaping both the narrow straits of the laboratory and the deep blue sea of the field study. Comput. Hum. Behav. 9, 171–184 (1993)

    Article  Google Scholar 

  71. Johansson, B., Trnka, J., Granlund, R.: The Effect of geographical information systems on a collaborative command and control task. In: Proceedings of the 4th International Conference on Information Systems for Crisis Response and Management (ISCRAM). Delft (2007)

  72. Schraagen, J., Van den Ven, J.: Improving decision making in crisis response through critical thinking support. J. Cogn. Eng. Decis. Mak. 2, 311–327 (2008)

    Article  Google Scholar 

  73. Arthur, W., Day, E., Villado, A., Boatman, P., Kowollik, V., Bennet, W., Bhupatkar, A.: The effect of distributed practice on immediate posttraining, and long-term performance on a complex command-and-control simulation task. Hum. Perform. 23(5), 428–445 (2010)

    Article  Google Scholar 

  74. Lew, R., Boring, R., Ulrich, T.: A prototyping environment for research on human-machine interfaces in process control use of Microsoft WPF for microworld and distributed control system development. In: International Symposium on Resilient Control Systems, pp. 1–6. IEEE (2014)

  75. O’Brien, K., O’Hare, D.: Situational awareness ability and cognitive skills training in a complex real-world task. Ergonomics 50(7), 1064–1091 (2007)

    Article  Google Scholar 

  76. Berggren, P., Johansson, B., Svensson, E., Baroutsi, N., Dahlbäck, N.: Statistical modelling of team training in a microworld study. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 58(1), 894–898 (2014)

    Article  Google Scholar 

  77. de Heer, J.: How Do architects think? A game based microworld for elucidating dynamic decision-making. In: Complex Systems Design & Management, pp. 133–142. Springer (2016)

  78. Chapman, T., Nettelbecka, T., Welsha, M., Millsab, V.: Investigating the construct validity associated with microworld research: a comparison of performance under different management structures across expert and non-expert naturalistic decision-making groups. Aust. J. Psychol. 58(1), 40–47 (2006)

    Article  Google Scholar 

  79. Mavrikis, M., Dragon, T., Abdu, R., Harrer, A., De Groot, R., McLaren, B.: Learning to learn together through planning, discussion and reflection on microworld-based challenges. In: European Conference on Technology Enhanced Learning, pp. 483–488. Springer, Heidelberg (2012)

  80. Frezzo, D., DiCerbo, K., Behrens, J., Chen, M.: An extensible micro-world for learning in the data networking professions. Inf. Sci. 264, 91–103 (2014)

    Article  Google Scholar 

  81. Djaouti, D., Alvarez, J., Jessel, J.: Classifying serious games: the G/P/S model. In: Felicia, P. (ed.) Handbook of Research on Improving Learning and Motivation Through Educational Games: Multidisciplinary Approaches, pp. 118–136. IGI Global, Hershey (2011)

    Chapter  Google Scholar 

  82. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z.: Predicting protein structures with a multiplayer online game. Nature 466(7307), 756–770 (2010)

    Article  Google Scholar 

  83. Sonnleitner, P., Brunner, M., Greiff, S., Funke, J., Keller, U., Martin, R., Hazotte, C., Mayer, H., Latour, T.: The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving. Psychol. Test Assess. Model. 54, 54–72 (2012)

    Google Scholar 

  84. Gonzalez, C., Vanyukov, P., Martin, M.: The use of microworlds to study dynamic decision making. Comput. Hum. Behav. 21(2), 273–286 (2005)

    Article  Google Scholar 

  85. Brehmer, B.: Micro-worlds and the circular relation between people and their environment. Theor. Issues Ergon. Sci. 6(1), 73–93 (2005)

    Article  Google Scholar 

  86. Endsley, M., Bolté, B., Jones, D.: Designing for Situation Awareness. Taylor & Francis, London (2003)

    Book  Google Scholar 

  87. Wagner, F., Schmuki, R., Wagner, T., Wolstenholme, P.: Modeling Ssoftware with Finite State Machines: A Practical Approach. CRC Press, New York (2006)

    Book  MATH  Google Scholar 

  88. Lee, M.: A testing framework based on finite automata for object-oriented software specification. J. Inf. Technol. Theory Soc. 1, 59–88 (2004)

    Google Scholar 

  89. Buchner, A., Funke, J.: Finite state automata: dynamic task environments in problem solving research. Q. J. Exp. Psychol. 46A(1), 83–118 (1993)

    Article  Google Scholar 

  90. Taylor, M., Endsley, M., Henderson, S.: Situational awareness workshop report. In: Hayward, B., Lowe, A. (eds.) Applied Aviation Psychology: Achivement, Change and Challenge, pp. 447–454. Ashgate Publishing, Aldershot (1996)

  91. Citera, M., McNeese, M., Brown, C., Selvaraj, J.: Fitting information systems to collaborating design teams. J. Am. Soc. Inf. Sci. Technol. 46(7), 551 (1995)

    Article  Google Scholar 

  92. Klein, G., Zsambock, C., Thordsen, M.: Team decision training: five myths and a model. Mil. Rev. 73(4), 36–42 (1993)

    Google Scholar 

  93. Berndtsson, J., Normark, M.: The coordinative functions of flight strips : air traffic control revisited. In: Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, pp. 101–110. ACM, New York (1999)

  94. Heath, C., Luff, P.: Collaboration and control crisis management and multimedia technology in London Underground Line Control Rooms. Comput. Support. Coop. Work 1(1–2), 69–94 (1992)

    Article  Google Scholar 

  95. Pettersson, M., Randall, D., Helgeson, B.: Ambiguities, awareness and economy: a study of emergency service work. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, New Orleans, pp. 286–295. ACM (2002)

  96. Monares, A., Ochoa, S., Pino, J., Herskovic, V., Rodriguez-Covili, J., Neyem, A.: Mobile computing in urban emergency situations: improving the support to firefighters in the field. Expert Syst. Appl. 38(2), 1255–1267 (2011)

    Article  Google Scholar 

  97. Whittaker, S., Amento, B.: Seeing what you are hearing: co-ordinating responses to trouble reports in network troubleshooting. In: European Conference on Computer Supported Cooperative Work, Helsinki, pp. 219–238. Springer (2003)

  98. Fallman, D.: Enabling physical collaboration in industrial settings by designing for embodied interaction. In: Proceedings of the Latin American Conference on Human–Computer Interaction, Rio de Janeiro, pp. 41–51. ACM (2003)

  99. Nulden, U.: Investigating police patrol practice for design of IT. In: CHI ’03 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, pp. 820–821. ACM (2003)

  100. Landgren, J.: Supporting fire crew sensemaking enroute to incidents. Int. J. Emerg. Manag. 2(3), 176–188 (2005)

    Article  Google Scholar 

  101. Salmon, P., Stanton, N., Walker, G., Green, D.: Situation awareness measurement: a review of applicability for C4i environments. Appl. Ergon 37(2), 225–238 (2006)

    Article  Google Scholar 

  102. Bai, X., Tsai, W., Paul, R., Feng, K., Yu, L.: Scenario-based modeling and its applications. In: Proceedings of the Seventh International Workshop on Object-Oriented Real-Time Dependable Systems, pp. 253–260. IEEE Computer Society (2002)

  103. Barrett, R., Kandogan, E., Maglio, P., Haber, E., Takayama, L., Prabaker, M.: Field studies of computer system administrators: analysis of system management tools and practices. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 388–395. ACM (2004)

  104. Sapateiro, C.: Evaluating mobile collaborative applications support of teamwork in critical incidents response management. Ph.D. Thesis, University of Lisbon (2013)

  105. Sapateiro, C., Antunes, P.: An emergency response model toward situational awareness improvement. In: International Conference on Information Systems for Crisis Response and Management. Göteborg (2009)

  106. McManus, S., Seville, E., Brunsdon, D., Vargo, J.: Resilience management: a framework for assessing and improving the resilience of organisations. Research Report 2007/01. Resilient Organizations (2007)

  107. Hollnagel, E., Woods, D.: Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. CRC Press, Boca Raton (2005)

    Book  Google Scholar 

  108. Endsley, M.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 31(7), 32–64 (1995)

    Article  Google Scholar 

  109. Smith, K., Hancock, P.: Situation awareness is adaptive, externelly directed consciousness. Hum. Factors 37, 137–148 (1995)

    Article  Google Scholar 

  110. Bolstad, C., Cuevas, H., Gonzalez, C., Schneider, M.: Modeling shared situation awareness. In: Proceedings of the 14th Conference on Behavior Representation in Modeling and Simulation. Los Angles (2005)

  111. Stanton, N., Stewart, R., Harris, D., Houghton, R., Baber, C., McMaster, R., Salmon, P., Hoyle, G., Walker, G., Young, M., Linsell, M., Dymott, R., Green, D.: Distributed situation awareness in dynamic systems: theoretical develoment and application of an ergonomics methodology. Ergonomics 49(12–13), 1288–1311 (2006)

    Article  Google Scholar 

  112. Tamhane, A., Dunlop, D.: Statistics and Data Analysis: From Elementary to Intermediate. Prentice Hall, Upper Saddle River (2000)

    Google Scholar 

  113. Hevner, A.: A three cycle view of design science research. Scand. J. Inf. Syst. 19(2), 87–92 (2007)

    Google Scholar 

  114. Kurbalija, V., Ivanović, M., Bernstorff, C., Nachtwei, J., Burkhard, H.: Matching observed with empirical reality-what you see is what you get? Fundam.Inf. 129(1–2), 133–147 (2014)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Systems and Informatics Polytechnic of Setubal, Setubal, Portugal

    Claudio Miguel Sapateiro

  2. School of Information Management Victoria University of Wellington, Wellington, New Zealand

    Pedro Antunes & David Johnstone

  3. Department of Computer Science, University of Chile, Santiago, Chile

    José A. Pino

Authors
  1. Claudio Miguel Sapateiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro Antunes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Johnstone
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. José A. Pino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pedro Antunes.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sapateiro, C.M., Antunes, P., Johnstone, D. et al. Gathering big data for teamwork evaluation with microworlds. Cluster Comput 20, 1637–1659 (2017). https://doi.org/10.1007/s10586-016-0715-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-016-0715-1

Keywords

Search

Navigation