Abstract
Real time data associated with the Building Information Model plays a critical role in the interpretation of the built environment, which is particularly relevant as an increasing number of education facilities and institutions promote sustainable engineering practices and monitoring data available to the public. However, it is challenging for non-technical audiences to fully comprehend or use information concealed in scientific data related to the performance of structures and materials. It is especially difficult for them to connect these concepts to physical contexts and phenomena. In this paper, we present how cross-reality paradigms in Architecture, Engineering, and Construction, coupled with multimodal representation techniques, enhance data literacy in both professionals and laypeople alike. In particular, we present the design of a learning environment where cutting-edge holographic interfaces and display technologies are combined with sonified and visual data to create a more immersive environment for data analysis and exploration, empowering users with situated data awareness and new ways of understanding real-time data.
Similar content being viewed by others
References
Porter, J.R., Morgan, J.A., Johnson, M.: Building automation and IoT as a platform for introducing STEM education in K-12. In: 2017 ASEE Annual Conference & Exposition (2017)
Dickel, S.: Trust in technologies? Science after de-professionalization. J. Sci. Commun. 15, 1–7 (2016). https://doi.org/10.22323/2.15050303
Buckman, A.H., Mayfield, M., Beck, B.M.: What is a smart building? Smart Sustain. Built Environ. 3, 92–109 (2014). https://doi.org/10.1108/SASBE-01-2014-0003
Carlson L.E, Sullivan J.F.: (1999) Hands-on engineering: learning by doing in the integrated teaching and learning program. Int. J. Eng. Educ. 15, 20–31
Petersen, J., Frantz, C., Shammin, R.: Using sociotechnical feedback to engage, educate, motivate and empower environmental thought and action. Solutions 5, 79–87 (2014)
Petersen, J.E., Murray, M.E., Platt, G., Shunturov, V.: Using buildings to teach environmental stewardship: real-time display of environmental performance as a mechanism for educating, motivating, and empowering the student body. In: Proc Green Campus VI Muncie Indiana (2007)
Petersen, J.E., Rosenberg Daneri, D., Frantz, C., Shammin, M.R.: Environmental dashboards: fostering pro-environmental and pro-community thought and action through feedback. In: Leal Filho, W., Mifsud, M., Shiel, C., Pretorius, R. (eds.) Handbook of Theory and Practice of Sustainable Development in Higher Education, pp. 149–168. Springer International Publishing, Cham (2017)
Squire, K., Klopfer, E.: Augmented reality simulations on handheld computers. J. Learn. Sci. 16, 371–413 (2007). https://doi.org/10.1080/10508400701413435
Kamarainen, A.M., Metcalf, S., Grotzer, T., Browne, A., Mazzuca, D., Tutwiler, M.S., Dede, C.: EcoMOBILE: integrating augmented reality and probeware with environmental education field trips. Comput. Educ. 68, 545–556 (2013). https://doi.org/10.1016/j.compedu.2013.02.018
Cocciolo, A., Rabina, D.: Does place affect user engagement and understanding? Mobile learner perceptions on the streets of New York. J. Doc. 69, 98–120 (2013). https://doi.org/10.1108/00220411311295342
Pimmer, C., Mateescu, M., Gröhbiel, U.: Mobile and ubiquitous learning in higher education settings: a systematic review of empirical studies. Comput. Hum. Behav. 63, 490–501 (2016). https://doi.org/10.1016/j.chb.2016.05.057
Rogers, Y., Price, S., Harris, E., Phelps, T., Underwood, M., Wilde, D., Smith, H., Weal, M.T.M.J., Michaelides, D.T.: Learning through digitally-augmented physical experiences: reflections on the Ambient Wood project (2002)
Brown, A., Green, T.: Virtual reality: low-cost tools and resources for the classroom. Tech. Trends. 60, 517–519 (2016). https://doi.org/10.1007/s11528-016-0102-z
Brown, J.S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning. Educ. Res. 18, 32–42 (1989). https://doi.org/10.2307/1176008
Smith, S.M., Vela, E.: Environmental context-dependent memory: a review and meta-analysis. Psychon. Bull. Rev. 8, 203–220 (2001). https://doi.org/10.3758/BF03196157
Chun, M.M., Jiang, Y.: Contextual Cueing: implicit Learning and Memory of Visual Context Guides Spatial Attention. Cogn. Psychol. 36, 28–71 (1998). https://doi.org/10.1006/cogp.1998.0681
Cook, A.E., Limber, J.E., O’Brien, E.J.: Situation-based context and the availability of predictive inferences. J. Mem. Lang. 44, 220–234 (2001). https://doi.org/10.1006/jmla.2000.2744
Smith, S.M., Glenberg, A., Bjork, R.A.: Environmental context and human memory. Mem. Cogn. 6, 342–353 (1978). https://doi.org/10.3758/BF03197465
Chi, M.T.H.: Active-Constructive-Interactive: a conceptual framework for differentiating learning activities. Top. Cogn. Sci. 1, 73–105 (2009). https://doi.org/10.1111/j.1756-8765.2008.01005.x
Jang, S., Vitale, J.M., Jyung, R.W., Black, J.B.: Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment. Comput. Edu. 106, 150–165 (2017). https://doi.org/10.1016/j.compedu.2016.12.009
Goldstone, R.L., Son, J.Y.: The transfer of scientific principles using concrete and idealized simulations. J. Learn. Sci. 14, 69–110 (2005). https://doi.org/10.1207/s15327809jls1401_4
Larkin, J., Simon, H.: Why a diagram is (Sometimes) worth ten thousand words. Cogn. Sci. 11, 65–100 (1987). https://doi.org/10.1016/S0364-0213(87)80026-5
Schnotz, W., Bannert, M.: Construction and interference in learning from multiple representation. Learn. Instr. 13, 141–156 (2003). https://doi.org/10.1016/S0959-4752(02)00017-8
Lowe, R.K.: Animation and learning: value for money. In: Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference. pp 558–561 (2004)
Sanchez, C.A., Wiley, J.: Sex differences in science learning: closing the gap through animations. Learn. Individ. Differ. 20, 271–275 (2010). https://doi.org/10.1016/j.lindif.2010.01.003
Sanchez, C.A., Wiley, J.: The role of dynamic spatial ability in geoscience text comprehension. Learn. Instr. 31, 33–45 (2014). https://doi.org/10.1016/j.learninstruc.2013.12.007
Schiefele, U.: Interest, learning, and motivation. Edu. Psychol. 26, 299–323 (1991). https://doi.org/10.1080/00461520.1991.9653136
Hsu, Y.-S., Lin, Y.-H., Yang, B.: Impact of augmented reality lessons on students’ STEM interest. Res. Pract. Tech. Enhanced Learn. 12, 2 (2017). https://doi.org/10.1186/s41039-016-0039-z
Moreno, R., Mayer, R.E.: Learning science in virtual reality multimedia environments: role of methods and media. J. Educ. Psychol. 94, 598–610 (2002). https://doi.org/10.1037/0022-0663.94.3.598
Buckley, P., Doyle, E.: Gamification and student motivation. Interact. Learn. Environ. 24, 1162–1175 (2016). https://doi.org/10.1080/10494820.2014.964263
Dicheva, D., Dichev, C., Agre, G., Angelova, G.: Gamification in education: a systematic mapping study. J. Educ. Tech. Soc. 18, 75–88 (2015)
McCombs, B.L.: Motivation and lifelong learning. Educ. Psychol. 26, 117–127 (1991). https://doi.org/10.1207/s15326985ep2602_4
Rashid, T., Asghar, H.M.: Technology use, self-directed learning, student engagement and academic performance: examining the interrelations. Comput. Hum. Behav. 63, 604–612 (2016). https://doi.org/10.1016/j.chb.2016.05.084
Dackermann, U., Crews, K., Kasal, B., Li, J., Riggio, M., Rinn, F., Tannert, T.: In situ assessment of structural timber using stress-wave measurements. Mater. Struct. 47, 787–803 (2014). https://doi.org/10.1617/s11527-013-0095-4
Dickel, S., Franzen, M.: The “Problem of Extension” revisited: new modes of digital participation in science. J. Sci. Commun. (2016). https://doi.org/10.22323/2.15010206
Posada, J., Toro, C., Barandiaran, I., Oyarzun, D., Stricker, D., de Amicis, R., Pinto, E.B., Eisert, P., Dollner, J., Vallarino, I.: Visual computing as a key enabling technology for industrie 4.0 and industrial internet. IEEE Comput. Graph Appl. 35, 26–40 (2015). https://doi.org/10.1109/MCG.2015.45
Reid J.B., Rhodes D.H.: Digital System Models: an investigation of the non-technical challenges and research needs. In: 2016 Conference on Systems Engineering Research. Huntsville, AL (2016)
Schleich, B., Anwer, N., Mathieu, L., Wartzack, S.: Shaping the digital twin for design and production engineering. CIRP Ann. 66, 141–144 (2017). https://doi.org/10.1016/j.cirp.2017.04.040
Ciribini, A.L.C., Pasini, D., Tagliabue, L.C., Manfren, M., Daniotti, B., Rinaldi, S., De Angelis, E.: Tracking users’ behaviors through real-time information in BIMs: workflow for interconnection in the brescia smart campus demonstrator. Procedia Eng. 180, 1484–1494 (2017). https://doi.org/10.1016/j.proeng.2017.04.311
Donalek, C., Djorgovski, S.G., Cioc, A. et al.: Immersive and collaborative data visualization using virtual reality platforms. In: 2014 IEEE International Conference on Big Data (Big Data), pp. 609–614. IEEE, Washington (2014)
Olshannikova, E., Ometov, A., Koucheryavy, Y., Olsson, T.: Visualizing Big Data with augmented and virtual reality: challenges and research agenda. J. Big Data (2015). https://doi.org/10.1186/s40537-015-0031-2
Napolitano, Rebecca, Blyth, Anna, Glisic, Branko: Virtual environments for visualizing structural health monitoring sensor networks, data, and metadata. Sensors 18, 243 (2018). https://doi.org/10.3390/s18010243
Alaloul, W.S., Liew, M.S., Zawawi, N.A.W.A., Mohammed, B.S.: Industry revolution IR 4.0: future opportunities and challenges in construction industry. In: MATEC Web Conf, vol. 203, p. 02010 (2018). https://doi.org/10.1051/matecconf/201820302010
Dallasega, P., Rauch, E., Linder, C.: Industry 4.0 as an enabler of proximity for construction supply chains: a systematic literature review. Comput. Ind. 99, 205–225 (2018). https://doi.org/10.1016/j.compind.2018.03.039
Wang, P., Wu, P., Wang, J., Chi, H.-L., Wang, X.: A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public Health 15, 1204 (2018). https://doi.org/10.3390/ijerph15061204
Ying, H., Lee, S.: Survey of the research of ICT applications in the AEC industry: a view from two mainstream journals. In: Proceedings of the 16th International Conference on Construction Applications of Virtual Reality, pp. 471–483. The Hong Kong University of Science and Technology, Hong Kong (2016)
Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J.: Sonification report: status of the field and research Agenda. In: Prep Natl Sci Found Memb Int Community Audit Disp (1997)
Munzner, T.: Visualization Analysis and Design. CRC Press, Boca Raton (2015)
Card, S.K., Mackinlay, J.: The structure of the information visualization design space. In: Proceedings of VIZ’97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium, pp. 92–99. IEEE Comput. Soc, Phoenix (1997)
Adcock, M., Barrass, S.: Cultivating Design Patterns for Auditory Displays. In: Proceedings of ICAD 04. Tenth Meeting of the International Conference on Auditory Display. Sydney, Australia (2004)
Barrass, S.: Sonification Design Patterns. In: Proceedings of the 9th International Conference on Auditory Display (ICAD2003). pp. 170–175. Boston, MA (2003)
Moore, B.C.J.: An Introduction to the Psychology of Hearing, 6th edn. Bingley, Emerald (2012)
Ware, C.: Visual Thinking for Design: Active Vision, Attention Visual Queries, Gist, Visual Skills, Color, Narrative, Design. Morgan Kaufmann/Elsevier, Amsterdam (2008)
Ferguson, S., Beilharz, K., Calò, C.A.: Navigation of interactive sonifications and visualisations of time-series data using multi-touch computing. J. Multimodal User Interfaces 5, 97–109 (2012). https://doi.org/10.1007/s12193-011-0075-3
Nesbitt, K.V.: A classification of multi-sensory metaphors for understanding abstract data in a virtual environment. In: 2000 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics, pp. 493–498. IEEE Comput. Soc, London (2000)
Chandler, T., Cordeil, M., Czauderna, T. et al.: Immersive analytics. In: 2015 Big Data Visual Analytics (BDVA), pp. 1–8. IEEE, Hobart (2015)
Bell, B., Höllerer, T., Feiner, S.: An annotated situation-awareness aid for augmented reality. In: Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology—UIST 0’02, p. 213. ACM Press, Paris (2002)
Feiner, S., MacIntyre, B., Höllerer, T., Webster, A.: A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. Pers. Technol. 1, 208–217 (1997). https://doi.org/10.1007/BF01682023
Höllerer, T., Feiner, S.: Mobile augmented reality. Telegeoinform. Locat. Comput. Serv. 21, 00533 (2004)
Langlotz, T., Nguyen, T., Schmalstieg, D., Grasset, R.: Next-generation augmented reality browsers: rich, seamless, and adaptive. Proc. IEEE 102, 155–169 (2014). https://doi.org/10.1109/JPROC.2013.2294255
Slay, H., Phillips, M., Vernik, R., Thomas, B.: Interaction modes for augmented reality visualization. In: Proceedings of the 2001 Asia-Pacific Symposium on Information Visualisation, vol 9, pp. 71–75. Australian Computer Society, Inc., Darlinghurst (2001)
Drascic, D., Milgram, P.: Perceptual issues in augmented reality. In: Bolas, M.T., Fisher, S.S., Merritt, J.O. (eds.) Stereoscopic Displays and Virtual Reality Systems III, pp. 123–134. San Jose, CA (1996)
Kruijff, E., Swan, J.E., Feiner, S.: Perceptual issues in augmented reality revisited. In: 2010 IEEE International Symposium on Mixed and Augmented Reality, pp. 3–12. IEEE, Seoul, Korea (South) (2010)
Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: Proceedings of International Conference on Intelligence Analysis (2005)
Yeh, K.-C., Tsai, M.-H., Kang, S.-C.: On-site building information retrieval by using projection-based augmented reality. J. Comput. Civ. Eng. 26, 342–355 (2012). https://doi.org/10.1061/(ASCE)CP.1943-5487.0000156
Riexinger, G., Kluth, A., Olbrich, M., Braun, J.-D., Bauernhansl, T.: Mixed reality for on-site self-instruction and self-inspection with building information models. Procedia CIRP 72, 1124–1129 (2018). https://doi.org/10.1016/j.procir.2018.03.160
Kotranza, A., Lind, D.S., Lok, B.: Real-time evaluation and visualization of learner performance in a mixed-reality environment for clinical breast examination. IEEE Trans. Vis. Comput. Graph 18, 1101–1114 (2012). https://doi.org/10.1109/TVCG.2011.132
Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Cyber-innovation in the STEM classroom, a mixed reality approach. Creat. Educ. 09, 2385–2393 (2018). https://doi.org/10.4236/ce.2018.915179
Messadi, T., Newman, W.E., Braham, A., Nutter, D.: Immersive learning for sustainable building design and construction practices. J. Civ. Eng. Archit. (2017). https://doi.org/10.17265/1934-7359/2017.09.003
Schmidt, E., Riggio, M., Laleicke, P., Barbosa, A., van den Wymelenberg, K.: How monitoring CLT buildings can remove market barriers and support designers in North America: an introduction to preliminary environmental studies. Portuguese J. Struct. Eng. III, 41–48 (2018)
Sorin, E., Lanata, F., Boudaud, C.: Behaviour of timber structures under variable environment through long-term monitoring. In: World Conference on Timber Engineering (WCTE 2016). TU Verlag, Vienna (2016)
Leyder, C., Chatzi, E., Frangi, A.: Structural health monitoring of an innovative timber building. In: Proceedings of the Second International Conference on Performance-based and Life-cycle Structural Engineering (PLSE 2015), pp. 1383–1392. School of Civil Engineering, The University of Queensland, Brisbane, QLD, Australia (2015)
Wang, J., Karsh, E., Finch, G., Cheng, M.: Field measurement of vertical movement and roof moisture performance of the Wood Innovation and Design Centre. In: World Conference on Timber Engineering, pp. 3120–3128 (2016)
Fast, P., Gafner, B., Jackson, R., Li, J.: Case study: an 18 storey tall mass timber hybrid student residence at the University of British Columbia, Vancouver. In: Proceedings of the World Conference on Timber Engineering (WCTE2016), Vienna, Austria, pp. 22–25 (2016)
Mustapha, G., Khondoker, K., Higgins, J.: Structural Performance Monitoring Technology and Data Visualization Tools and Techniques—Featured Case Study. UBC Tallwood House, Victoria (2018)
Oregon State University—College of Forestry: Testing tall wood buildings (2017). https://youtu.be/1GoeTY1U1ls. Accessed 13 Dec 2018
Udomchoksakul, K.: The Peavy Hall OSU in Unreal Engine 4, (2018). https://youtu.be/mk8dKbgd9Ms. Accessed 13 Dec 2018
Pike, W.A., Stasko, J., Chang, R., O’Connell, T.A.: The science of interaction. Inf. Vis. 8, 263–274 (2009). https://doi.org/10.1057/ivs.2009.22
Riggio, M., Shahbaz Badr, A., Prather, E.A., de Amicis, R.: Advancing AEC practice and data literacy using Digital Twins in Spatial Augmented Reality Environments. Monterrey, México (2018)
Hermann, T., Ritter, H.: Listen to your data: model-based sonification for data analysis. In: Lasker GE, Syed MR (eds.) Advances in intelligent computing and multimedia systems, Int. Inst. for Advanced Studies in System research and cybernetics, Windsor, Ontario, pp. 189–194 (1999)
Hunt, A., Hermann, T.: The importance of interaction in sonification. In: Proceedings of the International Conference on Auditory Display (ICAD 2004), Sydney, Australia (2004)
Hassenzahl, M., Platz, A., Burmester, M., Lehner, K.: Hedonic and ergonomic quality aspects determine a software’s appeal. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’00, pp. 201–208. ACM Press, The Hague (2000)
Sonderegger, A., Sauer, J.: The influence of design aesthetics in usability testing: effects on user performance and perceived usability. Appl. Ergon. 41, 403–410 (2010). https://doi.org/10.1016/j.apergo.2009.09.002
Tractinsky, N.: Aesthetics and apparent usability: empirically assessing cultural and methodological issues. In: Proceedings of the SIGCHI Conference on Human factors in Computing Systems—CHI’97, pp. 115–122. ACM Press, Atlanta, Georgia (1997)
Djamasbi, S., Siegel, M., Tullis, T., Dai, R.: Efficiency, trust, and visual appeal: usability testing through eye tracking. In: 2010 43rd Hawaii International Conference on System Sciences, pp. 1–10. IEEE, Honolulu (2010)
Castro-Alonso, J.C., Ayres, P., Paas, F.: Dynamic visualisations and motor skills. In: Huang, W. (ed.) Handbook of Human Centric Visualization, pp. 551–580. Springer, New York (2014)
McAndrew, P., Clough, G.: Affective factors in learning with mobile devices. Big Issues in Mobile Learning: Report of a workshop by the Kaleidoscope Network of Excellence Mobile Learning Initiative, pp. 14–19 (2006)
Santos, M.E.C., Chen, A., Taketomi, T., Yamamoto, G., Miyazaki, J., Kato, H.: Augmented reality learning experiences: survey of prototype design and evaluation. IEEE Trans. Learn. Technol. 7, 38–56 (2014). https://doi.org/10.1109/TLT.2013.37
Welsh, E.T., Wanberg, C.R., Brown, K.G., Simmering, M.J.: E-learning: emerging uses, empirical results and future directions. Int. J. Train. Dev. 7, 245–258 (2003). https://doi.org/10.1046/j.1360-3736.2003.00184.x
Akçayır, M., Akçayır, G.: Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20, 1–11 (2017). https://doi.org/10.1016/j.edurev.2016.11.002
Pfeiffer, V.D.I., Gemballa, S., Bizer, B., Jarodzka, H., Imhof, B., Scheiter, K., Gerjets, P.: Enhancing students’ knowledge of biodiversity in a situated mobile learning scenario: using static and dynamic visualizations in field trips. In: Proceedings of the 8th International Conference on International Conference for the Learning Sciences, vol 2, pp. 204–212. International Society of the Learning Sciences, Utrecht (2008)
Gune, A., De Amicis, R., Simoes, B., Sanchez, C.A., Demirel, H.O.: Graphically hearing: enhancing understanding of geospatial data through an integrated auditory and visual experience. IEEE Comput. Graph Appl. 38, 18–26 (2018). https://doi.org/10.1109/MCG.2018.042731655
Acknowledgements
The Living Lab @ Peavy Hall project is conducted through the TallWood Design Institute with funding by the U.S. Department of Agriculture’s Agricultural Research Service (USDA ARS Agreement No. 58-0202-5-001). The material presented in this contribution is also based upon work that is supported by the National Institute of Food and Agriculture, U.S. Department of Agriculture, McIntire Stennis Project Under 1009740. The author Eric Andrew Prather was supported by AFRI ELI Grant No. 2018-67032-27704, from the USDA National Institute of Food and Agriculture. Findings and conclusions are those of the Authors and do not reflect opinions or views of the supporting agencies.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
De Amicis, R., Riggio, M., Shahbaz Badr, A. et al. Cross-reality environments in smart buildings to advance STEM cyberlearning. Int J Interact Des Manuf 13, 331–348 (2019). https://doi.org/10.1007/s12008-019-00546-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12008-019-00546-x