Elsevier

Computers & Education

Volume 110, July 2017, Pages 88-104
Computers & Education

Mixed-reality learning environments: Integrating mobile interfaces with laboratory test-beds

https://doi.org/10.1016/j.compedu.2017.02.009Get rights and content

Highlights

  • Mobile devices and test-beds can be integrated according to a novel lab education paradigm.

  • Vision-based measurement and control, AR, and touchscreen enhance lab interactions.

  • The proposed paradigm can offer the benefits of hands-on, virtual, and remote labs.

  • An implementation is developed using an iPad and a motor test-bed to teach control.

  • Evaluation with students validates the implementation's educational effectiveness.

Abstract

Even as mobile devices have become increasingly powerful and popular among learners and instructors alike, research involving their comprehensive integration into educational laboratory activities remains largely unexplored. This paper discusses efforts to integrate vision-based measurement and control, augmented reality (AR), and multi-touch interaction on mobile devices in the development of Mixed-Reality Learning Environments (MRLE) that enhance interactions with laboratory test-beds for science and engineering education. A learner points her device at a laboratory test-bed fitted with visual markers while a mobile application supplies a live view of the experiment augmented with interactive media that aid in the visualization of concepts and promote learner engagement. As the learner manipulates the augmented media, her gestures are mapped to commands that alter the behavior of the test-bed on the fly. Running in the background of the mobile application are algorithms performing vision-based estimation and wireless control of the test-bed. In this way, the sensing, storage, computation, and communication (SSCC) capabilities of mobile devices are leveraged to relieve the need for laboratory-grade equipment, improving the cost-effectiveness and portability of platforms to conduct hands-on laboratories. We hypothesize that students using the MRLE platform demonstrate improvement in their knowledge of dynamic systems and control concepts and have generally favorable experiences using the platform. To validate the hypotheses concerning the educational effectiveness and user experience of the MRLEs, an evaluation was conducted with two classes of undergraduate students using an illustrative platform incorporating a tablet computer and motor test-bed to teach concepts of dynamic systems and control. Results of the evaluation validate the hypotheses. The benefits and drawbacks of the MRLEs observed throughout the study are discussed with respect to the traditional hands-on, virtual, and remote laboratory formats.

Introduction

The role that laboratories play in providing valuable learning experiences is well-established (Feisel & Rosa, 2005). In addition to reinforcing concepts introduced by conventional instruction, hands-on laboratory work has been particularly effective at promoting design, collaboration, and social communication skills, which are as desirable in graduates as is a rigorous theoretical background. Although they have enabled learners to put theory into practice, laboratories constantly seek to deliver the most engaging and cost-effective educational experiences possible using the prevailing technology. This is because laboratories require sophisticated laboratory-grade hardware and software to allow for effective interaction with test-beds and visualization of abstract concepts, such as the effects that certain parameters have on the behavior of a system. Thus, limited funding can often reduce access to facilities and inhibit the quality of in-lab activities. Moreover, learners must often focus efforts on installing, calibrating, and troubleshooting equipment at the expense of experimenting and building deep conceptual understanding.

We are living at a time when laptops and desktops are being replaced by smartphones and tablets as the primary personal computers (Bonnington, 2015, Gillett, 2012). In such a rapidly evolving technological landscape, the equipment used in laboratory education is no longer up to date with the expectations of learners, who have become accustomed to mobile, high-quality experiences with interactive media. To meet this challenge, virtual and remote laboratories have been made available on mobile devices, allowing learners to interact with simulated or real experiments from anywhere and at any time (Maiti & Tripathy, 2012). In addition to accessibility and mobility, researchers believe that implementations of mobile remote experiments will better engage and motivate learners (da Silva, Rochadel, Marcelino, Gruber, & Bilessimo, 2013). However, the mobile-based access to virtual and remote laboratories does not address the challenges facing traditional hands-on laboratories, where there have been few efforts for the comprehensive integration of mobile devices and where research into their potential educational and operational benefits remains largely unexplored.

In this paper, a new laboratory approach is presented wherein mobile devices such as smartphones and tablets are employed in roles that go beyond those of traditional graphical interfaces; rather, the devices become responsible for aspects of measurement, estimation, and control of laboratory test-beds. By integrating vision-based control, AR, and touchscreen interaction, mobile devices can provide enhanced interactive experiences with laboratory equipment that may deepen conceptual understanding and improve learner engagement. Specifically, when the camera of a mobile device is pointed towards a laboratory test-bed, the device augments live video with graphics that learners can directly manipulate to control test-beds and perform experiments. Thus, an intimate connection is made between the digital media running on the interface and the physical dynamics of the test-bed. The term mixed-reality learning environment (MRLE) has been used to refer to such systems (Chang, Lee, Wang, & Chen, 2010). To demonstrate the proposed approach, the development of an MRLE is presented that uses a mobile interface and a motor test-bed to teach concepts of dynamic systems and control (i.e., damping, stability, and the effects of pole locations on system response). To assist learners' understanding of these concepts, the mobile interface captures a live video of the test-bed with its camera and projects a 3D virtual motor arm that appears attached with the test-bed, providing a visual aid that learners can use to observe phenomena. Moreover, learners intuitively command the angular position of the motor arm by tapping and dragging on the touchscreen to manipulate the virtual arm. Additional interactive media, such as real-time plots of the system response and an interactive pole-zero plot, engage students by allowing them to conduct inquiry-based investigations with the motor test-bed from their personal devices. For example, by tapping on the interactive pole-zero plot, the learner relocates the closed-loop poles of the system, altering the system's dynamics so that the learner can examine the resulting effects on the system response.

The following is the supplementary data related to this article:

.

The novelty of the work presented in this paper lies both in the technological development and in the research conducted. Specifically, a novel approach is proposed for integrating the capabilities of mobile devices with those of laboratory test-beds in the development of learning environments for teaching dynamic systems and control concepts. In addition to providing interactive mixed-reality visualizations that enhance student engagement, these learning environments reduce the need for significant amounts of laboratory-grade equipment, thus improving the cost-effectiveness of the traditional laboratory. In some cases, these benefits provided by the mobile devices may even facilitate the development of more portable laboratory platforms that utilize learners' devices in various aspects of their operation. To validate the proposed MRLE approach in terms of its educational effectiveness and associated user experiences, a user study was conducted using the developed platform with two classes of undergraduate students. The objective of the study was to examine whether student participants demonstrate significant improvement in content knowledge or report having significantly beneficial experiences after using the MRLE platform compared to before using the MRLE platform and compared to student participants exposed to the content using traditional classroom and hands-on laboratory techniques.

The paper is organized as follows. In Section 2, we present a review of prior applications that utilize mobile devices as tools in laboratory education. Next, Section 3 provides a detailed overview of the proposed approach, outlines the development of an illustrative platform, and describes the design of evaluation conducted with undergraduate students. Section 4 presents the results of evaluation. Then, Section 5 discusses some observations and rationale for the technical approach and evaluation methodology as well as the evaluation results validating the hypotheses concerning the educational impact and user experience associated with the proposed MRLE approach. Finally, Section 6 offers some concluding remarks and future directions of this work.

Section snippets

Background

For many educational institutions, limited resources have made maintaining up-to-date laboratory facilities a serious burden. Moreover, in many areas of the world where there is lack of infrastructure, educators, and finances, access to quality education is simply not available. These issues have attracted research into providing novel technological solutions for science and engineering education that leverage the ubiquitous presence of mobile devices in people's lives. In particular,

Methodology

The applications discussed in Section 2 demonstrate how the sensing, storage, computation, and communication (SSCC) capabilities of mobile devices have been leveraged to provide learners with the information, tools, data, visualizations, measurements, or remote access needed for effective laboratory learning. For some applications, the SSCC capabilities of devices can be integrated further to provide learners with immersive mixed-reality experiences that can enhance their interactions with

Assessment results

To investigate the educational effectiveness of the proposed system, four scores are assigned to each of the participants' pre- and post-assessments, one for each of the three topic areas being tested (i.e., system damping, stability, and poles), and one overall score. Each assessment item was classified as belonging to one or more of the three topic areas. Specifically, damping, stability, poles were included in 8, 5, and 15 assessment items, respectively. Table 1 shows the average percentage

Benefits and drawbacks of MRLE

During the development and evaluation of the MRLE platform, a variety of notable benefits and drawbacks of the proposed MRLE approach were observed vis-à-vis the conventional hands-on, virtual, and remote laboratories. As an extension of traditional hands-on laboratories, the MRLE approach shares some similarities to hands-on laboratories, such as the need for real laboratory equipment that must be purchased and maintained. Moreover, platforms implementing the MRLE approach require learners to

Conclusions

Laboratories must constantly adapt their styles and strategies to changes in the economic and technological landscape. In this paper, a novel approach was proposed in which the hardware and software of mobile devices and the physical dynamics of laboratory test-beds are blended to create mobile mixed-reality learning environments. By developing mobile applications that incorporate interactive plots and AR, unique cost-effective and stimulating hands-on educational experiences can be provided

Acknowledgments

This work is supported in part by the National Science Foundation awards RET Site [grant numbers EEC-1132482 and EEC-1542286], DRK-12 [grant number DRL-1417769], ITEST [grant number DRL-1614085], and GK-12 Fellows DGE [grant number 0741714], and NY Space Grant Consortium [grant number 48240-7887].

References (39)

  • Frank, J.A., Kapila, V. (2016). Towards teleoperation-based interactive learning of robot kinematics using a mobile...
  • J.A. Frank et al.

    Development of mobile interfaces to interact with automatic control experiments [Focus on education]

    IEEE Control Systems

    (2014)
  • F. Gillett

    Why tablets will become our primary computing device

    (2012)
  • R.M. Haralick et al.

    Pose estimation from corresponding point data

    IEEE Transactions on Systems, Man, and Cybernetics

    (1989)
  • R. Hartley et al.

    Multiple view geometry in computer vision

    (2003)
  • T. Hesser et al.

    iPads in the science laboratory: Experience in designing and implementing a paperless chemistry laboratory course

    Journal of STEM Education: Innovations & Research

    (2013)
  • L. Kerawalla et al.

    Making it real: Exploring the potential of augmented reality for teaching primary school science

    Virtual Reality

    (2006)
  • P. Klein et al.

    Video analysis of projectile motion using tablet computers as experimental tools

    Physics Education

    (2014)
  • P. Klein et al.

    Classical experiments revisited: Smartphones and tablet PCs as experimental tools in acoustics and optics

    Physics Education

    (2014)
  • Cited by (79)

    View all citing articles on Scopus
    View full text