Teachers’ evaluation of MotionAR: An augmented reality-based motion graphing application

This research presents the results of the initial evaluation of MotionAR: an augmented reality (AR) based motion graphing application for physics classes. MotionAR uses AR trackers to measure relative motion between objects, hence producing a motion graph. The subject of the evaluation is an improved version of our application that addresses recommendations from the first prototype. Among these recommendations were: the use of words instead of arrows to identify targets, the use of the Cartesian coordinate system, and previews of the displacement vs. time, velocity vs. time and acceleration vs. time graphs. In-service teachers used a 10-item questionnaire for evaluation. The teacher-evaluators come from diverse educational backgrounds (rural and urban schools, 0 to 24 years in teaching, no e-learning model in school vs 1:1 student-to-tablet PC ratio, science vs. non-science majors) which makes the initial evaluation a good baseline data. This ongoing research will present the lessons learned from the evaluation, including insights in developing AR applications for physics classes, and recommendations for teaching motion graphs utilizing AR.


Introduction
Emerging technologies affect classroom environments as these can improve or enhance existing ones. When these technologies get introduced to the classroom, evaluations may need to be conducted so that both instructors and students can benefit from these new affordances. To put this idea into context, students learn about the concepts of displacement, velocity, and acceleration in an introductory physics class on mechanics. They also learn visualizations in the form of motion graphs, aside from the usual kinematic equations. Students are expected to understand and interpret the motion based on the given graph.
We explore the effectiveness of using augmented reality (AR) in developing this motion graphing application using a handheld device (e.g. tablet). Our application, which we call MotionAR, uses AR marker registration and tracking to measure distance between the markers. In essence, we developed MotionAR to be used by both students and teachers. Instructors can change the real objects (where the AR markers will be attached to) to make the lesson interesting. However, we perceive that users might have difficulty in using this application. This might be true for teachers that have no prior exposure or experience in AR. We measure this claim through a user study, where teachers evaluate our application upon using it. The observations derived from the study can be used to design guidelines on the use of the application, as well as to contribute further knowledge how fast and effective AR can be adopted in learning environments. IOP  In summary, we survey existing literature about AR and how it can affect learning in classroom settings. Then, we describe the development of Motion AR alongside an investigation of its usability through user studies.

Use of Augmented Reality in Learning Environments
Various learning activities employing AR have been found to be effective. A meta-analysis [1] of 87 related studies reveals a 0.56 effect size or a moderate effect of augmented reality learning experiences (ARLEs). Furthermore, another meta-analysis [2] of 32 research journals shows that AR has a positive effect on learning performance, motivation, student engagement and attitude.
An example of an ARLE for a physics class involves an AR-based application on linear momentum and collisions [3]. This was used in a physics class for 36 undergraduate students. A student changes the momentum of a virtual object by adjusting its mass and velocity. A partner student also changes the momentum of another virtual object. These two objects are made to collide on a shared virtual space between the two students. The experimental design of the study shows that students in the group which utilized the ARLE have significantly higher post-test scores than students in the control group.
Another affordance of AR is to help in the visualization of what is physically invisible, as in the case of magnetic fields. An AR system on magnetic fields [4] that arise from magnets, current-carrying wires, and their interactions has been designed. The potential use of this AR is on learning activities on magnetism. Invisible forces were also made visible through another ARLE that simulates that of a museum walk where students are free to interact with the exhibits [5]. In this ARLE, students wore ARgoggles that visualize the forces acting on a cart moving along a track. Such activity increased student engagement, motivation, concept understanding and test scores.
The ultimate objective of MotionAR evaluated in this research and currently being further developed is to increase conceptual understanding of motion graphs which results in increased test scores.

On Teachers' Use of Technology in the Classroom
Teachers affect the use of technology in the classroom due to their role in facilitating classroom-based learning. Their preference to integrate technology is affected by a variety of factors.
Intrinsic (such as attitudes and beliefs) and extrinsic factors (such as access, time, and training) affect teachers from designing technology-based learning activities [6]. Using perceptual control theory [7], researchers explored some of these intrinsic factors. The study shows that teachers will most likely infuse technology in their classrooms when the following three conditions are met: that technology (a) can help in achieving higher goals compared to an existing method, (b) will not disturb the other higher goals being aimed for, and (c) is accessible and usable. Meanwhile, a recent study [8] identified both intrinsic and extrinsic factors; technical problems, large number of students, lack of professional development training, and negative perceptions about the impact of technology on learning discourage teachers' use of technology.
Another study [9] used expectancy-value theory to derive an equation to serve as a heuristic in determining teacher's use of technology. The researchers found that the use of technology is dependent on three factors. The teacher's expectancy of success affects preference to use technology the most, while cost has the least effect. Perceived use of technology is also a significant variable. This result resonates with a research [10] on pre-service teachers: the perceived use of technology is the primary driver for technology integration in the classroom. Further, teacher's knowledge (especially technological pedagogical content knowledge or TPCK), self-efficacy (which includes allowing teachers to play with technology to develop self-confidence), pedagogical beliefs, and culture (a supportive environment in encouraging technology integration) are some variables that affect technology integration in the classroom [11]. In summary, teachers consider internal and external factors that impact their decision to implement technology-enhanced learning activities. Some of these factors are reflected on the evaluation form used in this study.

Research Questions
This research is descriptive in nature, aimed at gathering baseline data on how in-service teachers find the MotionAR. As an on-going project, the results of this evaluation will be used to refine the features of the application in the upcoming versions. The statements found on the evaluation form sought to answer the following questions: 1) Is the application acceptable for teachers practice? 2) Is the application perceived to have good usability?

Respondents
In-service teachers (N = 29) served as evaluators of the motion graphing application. They were participants of an educational technology training provided by the school where one of the researcher teaches. The schools where the evaluators come from implement an e-learning program at varying degrees, which makes them appropriate for the study. Other demographic details of the respondents are mentioned in the general discussion area.

Procedure
Changes in some features of the application were introduced to it after its presentation at the International Conference on Computers in Education (ICCE) in Hangzhou, China in December 2015 [12]. One of the authors worked at developing the second version of the application for a period of four months. Meanwhile, another author developed the 10-item survey form that was used in this study. The survey questions are focused on the perceived level of usefulness of the application, as well as potential challenges in trying to integrate the application to classroom instruction. Lastly, the evaluation was conducted in May 2015 during the Summer Institute on Technology in Education (SITE), an educational technology seminar provided by Miriam College High School where one of the researcher teaches. There were 30 respondents in the original pool of evaluators. One of the evaluators did not completely fill-out the evaluation form and thus was removed from the data pool. The 30 respondents were distributed in three groups (13, 9 and 8 teachers in separate groups). Each group attended a demonstration of the motion graphing application facilitated by one of the researchers. A picture of the demonstration appears in figure 1. The novelty of the application, unlike other commonly available methods (ticker tape, video loggers, ultrasonic motion sensor) is the motivation for the brief training on how to use MotionAR.
The demonstration focused on familiarizing the evaluators to the user interface of the application, teaching them how set-up an experiment, discussing the function of each button, running the application to gather data and showing the three motion graphs that can be viewed. The demonstration for each group was conducted twice. After the demonstration, the evaluators had hands-on interaction with the application. The activity lasted for 15 minutes before the evaluation forms were given. Figure 2 shows a picture of the evaluators trying the application for themselves.
Setting up the application first requires two pre-defined AR markers to be attached to objects. One marker is attached to a fixed object (the reference point) while the other marker is attached to another object. The distance between the two markers is measured by the application; this makes plotting the motion graphs possible. Figure 3 shows a screenshot of the application while figure 4 shows a sample implementation of the system with a recommended setup for an experiment on law of acceleration.  Figure 1: During the demonstration, the user interface that appears on the mobile device is projected on a screen (a) for all evaluators to see. MotionAR was installed to a cellular phone mounted on a tripod (b) to increase the accuracy of gathering data.

Data Analysis
The survey items that yielded numerical data were analysed using simple frequency count, means or percentages. Responses to open-ended questions were encoded, and then used to support some items for recommendation.
The survey instrument used consists of 10 statements either stated in the positive or in the negative. Evaluators signify their agreement or disagreement using a 7-point, Likert scale (1 -strongly disagree, 2 -disagree, 3 -slightly disagree, 4 -neutral, 5 -slightly agree, 6 -agree, 7 -strongly agree). We grouped responses according to level of agreement or disagreement, and we computed the corresponding percentage for analysis.  In reality, the AR markers only appear as printed images; however this is augmented in the application by identifying which of the markers is the reference point or the object, and measuring the distance between the two markers.

New Features Introduced in MotionAR
We designed the handheld AR motion graphing application to view the graphs in different modes while an object moves along a straight line. Users need to make sure that they can view two Objects A and B (with AR marker attached to each) within the device frame (see figure 3). Initially, the device can only detect these markers. Since our application is handheld, we limit the recording to detect only one dimension to keep the graphs simple. We also keep these objects moving in only one line so that the scale of the markers do not affect the distance computation. Thus, we adopt the physical constraints of current commercially available motion graphing instrument that works based on ultrasonic wave detection.
The distance is measured per frame as Object A moves toward Object B. The application transforms raw distance data into visual information via the different motion graphs. Users can use buttons on the side of the device to switch among three modes: displacement vs. time, velocity vs. time, and acceleration vs. time. The application updates the graph every frame, which can now accommodate negative values (e.g. position to the left of the origin, decreasing velocity). This means the graph also adjusts the position of the x-axis and the scale of the y-axis upon update.
We developed this second version of our application using the Unity 5 (https://unity3d.com) game engine and Vuforia (https://vuforia.com/engine) for the AR technology. We used the standard rock and wood images provided by Vuforia as our standard markers for Objects A and B. With this setup, potential developers can modify the application to add more features, or change the tracked marker into other images to suit a certain context.

Other Demographic Details of Respondents
The 29 evaluators come from a diverse background in terms of teaching experience, subjects taught, undergraduate degree and nature of e-learning program in the school. The data represent a total of 24 private schools in the Philippines, mostly in urban Manila (n = 15), while the rest is in rural regions. The average years in teaching of the evaluators is 9.24 years, ranging from 0 to 40 years. Table 1 shows a frequency count of years in teaching, grouped in ranges. Some of the teachers (n = 13) are multiple preparation teachers, which means that they teach at least two different grade levels of the same subject, and/or at least two different subjects. In terms of subject categories, 22 teachers teach Science, 5 teachers teach Math, while four and three teacher teach computer or ICT and robotics, respectively.
The academic preparation of the evaluators also varies. Those with science education degree consists 48% of the evaluators, followed by elementary education degree holders (21%). Five evaluators have unrelated pre-service academic preparations while two respondents had an incomplete response for this survey question.
The schools where the evaluators teach differ at the level of implementing an e-learning program in terms of gadget use. Twenty schools are in a transition phase and plans to use tablet computers within five years, and one school uses desktops. There are eight schools which use tablet computers; one of those schools has a 1:1 student-to-tablet ratio, five schools lend common tablets to its students, and two schools have 80% of their student population with tablets. Most teachers (93%) believe that the application is useful for teaching physics. Old teachers and those who have not taught physics were the ones who find the application difficult to use. Due to the perceived usefulness, all the evaluators also believe that the system will help students understand the physics topic better. The affordance of the application to generate the motion graphs in real time, and ease in setting up may be the drivers for this perception. During the application demonstration, the evaluators succeeded in setting up an experiment within 15 minutes, and this was only done once. Such user experience is related to ease of use. Teachers may also find the application easy to use according to 83% of the evaluators. Further, most evaluators (69%) also believe that students will find the application easy to use, and all agree that students will enjoy using the application. Considering that students nowadays are "digital natives", students may find the application easier to navigate than how the evaluators find it.

Results of the Evaluation
The academic preparation of the evaluators appears to impact their perception of the application. It is noticeable from the data that evaluators who do not have an undergraduate degree in science education believe that the application is not useful in physics teaching. Further, the evaluators who feel that teachers will not be comfortable in using the application, and that other teachers will not be interested in using the application, do not have a science-related academic preparation. Also, evaluators who believe that the novelty of the method and adeptness of the students with the device are sources of difficulty in using the application, stating, "They're (the students) are not used to this kind of teaching," and, "There are students who are not (adept) with gadgets, though others (are)." These perceived issues can be addressed with proper training and by allotting appropriate teacher "playtime" with the application to develop confidence in using the software [11]. As already stated, the novelty of MotionAR, unlike other commonly available methods, is the reason for conducting a brief demonstration on how to use the system. With many of the evaluators believing that the application is student-friendly, 86% of the teachers feel that the application is well-designed (3 evaluators responded 'neutral' while one evaluator did not provide a response). Specifically, the application was developed according to sound principles of designing AR apps [13]. Almost all the evaluators (97%) feel that the application will not be distracting for the students (the evaluator who thinks otherwise does not have teaching experience yet). The taskbased nature of the user interface should make the student focus on one task at a time.
As for teacher's lesson planning, 83% of the evaluators think that integrating the use of the application will not cause much change in writing the lesson plan. Inquiry-based learning (IBL) method in teaching physics is recommended by the state's education department. IBL in science includes handson activities and experiments that MotionAR can provide. Most evaluators (72%) agree that teachers will not be hesitant to use the application, and that majority of the evaluators (97%) believe that their colleagues will find the application useful.
Evaluators also provided recommendations for the future versions of the application. These recommendations include an offline mode (the current version of the application requires constant internet connection), a video save option where the graphs are animated while the video is being reviewed, and tabular presentation of gathered data.

Summary of Findings, Recommendations, and Future Work
In this paper, the second version of an AR-based motion graphing application, which we call MotionAR, was presented in terms of how 29 teacher-evaluators find its usefulness. A 10-item evaluation form was used to gather evaluator feedback. Despite the variety of the background of the evaluators in terms of years in teaching, academic preparation, and school where employed make the data, the application is perceived to be useful in teaching physics, specifically motion graphs. It is recommended that changes to the application be introduced and that expert evaluation be performed in the future.
The researchers plan to implement further work on the development of the application to yield a third version that shall undergo expert evaluation. Changes on the application will include some minor user interface changes (such as pausing the camera after gathering data) and data smoothening (changing the IOP Conf. Series: Journal of Physics: Conf. Series 1286 (2019) 012051 IOP Publishing doi:10.1088/1742-6596/1286/1/012051 8 sampling frequency), maximizing the screen area for a graph to enhance closer inspection and juxtaposing all graphs in one view. 14 13 0 0 0 0 2 0 I think the students will find this system difficult to use. Teachers will not be comfortable to use this system for physics classes. My co-teachers would be interested to use this system for their classes.