Evaluation of the ARTutor augmented reality educational platform in tertiary education

Abstract

Augmented Reality (AR) has seen a significant increase in applications in a number of fields in recent years. One of the areas in which AR has been applied to is education, most commonly by means of augmenting educational books. The present paper builds on previous work by using an AR authoring environment and a mobile application, developed by the authors, in undergraduate courses at the Eastern Macedonia & Thrace Institute of Technology. Using the authoring tool, the students were able to enhance existing secondary education textbooks by adding digital content to them, and, using the mobile application, view the digital content and retrieve information from the textual content of the books by asking questions in natural language form. At the end of the semester, the students were asked to evaluate the ARTutor platform by means of questionnaires and the SECTIONS framework. This study presents and discusses the results of this evaluation exercise and proposes new directions of the research.

Introduction

Augmented Reality is a technology which enriches human senses and mixes real and virtual environments, leading to a new, more informative and stimulating environment. The principle of an AR application is to use a device’s sensors to perceive the environment and to embed dynamic, context-aware and interactive digital content to it. AR applications can be used on many different platforms such as desktops, notebooks and mobile devices; however, AR applications are usually available through mobile devices, such as smartphones and tablets or wearable devices. Built-in cameras, GPS sensors, gyroscopes and other sensors are used to recognize objects, images and scenes. When recognition is successful, relevant digital content becomes available and is displayed on screen. The objective is to seamlessly blend the environment with the digital content. This enables the user to receive more information about his environment that would have otherwise been available. The advantage of AR is not only the increase of available information in the environment, but also the creation of a certain representation of the world which is appealing to the user. For these reasons, AR has been applied to a number of areas including entertainment, tourism, commerce, engineering and others (Bhorkar 2017).

The ability of enhancing what is perceived in the environment with interactive digital content has created an interest in using AR applications for educational purposes. In the field of mobile learning, the new possibilities for teaching and learning have increasingly been recognized by educational researchers (Wu et al. 2013), (Chen et al. 2017). AR is currently being applied across disciplines in primary, secondary and higher education, and has been found to increase the academic success levels and motivations of students (Martin Gutierrez and Meneses Fernandez 2014). In a recent systematic review of 17 studies between 2012 and 2017, it is argued that AR in education can potentially influence the students’ attendance, knowledge transfer, skill acquisition, hands-on digital experience in education in a variety of domains (Fotaris et al. 2017). A review presented in (Bacca et al. 2014) summarizes research where it is shown that AR is useful in increasing students’ motivation and presents learning gains through interaction, and collaboration. Other studies, such as (Shelton and Hedley 2002), conclude that AR is particularly suitable for the instruction of subjects that are not easily experienced and understood because of real-world constraints.

AR is not only beneficial to the individual students’ study, but with the aid of the appropriate software tools it can also be applied to collaborative learning approaches (Martín-Gutiérrez et al. 2015). For example, in (Gonzàlez et al. 2012) the work focuses on cultural heritage education, where students identify places of cultural interest using geolocation and use software to access publicly available multimedia content. In another work outlined in (Kaufmann 2003), students use specialized AR software for the collaborative study of mathematics and geometry.

Apart from mobile applications that use AR for educational purposes, there have been various authoring tools developed to allow teachers to create AR educational scenarios. For example, ARLearn is an authoring tool that supports the creation of training scenarios and serious games based on decision making (Ternier et al. 2012). Also, mLearn is a platform developed by the authors to help teachers create and manage educational field trips and enable students to access the educational material from their mobile phones (Lytridis and Tsinakos 2017).

A common practice when using AR in education is creating augmented books. These are software applications used to scan the pages of traditional printed educational material and display the digital content associated with those pages on a screen. The applications are installed in devices such as mobile phones, tablets, or even personal computers enhanced with cameras. Through the use of AR in printed book pages, books can become dynamic sources of information. Design aspects of mixed reality books have been first studied in (Grasset et al. 2008). A number of studies have shown the benefits in using AR books as learning enhancement tools for motivating children and helping with better understanding of concepts (Cheng and Tsai 2016), (Dünser and Hornecker 2007). Some examples of specialized applications for specific subjects have been introduced in (Ferrer-Torregrosa et al. 2015) for anatomy and in (Weng et al. 2016) for biology. (Karamanoli and Tsinakos 2016) created an Android application that uses the commercial Aurasma tool to provide augmentations in the Greek history book of the 4th grade of primary school. (Hung et al. 2017) developed an animated AR graphics book displaying 3D images of bacteria. It utilized a virtual reality head-mounted display with a small video camera attached at the same level as the students’ eyes and the results showed that AR improved students’ learning performance and raised their motivation. Also, (Tobar-Muñoz et al. 2017) presented an AR game inspired by traditional “pop up” books, which is played in a naturalistic environment. Each page of the book comes to life using book illustrations as AR markers.

It is obvious from the brief review of the literature presented above that there have been significant advances in the use of AR in education. However, there are still many challenges that need to be overcome especially in the area of AR books, summarized in the following:

• a) AR applications are currently being developed to be used with specific books. The development of such specialized applications clearly requires expert programmers. In the absence of suitable authoring tools, a teacher planning to introduce AR in his courses would either need to acquire fairly advanced software development skills or employ expert help in order to realize such plans.

• b) The educational material is, in most cases, built in an AR application and any changes to the material or the book itself would require a redesign of the application, which implies further engagement of skilled programmers to update the application. The additional costs and effort needed might not be attainable, especially for individual books.

• c) In several AR books today, markers are used as a way for an application to identify the pages where there is digital content. Some purpose-made educational material can indeed be created to include these markers. Existing books however, would need to be redesigned and reprinted, and this would incur a large unnecessary cost.

• d) AR applications are usually targeted to individual books and so the study of multiple subjects using AR by a student demands the installation of multiple applications.

• e) Current AR books only allow students to interact with the digital objects that appear on top of the physical books. These additional sources of information do lead to a better understanding of the learning material, but information retrieval from the textual content of the book is not aided by AR.

The identification of these problems has led the authors to develop ARTutor, a general-purpose AR-based educational platform, first presented in (Lytridis et al. 2018). The aim of this platform is to facilitate the creation of AR books by providing an easy to use authoring tool, and to provide a single mobile application that will work with every book created with the authoring tool.

The current paper describes the first experiences in integrating ARTutor in a higher education course at the Eastern Macedonia and Thrace Institute of Technology in Greece and the results of its first large scale evaluation. The paper is outlined as follows: In Section “The ARTutor platform” the ARTutor platform and its functionality is presented. Section “Evaluation of ARTutor” describes how ARTutor was made part of the undergraduate courses and how the software was evaluated. Section “Conclusions” discusses the experimental results and proposes future directions in the research.

The ARTutor platform

ARTutor is an educational platform that aims at integrating augmented reality in the educational process in a simple way for both educators and students. It has been developed at the Advanced Educational Technologies and Mobile Applications Research Lab at the Eastern Macedonia and Thrace Institute of Technology. Its focus is on adding digital content on traditional educational books and other texts, with the aim of assisting the students’ independent study and ultimately improving the understanding of the material.

In this section, an overview of the functionality of the ARTutor authoring tool and mobile application is presented.

The authoring tool

The authoring tool is a web-based application designed to enable teachers to create AR books (Lytridis 2017a). To access the authoring tool, the teacher must visit the ARTutor portal and register for free. The portal also allows users to browse and download existing AR books. After successful login to the system, AR books previously created by the user (with an option to edit or delete them) and the option to create a new AR book are displayed. When creating a new book, the form shown in Fig. 1 appears. In this form, the teacher needs to enter some information on the book, to select a book category and the book language (Greek and English are currently available), and upload his educational material (the book) in PDF format. The file is uploaded with a simple drag & drop action to the corresponding area.

Fig. 1

a Existing books table and (b) book creation form

After the uploading of the book is complete, the teacher saves all the book information. During this process, the PDF file is stored on the server and the book information on a database. Also, the text from the PDF file is extracted and indexed in order to support an information retrieval algorithm, as it will be described later. At this point, the teacher is now able to add augmentations to the book. To add an augmentation to the book, the teacher must create an asset, i.e. an object which associates an area of some page of the book to a digital object (i.e. the augmentation). This is done through an asset creation form, where the teacher must give a name to the asset, create the trigger image and upload the corresponding augmentation. As illustrated in Fig. 2, the creation of the trigger image is achieved by opening the PDF file, selecting a page, and marking the desired area of the page. When the area is selected, it is saved as an image and displayed in the asset creation form. Then, the teacher imports the corresponding digital object, which can be an image, a video, an audio clip or a 3D model.

Fig. 2

Asset creation

In the same way, the teacher can add multiple augmentations to the book. All assets are displayed in a table below the book information form, where they can be edited or deleted.

Τhe authoring tool is designed in such a way that teachers can enhance their traditional educational material, even if their IT skills are limited. Books do not need to be reprinted to include specialized markers, as is the case in other approaches. The simplicity of the process of creating an augmented book means that there is no need for specialized training in the use of the software or required software and design skills. In addition, the teacher is not expected to use other software tools in conjunction with the ARTutor authoring tool, since (a) the original books are common PDF files, and (b) the augmentations added in the books’ pages are common images, videos and audio clips that can be retrieved from various sources or created using non-specialized software. In contrast to other general-purpose augmented reality authoring tools, the trigger images are created on the ARTutor authoring tool itself, and do not need to be generated and uploaded separately for each page of the same book. This approach not only allows the easy creation of trigger images, but it also ensures that each book is treated as a single entity, and not as a collection of target images. The ARTutor authoring tool is free to use for academic purposes, and the resulting augmented books are to be visible both in the portal as well as to everyone using the mobile application.

The mobile application

The ARTutor mobile application is the tool which students use to scan the learning material used in the authoring tool, view the corresponding digital content and interact with the augmentations and book content. The application is available for both Android and iOS platforms (Lytridis 2017b).

Initially the application displays all the book categories and the number of books in each category. Selecting a category reveals the books of this category as defined in the authoring tool as seen in Fig. 3.

Fig. 3

The ARTutor mobile application user interface

By selecting a book, the application retrieves all the relevant assets from the server and initiates the camera. The student can then start scanning the various pages of the book and when a trigger image is detected, the corresponding augmentation is displayed. Example augmentations after trigger image detection are shown in Fig. 4.

Fig. 4

Example augmentations using the ARTutor mobile application

As it can be seen in Fig. 4, apart from the augmentations, when a trigger image is detected various additional buttons appear on screen. With these buttons the student can view which pages of the book contain augmentations and see which commands are available for each displayed augmentation. The commands are ways in which the student can interact with the augmentations either using touch gestures or by voice commands. Table 1 shows the available commands depending on the type of displayed augmentation.

Table 1 Commands for interacting with augmentations

To issue a verbal command, the user must press the ARTutor button displayed on the bottom right hand corner of the screen. Voice commands allow hands-free operation and also provide an accessibility feature which enables students with limited motor skills to interact with the augmentations.

The mobile application also has a novel feature where the student can ask questions in natural language form and receive the answer based on the textual content of the book. As was mentioned earlier, when the book is created, its textual contents are extracted, indexed and stored on the server. The student can use the ARTutor button to ask a question. The question is then sent to the server where an information retrieval algorithm is used to extract the relevant information and return the appropriate answer. The answer is sent back to the application where it is converted to speech and also displayed on the screen as a popup message. The capability if the application to perform searches on the content of the educational material is a novel approach designed to reinforce the self-study practice and independent learning process by providing a virtual tutor who, based on the learning material, will provide relevant information according to the students’ questions.

The ARTutor mobile application allows students to enhance their understanding of the teacher’s material by displaying explanatory interactive digital content on top of the traditional book. The concepts presented in the various pages of the book are better visualized using the various types of augmentations and additional information can be made available. The augmentations can be activated either by pointing to the electronic or the printed version of the book. Contrary to other augmented reality applications, the ARTutor application is not developed for a specific book: the content accessible to students is dynamic and more books can be accessed by the application as more books are added to the authoring tool.

In summary, the ARTutor platform is a novel educational tool with the following advantages:

• Easy enhancement of existing learning material by teachers

• Easy access to and interaction with the enhanced digital content by students

• Voice-based interaction with a virtual tutor for information retrieval

• Consideration of students with disabilities by the introduction of accessibility features

• Possibility of collaborative development of AR content

• The same book can have multiple versions. This allows the possibility of each teacher/user adding different content on a book, perhaps concentrating on different aspects of its content, and the students selecting the best or more appropriate version for their study.

Evaluation of ARTutor

As reported in previous work, ARTutor has already undergone the first stage of formative evaluation by a small number of users and experts. After some initial adjustments on the functionality and the user interface of the authoring tool and the mobile application during this initial stage, the platform was ready to be assessed by a larger number of users and test the entirety of its features. In this section the experimental setup and the results will be presented.

Setup

For the task of using and evaluating the ARTutor platform, students attending two undergraduate units, namely “Learning and Teaching Theories” and “Educational Technologies” were engaged. These units are taught during the second and fourth semester of the undergraduate studies of the Computing and Informatics Department at the Eastern Macedonia and Thrace Institute of Technology. ARTutor was made part of the semester’s coursework in both units in the following way: A set of 15 books taught in secondary Greek schools were selected, in subjects such as information technology, history, mathematics and geography. The electronic versions of these books are freely available in PDF format from the Greek Ministry of Education. These subjects were selected because of their potential in enhancing the material with graphics of educational value. Students of these two units were divided in groups of five and each group was assigned a portion of one of the selected school books. The assignment given to the students was to study their allocated portion of the book, identify images that can be used as triggers, and find multimedia content (images, videos, audio clips and 3D models) that would serve as augmentations and enhance the book content. They were also required to subsequently use the mobile application to view and interact with their augmentations. In this way, the students were able to experience the platform both as teachers as well as students.

In the case of the “Learning and Teaching Theories” unit, the students were also instructed to focus on the development of original educational videos relevant to the subject of their allocated book pages. All groups of students were then required to add their multimedia content to the books that were prepared in ARTutor beforehand. It should be noted that all students used the same ARTutor account, which means that groups assigned to different portions of the same book were working in parallel and essentially developing each augmented book collaboratively.

Moreover, the students attending the “Educational Technologies” unit were given group assignments where they were required to evaluate ARTutor using the SECTIONS evaluation framework, initially proposed in (Bates and Poole 2003), in order to determine the most effective use of the technology in various subjects and in various academic settings. The students submitted reports which contained their group analysis for each of the criteria of the SECTIONS framework, i.e. Students, Ease of use, Cost, Teaching and learning, Interactivity, Organizational issues, Novelty, Speed. The objective of this exercise was to gather students’ opinion on these issues and particularly on the suitability of this educational technology for all the stages of educations as well as for adult education and professional training.

In addition, at the end of the semester, the students of both units were asked to individually complete a questionnaire designed to assess the functionality of ARTutor in terms of ease of use and usefulness according to the Technology Acceptance Model (TAM). According to TAM, the user-perceived ease of use and usefulness of a technology are two determinants of its adoption (Davis 1989). The questionnaire was based on the Software Usability Measurement Inventory (SUMI) developed in the “Metrics for Usability Standards in Computing” project by the Human Factors Research Group at the University College Cork (Bevan and Macleod 1994). It contains 50 questions that the user has to reply to with either “Agree”, “Disagree” or “Don’t Know”. More specifically, the questionnaire investigates efficiency, affect, helpfulness, controllability and learnability. Finally, at the end of the questionnaire the students were given the opportunity to comment on which aspects of the software they found useful or problematic.

Results

In this section, the results gathered from the questionnaires will be presented and discussed and a summary of the analyses based on the SECTIONS framework will be provided.

Questionnaires

In total, 205 students completed the evaluation questionnaires. Initially, the degree to which the students found the software important and their computer skills were examined with two corresponding questions, as seen in Fig. 5.

Fig. 5

Importance of the technology and software skills of the students

The results presented in the charts above indicate the students’ motivation and incentive to use ARTutor and signify that the test group has average to advanced IT skills.

Questions 1–10 of the questionnaire are dealing with how the software affects users and are presented in Table 2. Overall, students find that the software made them feel comfortable while using it, with helpful prompts and instructions, reliable operation and easy-to-learn tasks. Finally, nearly half of the students would recommend the software to their colleagues as opposed to 16% that would not.

Table 2 Questions 1–10 on affect

Table 3 shows the questions and the student answers in questions regarding controllability. It can be seen that the users feel strongly that the user interface is well-designed and the various tasks are performed quickly. It is also reported that learning to use new functions is not difficult to do and feel confident that they can learn all the functions that the software has to offer. Users generally feel in control when using the software.

Table 3 Questions 11–20 on controllability

In terms of efficiency, students are generally satisfied with working with ARTutor. They report to be very comfortable with the way information is presented and feel that the accompanying documentation is informative. The students also report that they prefer to use functions that they are more familiar with (Table 4).

Table 4 Questions 21–30 on efficiency

As far as helpfulness is concerned, students appear to be quite satisfied with getting help from the software itself. They are not frustrated when using the software as the information provided is adequate and that tasks are straightforward to be performed in a timely manner, and it is reported that the need to go back to the documentation is rare. The results for this category of questions are presented in more detail in Table 5.

Table 5 Questions 31–40 on helpfulness

As far as learnability is concerned, students feel that it is fairly easy to learn to use the platform: the majority of students do not find it awkward and think that it is easy to gradually complete tasks with information on how to proceed at each stage being available. Ease of use is also asserted by the fact that most students feel that it is hard to forget how to use the software. Results for the learnability category are presented in Table 6.

Table 6 Questions 41–50 on learnability

As far as the students’ general comments on the pros and cons of the platform, the students mostly reported that they found the use of both the authoring tool and the mobile application very easy, and for most tasks they did not need to refer to the software’s documentation. They found the ability to interact with the augmentations quite useful and they were particularly interested in the questions they could pose to the ARTutor agent in order to retrieve information from the book’s contents, although it was reported that this system did not yield good results in some cases. On the other hand, negative comments mainly focused on the fact that the application required a lot of memory for books with many augmentations assigned to it. Students also found that even though the inclusion of 3D models in an AR book is appealing, it is not an easy task mainly because it requires 3D design skills given the scarce availability of free 3D models, especially those that are explanatory of some concept. It appears that 3D models are the most promising in terms of the capability of illustrating concepts and consequently they are of an important educational value.

SECTIONS framework

The most prominent students’ views are featured here, categorized by criterion:

• Students

  On this criterion there has been extensive analysis on the suitability of ARTutor at the various stages of education. Most students have discussed that the limitation in the adoption of ARTutor as an educational technology is the ability of students to use mobile devices. Since this happens at an early age, the possibility of using ARTutor as early as in primary education was suggested.

• Ease of use

  This was universally recognized as a prominent characteristic of the technology for both students and teachers, even those that have limited experience with software tools. Tasks are reported to be intuitively easily achieved and there were no issues with the user interface. However, there have been some criticism as far as the reliability of the mobile application was concerned, due to some bugs that were found on the Android version.

• Cost

  In this criterion the students stated that the registration to the authoring tool and download of the mobile application was free.

• Teaching and learning

  Here, students have focused on the learning theories that are compatible with ARTutor as a learning tool. They report that constructionism and independent learning will be the underlying learning theories in courses where ARTutor is employed. It is highlighted that videos and especially 3D models are the types of content to impact the better understanding of concepts more significantly.

• Interactivity

  It was generally noted that a strong interactivity feature was the ability of students to interact with the augmentations via gestures and voice commands. Some students interpreted the ability to ask questions as a form of student-teacher interaction. The lack of student-student interaction capabilities and the absence of a feedback mechanism were also noted.

• Organizational issues

  Most students agree that there are no organizational issues that would prevent the choice of this educational technology, since it is designed for self-study and so no specialized equipment or technical staff would be required at the institute adopting it. The teachers would only need a computer and an internet connection to develop and upload his educational material, and the students would only have to download the corresponding mobile application.

• Novelty

  Students collectively mentioned that ARTutor is a novel application suited to education and highlighted differences between other AR platforms designed for advertisement and commercial purposes.

• Speed

  Due to the simplicity of the processes involved in creating AR books that is offered by the platform, an educator wanting to integrate ARTutor to his courses would not require a long time to adopt this technology. In addition, the fact that existing books and notes are used with ARTutor means that the educational material would not need modifications in order to incorporate AR, except for the virtual addition of the digital content.

Conclusions

This paper presented the results of an evaluation study of the ARTutor platform, carried out by students at the Department of Computing and Informatics at the Eastern Macedonia and Thrace Institute of Technology. The value of the platform in terms of user-perceived ease of use and usefulness as well as its suitability for AR-based learning were assessed. The main theme which arises from the large-scale evaluation of the software is its evident ease of use both from the point of view of the teacher adding augmentations to books, as well as from the point of view of the student that uses the mobile application to view the interactive digital content superimposed on the printed book. This reinforces the argument in favor of ARTutor’s suitability as a general-purpose AR educational tool.

The evaluation of the platform by a large number of users has led to a number of improvements of the ARTutor platform in terms of user interface and functionality. Feedback from students has been particularly useful in highlighting issues which were not evident in the first phase of the evaluation and in suggesting additions that could improve the user experience. It must be noted that the results are clearly influenced by the students’ subject of study. It therefore has to be assumed that their evaluation was more technically-oriented, rather than focused on the educational aspects.

While the usability aspect of the ARTutor platform was investigated in this paper, it is crucial to also establish the educational benefits of this educational technology and determine its effect on the learning process. The obvious next step is therefore to use ARTutor as part of the curriculum in the various stages of education: primary, secondary and tertiary. The results of such a systematic study would help establish a link between the educational use of AR and learning outcomes. For this study, the books already developed by the students could be used, or teachers could participate by developing material for their own courses. In this way, teacher attitudes towards AR and ARTutor in particular could be measured. In addition, collaborative course material development by students for their own courses can also be investigated in order to test whether their motivation is increased.

The software itself can be further developed. For example, the possibility of cloud-based recognition can be explored, in order to solve the problem with phone memory usage and large downloads. Also, the characteristics and preferences of the student can be taken into account in order to create a student profile and therefore deliver a more personalized experience when using the application. Moreover, the authors are currently investigating the incorporation of ARTutor with a Learning Management System (LMS) such as Moodle so that ARTutor is more integrated with distance learning courses. Other aspects such as support for more languages for the user interface and the information retrieval algorithm and improvements in accessibility are also considered important and will be pursued further.