The usability attributes and evaluation measurements of mobile media AR (augmented reality)

: This research aims to develop a tool for creating user-based design interfaces in mobile augmented reality (MAR) education. To develop a design interface evaluation tool, previous literature was examined for key design elements in the educational usage of MAR. The evaluation criteria identified were presence, affor-dance, and usability. The research used a focus group interview with 7 AR experts to develop a basic usability evaluation checklist, which was submitted to factor analysis for reliability by 122 experts in practice and academia. Based on this checklist, a MAR usability design interface test was conducted with seven fourth-grade elementary students. Then, it conducted follow-up structured interviews and questionnaires. This resulted in 29 questions being developed for the MAR interface design checklist.


Introduction
Augmented reality (AR) is an interactive technology, involving advanced hardware and graphics processing technology, rapidly emerging within the wider advent of ubiquitous environments and the prevalence of smart phones. AR enables the overlay of computer-generated digital images or information onto either a live direct or indirect real-world environment in real time (Azuma, 1997). From the user's perspective, AR seamlessly connects the gap between the real and virtual information, merging them as one environment (Chang, Morreale, & Medicherla, 2010;Hollerer & Feiner, 2004).

PUBLIC INTEREST STATEMENT
This study aimed to examine the usability attributes of mobile media augmented reality (AR) and to develop a usability evaluation tool via concretization. The objective of this study was to determine the method for applying the usability attributes of AR to interface design rationally. AR is being developed rapidly for mobile devices. In particular, educators are constantly looking for new ways of teaching students, and consider AR as the potential for pedagogical applications. For the utilization of mobile augmented reality (MAR) for education, an interface design needs to ensure universal accessibility for users or learners, and proper usability evaluation measurements to evaluate them are also needed.
Furthermore, AR is being developed rapidly for mobile devices. With mobile devices, AR services can reach an extensive set of use cases and scenarios in various mobile environments (Wagner, Schmalstieg, & Bischof, 2009). In particular, educators are constantly looking for new ways of teaching students, and numerous researchers consider AR as one of the new technologies that have the potential for pedagogical applications (Bower, 2008;Dalgarno & Lee, 2010;Dunleavy, Dede, & Mitchell, 2009;Johnson, Smith, Willis, Levine, & Haywood, 2011;Kye & Kim, 2008). Billinghurst (2002) pointed out that the use of tangibles as a metaphor for object manipulation and the ability to smoothly transition between reality and virtuality can create new educational experiences. Besides that, the emergence and widespread ownership of mobile devices has led to an increased interest in integrating the benefits of mobile learning and AR applications.
The advancements in handheld computing have opened new opportunities for AR (Martin et al., 2011;Squire & Klopfer, 2007). Interestingly, the development and rapid increase in mobile phone usage have made mobile augmented reality (MAR) possible (Azuma, Baillot, Behringer, & Feiner, 2001;Papagiannakis, Singh, & Magnenat-Thalmann, 2008), which is also expanding rapidly. However, like many other innovations, the educational values of AR are not strictly based on the use of technologies but also on how AR is designed, implemented, and integrated into formal and informal learning settings. Gabbard et al. (2005) argue that user-based experiments are critical for driving design activities, usability, and discovery early in an emerging technology's development, such as in the case of AR. For the utilization of MAR for education, an interface design needs to ensure universal accessibility for users or learners, and proper usability evaluation measurements to evaluate them are also needed.
Nevertheless, educational research concerning the MAR learning system is in its infancy (Martin et al., 2011). Besides, while many studies have looked at the development of technologies or contents of AR, very few studies have evaluated the usability of AR. Because these studies are led by technologies, usability researchers point out that they lack consideration of user convenience and usability (Dünser, Grasset, & Billinghurst, 2008;Livingston, 2005;Nilsson & Johansson, 2006). Most previous studies have evaluated the aspects of cognition and performance (Dünser et al., 2008) and have not considered the special contexts of AR environments (Ullmer & Ishii, 2000). Dünser et al. (2008) stated that although AR has been studied for over 40 years, only recently have researchers started to evaluate AR applications. Most of the published AR research has been on enabling technologies or on experimental prototype applications, and user evaluation of AR interfaces is rare (Dünser & Hornecker, 2007). Swan and Gabbard (2005) conducted a literature survey involving a total of 1,104 articles from leading journals and conferences. They identified 266 AR-related publications out of which only 38 (~14%) examined some aspects of human-computer interaction and 21 (~8%) included formal user evaluation. Thus, this research aims to develop a user-based design interface evaluation tool for MAR education. The following sections explore the key design interface elements for the educational usage of MAR. Then, based on identified evaluation criteria, the different stages of design research to develop a usability evaluation tool for a MAR design interface will be outlined.
immersive experiences which blend real and virtual worlds (Klopfer & Sheldon, 2010) where the users' interactions and engagement are augmented (Dunleavy et al., 2009). AR employs the affordances of the real world by providing additional and contextual information that augments learners' experience of reality (Squire & Klopfer, 2007).
For the educational purpose, several researchers recommend employing a broad definition of AR. AR has been created and implemented by varied technologies and has now expanded to various systems, such as desktop computers, handheld devices, head-mounted displays, and so on (Broll et al., 2008;Jeon & Lee, 2011;Johnson, Levine, Smith, & Stone, 2010;Liu, 2009). In addition, current mobile devices, such as mobile phones, digital cameras, and navigators are becoming a powerful platform for AR. The development of sensor (e.g. cameras, location, and orientation) and communication technologies further enables mobile devices to gradually become context-aware (Schmidt, 2002). In other words, AR technologies support learners to engage in authentic exploration in the real world with virtual objects, which can be texts, videos, and pictures, as supplementary elements; thus, learners can conduct investigations of the real-world surroundings (Dede, 2009). In addition, research on education has been defining specific actions for teachers to increase motivation in classrooms (Huitt, 2011;Keller, 1987aKeller, , 1987bRaffini, 1993;Taran, 2005;Wlodkowski, 1978). Several researchers indicate that various educational benefits of AR will make it one of the key emerging technologies for education in the near future (Johnson et al., 2010(Johnson et al., , 2011Martin et al., 2011). AR has been applied to various educational contexts: (1) Experiencing impossible phenomena in the real world (Klopfer & Squire, 2008) such as learning abstract organic chemistry concepts, e.g. molecular forms, the octet rule, bonding (Fjeld, Juchli, & Voegtli, 2003), and chemical reactions (Copolo & Hounshell, 1995); viewing the virtual solar system on the classroom table; visualizing the process of photosynthesis (Liu, Cheok, Mei-Ling, & Theng, 2007) and understanding the spatial concept of Earth-Sun relationships as well as other astronomical concepts by means of a head-mounted display (HMD) (Shelton & Stevens, 2004).
(2) A mobile tour guide AR service system that provides a variety of digital information based on the users' location and direction through an HMD (Höllerer, Feiner, Terauchi, Rashid & Hallaway, 1999) or the camera in a mobile device. It annotates existing spaces with an overlay of location-based information (Johnson et al., 2011).
(3) Learning about the importance of the conservation of fish (Koong Lin, Hsieh, Wang, Sie, & Chang, 2011) by AR book and game playing. This learning activity was designed to facilitate the conceptual understanding of biology.
(5) Combination of mixed realities and remote laboratories is made possible (Andujar, Mejias, & Marquez, 2011). By overlaying virtual elements on remote devices, students can remotely manipulate and interact with the real as well as virtual devices.
The immersion, interaction, and navigation features of these technologies are expected to improve student satisfaction; enhance knowledge comprehension; and facilitate learning tasks that require experimentation, spatial ability, and collaboration (Dalgarno & Lee, 2010;Dunleavy et al., 2009). This means these AR technologies allow ubiquitous learning enhanced by computer simulations, remote laboratories, physical models, and 3D or virtual objects (Broll et al., 2008;Dunleavy et al., 2009). In particular, as the above examples indicate, numerous studies have found that mobile devices play a major role in current education and highlighted the impact and advantages of these devices with regard to the potential for application in pedagogical contexts (Chen, Kao, & Sheu, 2003;Denk, Weber, & Belfin, 2007;FitzGerald et al., 2012;Hwang, Yang, Tsai, & Yang, 2009;Uzunboylu, Cavus, & Ercag, 2009;Zurita & Nussbaum, 2004). Some authors highlight the unique features of AR such as its capacity to promote kinesthetic learning and its support for memory/cognitive processes (Chien, Chen, & Jeng, 2010;Dunleavy et al., 2009). Furthermore, mobile devices enable the mobility necessary for enhancing the authenticity of a learning environment as well as increasing learners' interactions with others (Klopfer & Sheldon, 2010).

Characteristics of MAR education
The pedagogical characteristics of MAR education need to be considered when MAR systems are implemented in classrooms. While static learning materials like paper do not show information dynamically such as motion or continuous movement (Kühl, Scheiter, Gerjets, & Gemballa, 2011), technology can influence the transformation of teaching and learning opportunities to design learning environments that are realistic, authentic, engaging, and extremely fun (Kirkley & Kirkley, 2004 Traditional methods of education involve face-to-face instructions where a teacher plans and delivers the knowledge and learning activities (de Freitas, Rebolledo-Mendez, Liarokapis, Magoulas, & Poulovassilis, 2010). However, the nature of AR education is somewhat different from the teachercentered, delivery-based focus in conventional teaching methods (Kerawalla et al., 2006;Mitchell, 2011;Squire & Jan, 2007). The learning activities associated with AR usually involve participatory simulations and studio-based pedagogy. This learning-by-doing paradigm (Schank, Berman, & Macpherson, 1999) allows students to learn by performing experiments and further reflect on their results. This paradigm explains that trying to do something is the most natural approach of learning how to do it, and this learning-by-doing approach is called as experiential learning (Beard & Wilson, 2006;Kolb, 1984). Dunleavy et al. (2009) state that AR environments provide better opportunities for learning-by-doing through physical movements in rich sensory-spatial contexts. Students get an opportunity to have hands-on experience in their real environments, something that is not possible with the traditional teaching approaches. Lave and Wenger (1991) state that this feature of AR offers situated learning, which takes place in the context where it is going to be applied, i.e. students can seamlessly combine the learning environments and the real world they live in to apply the knowledge and skills learned. Thus, spatial ability, practical skills, conceptual understanding, and inquiry-based activities are the key characteristics of education involving AR environments (Cheng & Tsai, 2013).
However, there are some complications in AR operation in education. Koong Lin et al. (2011) found that the system's procedure is complicated for students. The system can be unstable resulting in system crashes, so the students would require the assistance of technical staff. In addition, the AR content and system need to be more flexible and controllable (Kerawalla et al., 2006). Students should take the advantage of AR to interact with virtual objects instead of only watching educational contents. Therefore, the designers of AR learning environments need to recognize the gap between teachers and students and provide support to help bridge this gap (Kerawalla et al., 2006).
In addition, the experience gained within an AR education environment can be the basis for reflection and lead to further group discussion in a classroom. As students are actively engaged in the learning process, most learners become intrinsically motivated to learn, and this enhances understanding (Yang, 2012). According to Chang, Morreale and Medicherla (2010), students and trainees can enhance their motivation for learning and their educational realism-based practices with virtual and augmented reality. Di Serio et al. (2012) reveal that AR technology has also made a positive impact on the motivation of middle-school students to develop a better understanding in learning contents. Numerous researchers and teachers indicate that motivated students willingly learn more to engage, persist, and expend effort for task completion than unmotivated students (Csikszentmihalyi, 1991;Efklides, Kuhl & Sorrentino, 2001;Keller, 1979;Schmidt, 2007). Lave and Wenger (1991) state that learners can easily share the gained information and experiences with the group in AR environments and directly interact with each other. This means a key effective characteristic of experiential learning is interactivity (Roussou, 2004). Thus, the main advantages of AR application in education, per the learning-by-doing paradigm, are increased activity of students and enhanced motivation to learn and engage with other students.

Evaluation criteria for design interface
According to Squire and Jan (2007), well-designed interfaces or protocols guide students' actions such that they do not face difficulties in interpreting the clues in the devices and the real-world environment. Several researchers propose various AR design principles for learning: AR systems should be flexible enough for the teacher to adapt to the needs of their students and the content should be withdrawn from the curriculum and delivered in periods as short as other lessons (e.g. Billinghurst, Kato, & Poupyrev, 2001;Fishkin, 2004;Fjeld et al., 2003;Hornecker & Buur, 2006). Other researchers propose the following five design principles to be employed in a classroom: integration, awareness, empowerment, flexibility, and minimalism (Cuendet et al., 2013). These principles specifically focus on what makes an AR learning system work in an everyday classroom, not in a lab study.
Of note, there is a dearth of studies on AR design interface from a technical perspective, but some AR usability studies indicate specific areas to be considered when designing the interface. Swan and Gabbard (2005) highlight three areas based on a review of relevant research: (1) Perception: how human perception and cognition operate in AR contexts based on low-level tasks, (2) Performance: how AR technology could impact underlying tasks in user task performance within specific AR applications, and (3) Collaboration: examining generic user interaction and communication between multiple collaborating users. Recently, Ko, Chang, and Ji (2013) pointed out four areas of MAR application that need to be considered: (1) interacting with AR on small mobile display screens (Jones, 2006), (2) simultaneously maintaining a vast amount of information such as graphical information and tangible virtual objects (Nilsson & Johansson, 2006), (3) limited manipulation as one hand would be holding a device and the other hand, manipulating the display (Henrysson, 2007), and (4) manipulating the display while moving toward different locations, requiring location-based applications (Chincholle et al., 2002). All of these considerations imply that too much AR information on a small mobile display may cause usage difficulty for the users.

Presence
AR provides a situation where reality and fantasy are blended; however, this mixed reality could cause confusion among students. Klopfer (2008) states that some students "lose sight of where the game ends and reality begins" (p. 100). Such confusion necessitates the authenticity of an AR system. In this study, the concept of presence was defined as "the degree of a user feeling like being in reality while experiencing MAR." The items for measuring presence were drawn from Barfield's study (Barfield, Baird, & Bjorneseth, 1998). Hartson (2003) refers to affordance as perceptibility, per the findings of McGrenere and Ho (2000), who credited Gibson for introducing this concept in psychology (1979) and Norman for introducing this concept in human-computer interaction (1999). In this study, affordance was defined as "a factor inducing an action for a user to accurately recognize and operate the meaning of an object of interaction in an augmented reality environment." The items for measuring affordance were drawn from Hartson's classification of affordance types (2003), cited by Dünser and Hornecker (2007

Usability
In this study, the concept of usability was redefined as "the degrees of effectiveness, efficiency and satisfaction for a user to complete a task in a MAR environment." The evaluation items of usability were drawn from the findings of Anderson and Shapiro (1989), Nielsen (1993), and Dünser and Hornecker (2007).

Research questions
The ultimate goal of this study is to determine the elements for conducting usability evaluation of mobile media AR interface designs centered on instructional contents and developing an evaluation tool applying them. For the purpose of this research, the following research questions were proposed.
[Research Question 1]. What attributes of presence and affordance affect the usability of mobile media AR? How should they be operationalized? [Research Question 2]. In what ways are presence and affordance of mobile media AR applied to usability evaluation? [Research Question 3]. What is the method of improving the usability of a mobile media-based instructional AR application?

Research method
This study analyzed previous research to develop the usability evaluation elements of instructional AR contents reflecting the media characteristics of mobile devices, examined the usability related to AR, and attempted to operationalize the usability factors including the items of affordance and presence.
Next, along with identifying issues in usability to be considered when developing MAR applications through an expert focus group interview (FGI), the accuracy and classification of the evaluation items collected from existing research were used as basic data for developing the usability evaluation checklist.
Then, a survey was conducted with 122 experts, and after factor analysis and reliability analysis, the final checklist for each usability evaluation item was prepared. Also, a heuristic evaluation of the currently used instructional MAR contents was carried out with seven experts and a usability test was conducted with the general public to determine the problems faced when using the interface. After comparing the results, an improvement plan was proposed to verify the usability and reliability of the usability evaluation tool proposed in this study.

Focus group interview
The subjects of the FGI in this study needed to understand the professional usability of MAR. Thus, the focus group of this study included 7 experts with ≥7 years' experience in interface design or related occupations. In the interview, discussions about the characteristics and problems in the use of instructional MAR applications as compared to the use of existing media were carried out for the first 30 min and the researcher asked additional questions according to their responses. Then, to determine the segmented evaluation items, the researcher grouped them based on the usability elements proposed in existing studies and reviewed the conformity of the organized contents. First, 3 items and 47 usability evaluation questions were presented to the respondents, and after sequentially determining their appropriateness, the usability evaluation questions were determined by eliminating unnecessary items, integrating similar items, modifying phrases, examining category changes between items, and considering the reflection of previously discussed items related to MAR usability issues.

Factor analysis and reliability
A confirmatory factor analysis was carried out for the evaluation items outlined in previous research and the FGI and to determine whether each question fell under the same factor in the three evaluation items as the researcher expected.
This study involved an expert group with knowledge about usability evaluation of AR, which is the focus of this study. Hence, the general public with no knowledge about the research topic was excluded. Among experts in the academia and industry, 122 participants were recruited for the survey using the snowball sampling method.
The survey was conducted from 14 October to 28 October 2014. The survey was carried out using offline questionnaires and online survey. The questionnaire consisted of questions about demographic characteristics and evaluation of affordance, presence, and usability.
The data collected in this study were analyzed using SPSS 18.0 and were subjected to the following methods of analysis.
(1) A validity test on the measuring items presented in this study was carried out using the confirmatory factor analysis method.
(2) An internal reliability analysis of the survey questions was conducted using Cronbach's alpha coefficient.
Regarding the analysis methods, to verify the validity of the evaluation questions and identify the factors constituting each evaluation item, an analysis of principle components was carried out, and for factor rotation, Varimax rotation was conducted. To evaluate the validity of the questions, the acceptance criterion for factor loading was increased to over ±0.6, greater than the general criterion of ±0.4 and the Eigen value was set at over 1. In the reliability analysis of the study, Cronbach's α value of over 0.6 was chosen as the criterion.

Results of analysis-presence
The results of the factor and reliability analysis of the evaluation items for presence are as follows: With the six questions to evaluate presence, two factors were extracted as expected by the researcher. First, considering the characteristics of the first factor questions since all included realism of expression of all visual elements, the factor was named as "visual realism". The next two questions extracted another factor, and since it was an item related to a sensible element other than the visual element and its realism, it was named as "sensible verisimilitude." The eigenvalues of the 2 factors were 2.439 and 1.244, both of which were over the base value of 1.0. Further, in the reliability analysis carried out by selecting the questions related to the factors, the Cronbach's α value was 0.716 for visual realism and 0.603 for sensible verisimilitude, so the reliability was judged as good.

Results of analysis-affordance
The results of the factor and reliability analysis of the evaluation items for affordance are as follows: The questions to evaluate affordance yielded three factors with eigenvalues of over 1 as expected by the researcher. These factors were named as cognitive affordance, sensible affordance, and physical affordance, based on Hartson's results (2003).
The factor consisted of cognitive affordance-related elements, classified by Hartson, e.g. operation by the meaning and practice of an interface object. The eigenvalue was 3.502, which accounted for 38.91% of the total variance, and the factor loading of all items was over 0.6.
The reliability analysis of each factor showed that the Cronbach's α value was 0.752 for cognitive affordance, 0.685 for sensible affordance, and 0.613 for physical affordance, all of which satisfied the base value of 0.6 of this study, so the reliability was high.

Results of analysis-usability
The results of the factor and reliability analysis of the evaluation items for usability are as follows: The items evaluating usability yielded five factors, as expected by the researcher. These factors were ease of operation, ease of learning and memory, usage convenience, satisfaction, and personalization. The extracted five factors consisted of the factors that satisfied the eigenvalue of over 1, the base value.
The factor analyses and reliability analyses of 32 questions for 3 items showed that all the factors were bound by items predicted by the researcher. Only 3 questions did not reach the set factor loading, so a total of 29 questions were selected as usability evaluation questions.
This study attempted to increase the validity and factorial explanatory power of the measurement tool by adjusting the factor loading to 0.6 from the general criterion of 0.4 (Tables 1-3). Finally, 6 questions for 2 factors, 9 questions for 3 factors, and 14 questions for 5 factors were selected for the evaluation of, respectively, presence, affordance, and usability of the MAR application.

Heuristic evaluation of experts
The previously prepared usability evaluation checklist was used for an heuristic evaluation of the actually used instructional mobile application to verify the usability of the usability evaluation tool proposed in this study (see Table 4).  The subject "Beetle of Oak Forest" is an instructional AR application for elementary school students, and it is structured such that the marker of the shell shape of an oak tree is printed and positioned near the camera part so that a beetle's image can be shown in 3D stereography.
For the heuristic evaluation, seven experts in the field of mobile interface design were asked to use the usability evaluation checklist outlined in this study as an evaluation tool for the subject.
The evaluators found that certain areas of the interface design that could be improved. Interviews were conducted with them, and through an analysis and further review of the evaluation results of each question with a five-point scale, the final improvement plan for the interface design of the instructional mobile application was prepared.
After the heuristic evaluation, among all the problems highlighted, those with duplicated meaning were excluded to yield 16 problems, and among these, 4 problems of usability that were pointed out by 5 or more evaluators were classified as high-importance problems, so the improvement plans for these problems are given below:

Problem 1:
The Back button to go back to the Home screen at the upper-left corner is too small and overlaps with the smartphone's call sensitivity display section, so it is very difficult to use that button.

Improvement:
It is necessary to enlarge the Back button and move it to a spot not overlapping with the call sensitivity display section.

Problem 2:
The text on the button part to choose a beetle at the right bottom seems squashed, so it lacks readability. Improvement: It is necessary to change the font of the text on the button for higher readability.

Problem 3:
To move a beetle, the user needs to touch the screen with two fingers, but it is difficult for a child to do that when holding the mobile device in one hand.

Improvement:
The interface should improve such that the user can move the beetle with onefinger touch while holding the device with two hands. For example, if a user touches a beetle for more than 2 s, they should be able to pick it up and move it.
Problem 4: Touching different parts of the beetle's body shows different reactions depending on the parts touched, but sometimes, the touch zone recognition rate is faulty and accurate reactions are not elicited. Improvement: Correct settings of the touch zone are necessary for each part of the beetle.

User test
This chapter sections describes the user test carried out with the users of the actual application to verify the effect and usability of the usability evaluation tool proposed in this study through a comparative analysis of the results of the expert heuristic evaluation explained above. Children, the main users, were selected as the subjects of the test and 7 fourth grade students of elementary school (4 boys, 3 girls) participated in the experiment. The user test included a task scenario, and four tasks were presented including three usability questions of high importance drawn from the heuristic evaluation. The tasks presented to the users are given below: (1) If a beetle comes out on the screen, try to observe it by rotating its body.
(3) Move the beetle here and there on the screen.
(4) Return to the first screen before a beetle comes out.
The user test was carried out sequentially in the same laboratory environment with the seven children and the entire process was video recorded. The medium used was iPhone 5 that a child can hold with one hand.
During the user test, observations were made to check whether any problems occurred during the test. After the test, it was found that two-point touch, Back button operation, and touch zone were the three major issues (see Figures 1-3).
After completing the 4 tasks, they were allowed to use the device freely for 5 min and evaluate it by answering 29 questions on usability evaluation using a five-point Likert scale. This was done to determine any significant difference in the results of usability evaluation tool between the experts and general users.
An independent sample t-test was carried out, and it was found that the results of the evaluation of the presence and affordance items carried out by the experts and the general users were significantly different. For the item of usability, among the 14 questions, the results for 4 questions showed statistically significant differences between the expert group and the general group, and for the items of presence and affordance, the results for many questions showed no difference between the groups.

Summary and conclusion
This study aimed to examine the usability attributes of mobile media AR and to develop a usability evaluation tool via concretization. The objective of this study was to determine the method for applying the usability attributes of AR to interface design rationally, and the results are as follows: Research Problem 1 focuses on the attribute of presence and affordance which affects the usability of mobile media AR and the method of operationalization.
This study examined previous research and defined the concept of presence through an expert FGI as "the user seems to feel real-world for the MAR experience" and the related six measuring questions were drawn. Affordance was defined as the "inducing action factor to help the user can recognize and operate the meaning and the interaction target in the augmented reality environment" and 10 questions for measuring it were outlined. Comprehensive usability in combination with presence and affordance was redefined as the "degree of effectiveness, efficiency and satisfaction for the user to complete the work in the MAR environment" and 16 questions were proposed for measuring it.
Research Problem 2 shows how the "presence and affordance of the mobile media augmented reality can be applied to the usability evaluation." In this study, the factor analysis and reliability analysis of the survey data were conducted via a survey with an expert group for ensuring the validity and reliability of the evaluation questions which were drawn from research problem 1.
For presence, the factor analysis yielded two factors: visual realism and sensible verisimilitude, for which, respectively, four and two questions were loaded.
In the case of affordance, three factors were extracted, based on the result of Hartson: cognitive affordance, sensible affordance, and physical affordance.
For the usability, five factors were extracted: ease of operation, learning and ease of memory, usage convenience, satisfaction, and individuation. Based on the result of the factor analysis and reliability analysis, 6 questions for presence, 9 questions for the affordance, and 14 questions for usability were adopted for the usability evaluation of the instructional MAR application. Research Problem three focuses on the "usability improvement method for the mobile media-based AR application". Using the evaluation questions drawn from research problem 2, a test was conducted to determine the difference or similarity between the experts' heuristic evaluation and the users' evaluation of the instructional mobile application which was used in practice. As the result of the evaluation of experts, it found the important usability problem as the same form from the test for the general user. In the result of the comparative analysis between the groups by the evaluation questions for the general group and experts, there was the significant difference of the result of the evaluation of the general group and experts in the affordance and presence evaluation item. The cause of the difference between the groups was analyzed through the comparison of the average point, and the effect of the usability evaluation tool and usability presented in this study can be examined by the problems and improvement plan. The result of this study can be judged as significant at present as very few studies have looked at the usability evaluation of AR. However, this study has the following limitations: First, this study developed the usability evaluation tool and applied it to the real instructional application for an expert heuristic evaluation and a general user test in parallel way, so the problems faced when using the interface were noted, and the usability, validity, and reliability of the evaluation tool were verified through a comparative analysis of the results of the 2 groups. Second, this study drew the evaluation questions using snowball sampling method the evaluation tool development. Therefore, follow-up studies should look at AR interface design evaluation with regard to the systematic aspect. Various usability evaluation techniques and methodologies can be used for measuring the learning effect of the usability evaluation covering various major issues or developing a usability evaluation tool appropriate for different types of contents other than instructional ones. In addition, generally, in an experiment involving children, the experimental method adopted should be such that the effect of the element of media fascination be as low as possible.