Research on the application of mixed reality technology in the field of art design

. Aim: summarize the application of mixed reality technology in the field of art design and analyse its shortcomings and trends. Method: by searching the Web of science as the main source of information to explain the basic concept and development of MR and state of art in product design, display design and interactive design. Discuss the pros and cons of mixed reality technology in human-computer interaction, hardware, and user experience. Result: Mixed reality technology is widely used in art, design, especially in product design, display design, and interactive design. The future trend of mixed reality technology is summarized the combining status of market and industry. Conclusion: The application of mixed reality technology in the field of art design is already drawing more and more attention from industry, but it still lacks enough attention from academia community. This paper systematically introduces the state-of-the-art applications and trends of MR in the field of art design which helps readers to get an insight on this fascinating research area.


Introduction
The development of mixed reality technology offers new ideas for solving problems in various industrial situations.So far, mixed reality technology has been widely used in education, industry, maintenance, medical, etc., but the study of mixed reality technology in the field of art design is still in its infancy.This study summarized the application direction of mixed reality technology in the field of art design through the collection and sorting of literature, summarizing the deficiencies of current research and future trends of mixed reality technology in the field of art design.

Mixed reality definition
Mixed reality (MR) is a technology that mixes the real world with the virtual world and enables natural and intuitive interactions between people, machines, and environments [1].With the continuous improvement and development of technology, the field of mixed reality technology is constantly expanding.Paul Milgram [2] defined Mixed Reality (MR) as a collection of augmented reality (AR) and augmented reality (AV) where virtual objects are presented in superposition with real space.Though AR has been considered as a kind of MR for a long time, the difference between AR and MR has been further clarified as people dig into this research area.AR refers to the superposition of virtual objects in physical space, where the placement of virtual objects in real space is not affected by spatial perspective and distance while MR refers to presenting virtual objects based on physical spatial data rather than simply superposition, and virtual objects in mixed reality can change according to changes in space [3][4].In 2002, Piekarski and Thomas developed AR Quake which is one of the first outdoor mobile AR applications and demonstrated at an international symposium on wearables.[5] In 2013, Google produced a new Google Glass with AR features, which connected to users' phones based on user feedback.[6] In 2016, Microsoft launched a headset called HoloLens, which quickly became popular worldwide [7] and has a wider range of interactive gestures compared to its former generation which was updated in 2019 [8].(Figure 2) The development of MR involves multiple research areas such as computer geometry, computer network, imaging processing, 3D modeling and rendering, speech recognition and motion recognition, etc.The advancement of all these research areas will benefit next generation MR devices which typically include space registration, hardware, facilities, space building, human-computer interaction.

Space registration
The coordinates of the virtual world and the real world corresponds, so that the virtual object and the real time form a correct spatial perspective relationship.

Hardware facilities
Mixed reality systems primarily involve hardware devices such as displays, computers, tracking cameras, and input devices.

Space building
Broadly speaking, a mixed reality space is the state of virtual objects in real space, the rules that objects follow, the relationships and interactions between them.

Human-computer interaction
Interaction between the user and the virtual object.

Applications of mixed reality technology in art design neighbourhoods
Mixed reality technology is widely used in maintenance, education, medicine and art design, etc.In the field of art, design, the use of mixed reality technology mainly focuses on three aspects: product design, interactive design, and display design.In product design, MR are applied to help designers validate the product quickly by building hybrid prototypes and improve the user's participation in the design process through virtual collaboration.In interactive design, MR are applied to improve the design's interaction efficiency and improve the user's interactive experience by studying the forms of interaction.In display design, MR are applied to build virtual objects in real space, relying on the unique space registration characteristics of mixed reality technology.In the next sections, detailed application of MR in these three fields will be discussed in detail.

Product design
Most times, in the demand acquisition stage of the product design, the differences in the designers' educational background and users' demands lead to different perceptions of the same need which affects the final design result.To solve this problem, Arrighi et.al.
introduced a new modular digital tool kit that allowed users to become active participants in the design process [9].The tool kit consisted of a virtual interface, a physical interface, and a physical environment, these three elements formed a mixed reality collaboration tool that combined hardware and software.The hardware of the system was made of modular tangible user interface (TUI) where software-level interactive content was able to be displayed in mixed reality.The tool kit allowed users to invoke different TUI and 3D interactive content according to their own needs so that the friendliness and visibility of interaction was improved.Another serious problems in that the designers' proficiency of using different tools for prototype designing seriously limits designers' creativity.[10] To make design tools more feasible, Maurya et.al. created an interactive product prototyping tool based on MR which enabled designers to generate interactive prototypes which realized similar functions of real products.Freitas [11] et. al. illustrated a set of tools for rapid development of virtual prototypes which was used for designing HMD devices and AR applications for smartphones.
Similarly, an interactive prototype called PitAR has been developed by Gasques and coworkers [12] which allowed designers to present their ideas in a visual way.In these two works, the application virtual prototype gradually changed the rules of the game.Compared to traditional methods of verifying product such as 3D printing and oil mud modeling, building virtual prototypes through mixed reality can quickly validate design results and save money on product development.Although the physical prototype is closer to the final product plan, physical prototyping is expensive and takes a lot of time and effort to modify the prototype in time.Therefore, combining MR prototyping and traditional physical prototyping offers better results.[13][14][15].By interviewing experts, Gomes et.al. [16] constructed the evaluative dimensions of mixed reality prototypes to evaluate the availability and user experience of MR prototypes.At the same time, a comparative usability experiment showed a clear positive correlation between the tasks performed by users on MR prototypes and real devices.In other words, the completion of tasks highly relies on the complexity of the task.Based on the research above, some experts appeal that the MR prototypes can be used for evaluating product performance, human-computer relations, and product performance.An innovative virtual prototype has been built by Morozova and co-workers which showed the great possibility of using the MR prototype in complex task [17] in verifying the availability of digital products.This prototype aims to facilitate the study of interactive interfaces and the testing of complex prototypes.
Physical prototypes have obvious advantages in verifying the effectiveness of prototypes and perception of models, while MR prototypes have flexible and varied characteristics.Therefore, MR technology is widely used in different stages in product design not only to eliminate cognitive friction, increase user engagement, and validate the product design phase, but also shortens the product's R&D cycle, reduces R&D costs and provides more flexible solutions [18].

Display design
A main venue for cultural transmission is through going to the museum.However, in recent years, due to the pandemic, people are less motivated to visit museums offline.Considering this, various MR techniques are used to provide immersive museum experience with virtual interactions at home.By utilizing various sensors Apostolakis et.al. [19] developed an interactive museum exhibiting the field of natural history.In the MR Museum, the scenery of immersive Crete was exhibited in front of the user so that one could understand the history Crete better through intuitive natural interaction.Similarly, aiming to protect ancient Greek culture, Plecher et.al. [20] created a museum of ancient Greek culture based on MR technology.Combination of the virtual and the real surroundings enabled an ancient Greek museum to be built by simply scanning the surroundings through MR devices.By 'reconstructing' ancient Greek scenes, users felt as if they were in the real historical environment which helped visitors deepen their understanding of Greek culture.Pietroszek and co-workers [21][22] designed the Peace Corps Experience Museum through the Ad-Hoc MR Toolset.Users' browsing experience was obviously improved by employing virtual physical environments and gesture interactions.Interestingly, Fistola et.al. [23] suggested a new way to construct a mixed reality museum to enhance user engagement and immersion.By exhibiting the discovery site, a virtual historical site was reproduced.After all these processes, the user became a character in history through storytelling.At the same time, attentions from academic community also tried to understand the recognition mechanism behind MR.Oliva et.al. [24] rebuilt the deworming building of the Bergen Belsen concentration camp through MR technology.Users could learn about the history of the building from the virtual guide.Results showed that prior knowledge affects users' memory in MR environments.From a user experience point of view, the research found that narration, interaction, and collaboration elements affected the user's experiencing process, while immersion affected the user's impression of content during the tour [25][26].
From the examples above, one can easily imply the importance of the application of MR in the digital preservation of historical heritages.Digital display of cultural heritage through MR technology is feasible and facilitate the spread of culture.The immersive MR features enable users to feel as if they are in real historical scenes so that the understanding of history and culture are deepened.At the same time, the combination of MR and physical models of historical relics reproduces historical glory in physical space and the natural interaction between visitors with MR makes the whole exhibition more attracting.

Interactive design
Studies of the interaction between users and virtual information often involve tangible user interface (TUI), MR, and human-computer interaction.As in the studies of MR, the main aim is to provide users with intuitive, efficient, and natural interactive experiences.Currently, studies of MR technology can be divided into four categories: tangible interaction, collaborative interaction, unconscious and hybrid interaction [27][28].

Tangible interaction
TUI is an interactive way to manipulate digital information directly using physical objects [29].TUI presents two different forms of interaction by combining physical and digital manifestations: time reuse and spatial reuse [30][31].More specifically, time reuse includes input devices such as controllers and buttons that users can pick and operate virtual objects by grabbing physical devices.Spatial reuse refers to allocating inputs and outputs on space so that different virtual objects have different dedicated physical interfaces, such as gestures and gazes.By comparing the most commonly used space reuse interactions, Kang et.al. [32] Suggested users were more inclined to touch virtual objects directly than interact with them through objects indirectly, such as gaze, as direct touch was consistent with the way they interact in physical space.However, spatial reuse could be blocked easily by obstacles, which led to more interaction errors.Meanwhile, spatial reuse had limited gestures that could be recognized for complex interaction operations.Therefore, a more favorable way of interactions should be found to support MR operations.Hassoumi et.al. [33] found that eye movements provided interaction with high accuracy in mixed reality environments.Therefore, a new method, eye tracking technology were introduced in MR to help people with ALS or other motor disorders when interacting with computing devices.Generally speaking, space reuse is more effective than time reuse.However, it is believed that time reuse is significantly superior in interaction efficiency and accuracy, while time reuse is heavily dependent on hardware devices.Therefore, time reuse and space reuse are often combined to improve the user experience [34][35].

Collaborative interaction
Collaborative interaction interface refers to the use of multiple devices to interact remotely or face to face [36].Collaborations are facilitated by projecting interactive scenes into real spaces or wearing HMD devices when users are in the same real space, or remotely in the same virtual space.So far, there are very few research in the field of collaborative interaction.

Unconscious interaction
The way the virtual environment interacts through the sensors is defined as unconscious interaction.Unlike TUI, device-based interactions are nonsensical.Users' needs are perceived without any physical contact, such as gaze, eye tracking, and motion following.Compared to other interaction methods, unconscious interaction shows great potential in real time interacting without additional conscious motion.Wang et.al. [37] established an MR table tennis system, where locators were used as primary input devices.Channels for data exchange were established for devices, enabling the system to update the racket movement in real time.The thickness of the collider was dynamically adjusted according to the detector, so that the bounce result could be calculated according to the racket and the ball's motion.Experimental results showed that this method not only avoids collision detection loss, but also improved the authenticity of the simulation and maintains a good interactive experience.

Hybrid interaction
Among all MR interaction methods, hybrid interaction is the most used one.In hybrid interaction, specific needs are met by adopting different modes of interaction which maintains the diversity of interacting and makes interactions more user-friendly and attracting [38].For example, Zhang et.al. [39] constructed mixed reality systems combining gesture-based interaction (GBI) and physical-based interaction (PBI) to assess the pros and cons of different interaction patterns.Results showed that when compared to GBI, PBI alone had a significant increase in users' productivity through the T-test and the one-factor variance analysis in the optical headset of mixed reality applications.To further enhance the user experience, they presented experiments that combined both PBI and GBI interactions, proving that hybrid interaction could help increase the efficiency of complex tasks.Similarly, Song et.al. [40] suggested a narrative system based on MR technology that used artifacts as tangible interactive tools.Through this system, users could learn how to make handicrafts and then combined them with HoloLens2 for creating stories and telling stories.To make virtual things touchable with feedback on the HoloLens gesture interaction, the system used hand tracking to achieve touch-like effects on the desktop with Kinect.Research showed that the system was feasible and was becoming more and more popular among users.In addition, hybrid interactive methods can not only improve the fun of experience, but also increase the interaction between users.

Problems and prospects of mixed reality technology in art design
Mixed reality technology has unique features of immersion, virtual combination, natural interaction, and spatial registration which other techniques could barely provide.Firstly, the immersive environment that MR provide is an important component of user experience.The degree of immersion affects the depth and interest of the user experience.Secondly, the superposition of virtual and physical objects in real space creates unique illusional feelings of mixed reality.Thirdly, MR provide a brand-new interactive mode in which users do not need to interact with physical objects which can lead to more efficient and senseless interaction through various sensors.Lastly, mixed reality technology's unique space registration features that enables placing virtual reality based on the perspective of physical space, avoiding the degradation of reality.However, the application of mixed reality technology in the field of art design still has some serious problems.Firstly, hardware limitations are one of the main problems MR suffers, such as the tracking of sensors and the visual angle of HMD devices.All these leads to degradation of the quality of MR.Secondly, interaction design still uses the WINP (window, icon, menu, pointer) interface paradigm which applies to the mouse as the main input hardware.This old-fashioned interface paradigm not only increases the difficulty of interaction, but also reduces users' motivation.Last but not least, mixed reality technology is still in its early stage.Theories related to MR are not complete, therefore, the experience users obtained are limited.At the same time, most mixed reality applications still suffer from lack enough user immersion, redundant operation process and difficulty in understanding the operation methods.In the following sections, the trend of applying MR in different areas of the art design is summarized.

Visual design
Because of the hardware constraints and traditional interaction method, predominant MR systems lack interactive efficiency, which can be improved by developing design standards.When loading large files or documents, it often takes a long time.The delay in response often irritates users.Therefore, presenting the waiting time with corresponding jumping animation and visualization can soothe the anxiety of users.Secondly, the model grid can be adjusted when switching the model.For example, by adding extra model grids, details of the models can be improved, when necessary, on the contrary, when using the model as the background, the model grids can be minimized to decrease the processing burden of the device.Thirdly, interface interaction can be relieved by developing interaction specifications.Since MR devices' displays are cumulative, a bright-colored hologram that absorbs more light not only affects reality, but also increases the user's visual fatigue.At the same time, bright colors make it difficult for users to see their hands resulting in difficulties in interface operation.Therefore, when developing interaction specifications, dark translucent color can be used to minimize eyeglass fatigue and increase the accuracy of user operation.

Collaborative designs
Collaborative design not only enables different devices to incorporate into the same design process, but also integrates different stakeholders, making designing more efficient.At the same time, from the users' point of view, incorporation of different sensors also provides a richer interactive experience.In terms of device collaboration, MR has a rich expansion SDK (Software Development Kit) that allows different devices to work together enabling various devices maximize their functions.The involvement of different stakeholders can not only improve communication, but also help reduce cognitive friction.Various sensors such as motion tracking, eye tracking, gesture tracking and temperature tracking can enrich the way MR systems interact and increase user interest.Therefore, when designing a collaboration system, the properties of the device and the needs of users should be fully considered.

Design review
When inspecting the final design product, a through test and review should be carried out.The traditional method is to use physical prototype.But in the context of the epidemic, it is almost impossible for experts worldwide to gather and test the final results jointly.This is where MR prototype comes into play, by using MR technique, the final product can be built in a physical space through spatial registration, allowing experts to inspect the final design product in an MR way.

Design education
Art design is a practical subject that encourages students to actively participate in design practices, which not only improves their understanding of professional knowledge, but also helps to improve their abilities to design new products.However, due to the limited infrastructures, there are many college courses remain literal which demotivate students to practice.Considering this, MR may be a solution.MR has the features of combining the virtual and reality, which enables students majoring in architecture and product present their design practices directly into physical space and verify them.Meanwhile, MR can also offer students majoring in fine art a possibility for their artistic creation.In a word, MR can not only provide students with a way to visualize their creation, but also inspire their passion for art.

Summary
After years of development, though MR technology is constantly expanding, MR technology is still in its early stage in the art design field.This article discussed the application of mixed reality technology, its basic components and trend in the design field.Although mixed reality technology still has many shortcomings in terms of hardware and interaction method, we can still expect it has a bright future in designing, and presumably be one of the most helpful tools for designers in the future.

Fig. 6 .
Fig. 6.Mixed interaction.(https://www.pexels.com) The paper was funded by Education Reform Project of Jiangsu University (Discussion and Practice of Continuous Teaching Mode Combining Virtual and Real, Multi-dimensional Dynamic, No. 2017JGYB034), Key R&D Program of Zhenjiang City (Research and Development of Personalized Customization System for Modified Vehicles Based on Mixed Reality Technology, No. GY2021014), the Project of Industrial Design Research Center of Humanities and Social Sciences Research Base of Education Department of Sichuan Province (Research on Customized System of Refitted Vehicle for End Users based on MR), Degree and Graduate Education Research Project of Chinese Society of Chinese Degree and Graduate Education (No. 2020ZDB75).

Table 1 .
Composition of MR.