Product Innovation: A Multimodal Interaction Design Method based on HCI and TRIZ

The border between tangible and digital products is getting more diffuse, which has caused the focus of product innovation to migrate from technology to user experience and Human-Computer Interaction (HCI). In HCI, the comprehensive interactions between users and products are collectively referred to as multimodal interaction, and multimodal interaction defines the user experience of the product. However, there is currently a lack of design methods for multimodal interaction. This paper presents a multimodal interaction design method by exploiting the synergy of HCI and TRIZ. This method covers the whole design process, from the analysis of the user needs to the generation of the design solutions and it contains tools as flexible as active, a more valid view on user experience is provided during all design stages. A case experiment validated the effectiveness of the method. The implications of this study will help designers to innovate products with multimodal interaction.


Introduction
These days, ubiquitous computing has emerged as the third wave of computerization [1]. Users can access the Internet via advanced computer interfaces in daily life. Such computer-user interaction methods include visual interaction [2], gesture interaction [3], to voice interaction [4], and motion capture [2]. These interaction methods are collectively referred to as Interaction Modalities (IM) [5][6][7]. Through the new interaction modalities, the input to User Interface (UI) has been allowed to go beyond the traditional inputs such as mouse and keyboard [1,5]. As a result, UIs are more embedded in various products and become multimodal. This has resulted in diffusion between tangible and digital products [8], and the focus of product innovation shifting from technology to user experience and Human-Computer Interaction (HCI) [9].
In HCI, a modality is a single independent channel, communication channel, of sensory input or output between a computer and user [5]. These channels are defined inherently from the human senses: Touch, Visual, Auditory, Olfactory, and Gustatory. HCI types include but are not limited to these senses [6]. The HCI system based on one modality is called "Unimodal", while "Multimodal" refers to a combination of multiple modalities [5][6][7].
According to Collina et al. [1], the user experience of a product is determined by a complex interplay of user interactions. Saktheeswaran et al. [10] have compared the unimodal interaction to the multimodal interaction experimentally. The results showed that multimodal interaction was remarkably preferred than unimodal interaction due to; 1) the combination of complementary modalities, 2) freedom of expression, and 3) integration of the multiple modalities.
Therefore, multimodal interaction acts as a facilitator of user experience of a product through the integration of multiple modalities [1].
Although the rapid development of HCI promotes product innovation to pay more attention to user experience [9], the research of multimodal interaction is overlooked. In the studies of user experience for identifying requirements for product innovations, it is essential to consider which types of multimodal interaction are relevant and how they are related to the user experience [1]. Currently, there is no complete design method to capture the multimodal interaction in product innovation. However, design method is required since it will help the study for user experience in the concept generation phase, which ultimately will optimize product innovation. Also, the recent rapid advance of technology leads to many challenges that HCI facing [2]. Therefore, an efficient and unique new method is required to rapidly develop designs, evaluations, and interaction strategies for fast-changing interface and interaction systems [11].
For product innovation, the Theory of Inventive Problem Solving (TRIZ) is one of the many powerful problem-solving methods [12]. However, since TRIZ is only focusing on technological issues and functionalities, it is not enough to complete the innovation of current products. Even if a product is well designed, it can be ignored by users if it does not consider users' needs [9].
To overcome this limitation, the potential combination of TRIZ and HCI is investigated to establish a multimodal interaction design method for product innovation, which are aimed at that can help designers and researchers implement more holistic tools to enhance the user experience to solve the challenges of product innovation when designing multimodal interactions of products. In the design method, the product requirements and user needs are analysed. Then, the solution is generated based on the information, containing tools as flexible as active. During all design stages, a more valid view is provided on user experience.

Multimodal interaction concepts and methods
The multimodal system is defined as a system that allows users to input multiple modalities, such as touch, gesture, and voice, to provide an efficient and flexible UI environment. The multimodal system, for example, can output synthesized speech, intelligent graphics, and others [7]. Because most interaction modalities can be used as input channels and output channels, in this paper, each interaction modalities and its specific interaction methods briefly introduced according to the classification of sensory communication proposed by Karray et al. [5].
2.1.1. IM 1: Touch. It bidirectionally senses a physical interaction with the human body, supporting both action and perception of the environment [6]. They are classified into single-touches and multi-touches. The pinch-to-zoom gesture is a typical single-touch application. In the application, two fingers are used to zoom in and out via moving two fingers farther apart or closer to each other. In contrast, in a multitouch system, the presence of more than one touch-point is recognized at the same time [7].

IM 2:
Visual. It is one of the most actively researched areas in the HCI field [5]. In the methods, visual inputs, e.g., texts, graphics, and animation, are used in interacting with electronic devices [13]. In addition, more precise gaze detection of users is achieved with sophisticated techs, such as eye-tracking, so that the gaze patch is used in various ways to interact with the interface [14].

IM 3: Auditory.
It allows users to interact bidirectionally with the HCI system by voice to complete tasks like face-to-face communication between people [4], supporting both action and  [6]. Voice interaction has been adopted in daily life such as: home appliances, computer operating systems, home automation, and automobiles [7].

IM 4: Body.
In this modality, the integrated sources are used to generate feedback to the sources and interact in a digital environment. Multiple human sources are used as inputs from physical, physiological, cognitive, and emotional sources [7]. Its specific interaction methods include body posture, facial expression, and gestures. First, body posture (motion capture) exploits users' body position, orientation and movements to present computer vision so that the HCI system can determine the user's intention. [13]. Second, a camera is used to recognize the facial expressions of the user for the system to react to the user's emotions. Based on the extracted expression, a character on the screen interacts well with the user with the proper emotion of faces [13]. Finally, hand gestures can be used as a touchless input to conduct some tasks, controlling the devices [3].

IM 5:
Olfactory. The olfactory interaction modality-based HCI systems provide information to users via smells [6]. More precisely, a device for releasing odor is integrated into the HCI system to provide humans with olfactory stimulation so that the user's sense of smell or electroencephalogram caused by the olfactory stimulation [7]. The olfactory interaction modality has both input and output. The input is called an odor sensing system, whereas the output is called an olfactory display [6,7].
2.1.6. IM 6: Gustatory. Even though the taste is an essential sense, it is very challenging to be implemented in traditional unobtrusive devices [5]. Its theoretical concept is that HCI systems provide users with information through taste [6]. However, no interaction methods were developed to display taste [7].

The concepts and tools of HCI
Over the decades, methods and tools have been actively studied to develop efficient and effective interfaces between humans and computers [2]. The researches include assessing the usability of HCI systems, human-centric issues, and how people interact with devices [14]. The HCI field contains a series of different types of tools. In this paper, only some tools related to this study are introduced. The extraction of these tools is aimed to help establish and constrain the multimodal interaction design method.

HCI 1: Laws of Interaction Design.
It includes Fitts's Law and Hick's Law. Fitts's Law was primarily used to model the human movement in HCI and ergonomics. The predictive model is used to model the act of pointing in both physical touching and virtual pointing with an additional device. Hick's Law describes the time required to make a decision and is used to justify the decision for UI design [9,14,15].

HCI 2:
Nielsen's Heuristics. These heuristics consist of ten principles for the UI design. Nielsen originally developed and refined his heuristics principles by analyzing 249 usability problems. The ten heuristics include the visibility of the system status, the match of the system to the practical world, the consistency and the standards, the freedom of user control, and the aesthetic and minimalist design. [16].

HCI 3: Usability.
The high usability is essential for most products. Many evaluation schemes for usability have been developed and are classified into three groups: usability testing, usability inquiry, and usability inspection [2,14,15]. Usability testing employs representative users using a system or a prototype to perform typical tasks and then evaluates the cognitive compatibility of the user interfaces. Usability inquiry evaluates the system by users' feelings when they are using the system. The users are monitored and requested to answer the prepared questions. Finally, in usability inspection, the experts examine the usability of the interfaces analytically [15].

HCI 4: Interaction Paradigms.
This concept refers to a particular philosophy for HCI [2]. Designers are oriented to the questions they should ask themselves during their activities, to guide the future design and development of the interaction system. For example, these paradigms are ubiquitous computing, seamless integration of techs, augmented reality, wearable computing, physical/virtual integration, etc. [14,15].

The concepts and tools of TRIZ
The systematic approach, TRIZ, was proposed by Altshuller [17] to guide people and avoid random exploration in the problem-solving process. TRIZ guides the solvers to find an innovative solution by restricting searching space based on the patterns that were found in the past and applicable to both in the same or completely different domains [18]. In this paper, only TRIZ tools related to this research are extracted.

TRIZ IFR: Ideal Final
Result. It refers to the best solution without considering other constraints, such as time, space, and cost. In TRIZ, ideality is used as an objective, and the system evolves for increasing the ideality. The ideality is measured as the ratio between the harmful and negative functions and the useful and positive functions. IFR is the ideal system that has only benefits but no bad effects, which could guide toward seldom-explored directions [12,18,19].

TRIZ FAM: Functional Analysis
Model. FAM decomposes the system and represents the functional relationships of the components. The generated map depicts the delivered function composed of small units, created by a subject/action/object triad [12]. The subject is the function provider, while the object is the receiver. Different values can be assumed in the interactions between the subject and object according to the modified object parameter. For example, useful/sufficient or useful/harmful can be considered. The parameter is changed through the action determined by the subject [12,17,20].

TRIZ Contradiction.
In TRIZ, an inventive solution should overcome contradictions that are often contained in the complex problem when two parameters are conflict with each other. At least one totalcontradiction or partial contradiction usually arises in practical problems. To address the contradictions is essential to achieve the goal of the system [17][18][19].

TRIZ 40-IP: 40 Inventive Principles.
Altshuller lists up the 40-IP through the analysis of a massive number of patents. These principles are particularly important in resolving contradictions. Practitioners of TRIZ utilize these principles to develop useful concepts of solution for inventive situations. Each solution recommends a specific modification of the system to address the contradictions [12,[17][18][19].

Synergism tools
The following inclusion and exclusion criteria were used to search for possible synergism tools in the HCI and TRIZ fields. The criteria were organized into 3 inclusion criteria (IC) and 3 exclusion criteria (EC), and the inclusion criteria are as follows: • Independent and complete tools.
• An active role and clear goals.
The exclusion criteria are the opposite to the IC. The tools that do not meet the IC will be excluded. Table 1 shows the tools highlighted in this investigation and numbered them according to related fields. In order to help researchers and designers focus on multimodal interaction and implement a more holistic approach in all stages of the product innovation process, the tools included in this method must be active and flexible, and related to the entire design process. Therefore, the definitions of the three main design phases presented by Michailidou et al. [21] are used to classify these tools ( Table 2).  Table 2. Classification of tools that could be used for the synergism. Grouped by the three main design phases proposed by Michailidou et al. [21]: the "Analysis", "Creation" and "Evaluation". • Analysis Category. This set contains the Interaction Paradigms (HCI 4), the TRIZ FAM and the TRIZ Contradictions. They cover the analysis of product expectations, product functions and user needs. They can generate a new requirements list, which contains information from products to users. • Creation Category. Here there are Nielsen's Heuristics (HCI 2), the TRIZ IFR and the TRIZ 40-IP. The design strategies can be generally used to find the best solutions. Also, these strategies are applicable in different contexts and types of problems. • Evaluation Category. This set contains the laws of interaction design (HCI 1) and usability (HCI 3). The combination of these two tools can effectively expand the area of the evaluation, which simultaneously considers the resources, time, costs, the accuracy, the objectiveness, and the development stage. [15].

Design method for multimodal interaction
The classification of tools is to allow designers and researchers to selectively extract tools for synergy to establish a targeted design method when designing multimodal interaction of products. Figure 1 demonstrates the generation of this result, where the boundary of the interaction domain is described by the larger circle. Its upper part is the Analysis area, and the lower part is the Creation and Evaluation area respectively, each area contains the above tools. The smaller circle contains all the interaction modalities and the interaction method in each modality, it is included in the Analysis area.  Figure 1. Representation of the relationships among the three categories.
In HCI, the product is a medium to get precise functions that are used for problem-solving and meet the needs of the users [9]. Therefore, the analysis of the functions is the fundamental point for the design method. In the beginning phase, functions are separately considered from products so that the aspects of interaction related to cognitive compatibility can be focused instead of on technological and constructive details. Thus, the requirements and expectations of users are referred to functions rather than product. Then, functions are analysed to identify the key components. Functions are converted into products in the next phases. Figure 2 shows an overview of this method, namely the multimodal interaction design method for product innovation. It retains the classic design process from analysis, creation to evaluation. This method allows a synergy between the TRIZ tools and the HCI tools, mainly in the idea and concept generation phase. An important part of the method is the introduction of the multimodal interaction concept, which guides the definition of product functional components to tend to throughout the entire design activity, where there are some new phases: in the Function Identification, the designer or researcher will define the multimodal interaction in the product according to the Requirement List and Interaction Paradigms (HCI 4), which can be a random combination of five interaction modalities and interaction methods in each interaction modality while in the Function Analysis, the definition of multimodal interaction guides the determination of the components related to the interaction modalities in the product. The TRIZ FAM will analyse the mutual functions between the components, and the TRIZ Contradiction will constitute a general description of the problems arising between the components. These two stages are included in the Analysis phase. Moreover, this method provides the TRIZ IFR, TRIZ 40-IP and Nielsen's Heuristics (HCI 2) to guide the generation of design solutions in the Creation phase while the Usability (HCI 3) and Laws of Interaction Design (HCI 1) for the Evaluation phase.
Importantly note that it provides the tools of HCI and TRIZ for designers and researchers but is not limited to these tools. The process of this method is open-ended, which allows designers and researchers to choose or add different kinds of tools. For example, the Analytic Hierarchy Process (AHP) [22] can be used to determine the level of needs when analysing user needs. Moreover, the tools provided are not limited to the phase they belong to. For example, Usability can be used both in the Evaluation phase and in the Creation phase. This method is aimed to help researchers and designers implement more comprehensive approaches when designing multimodal interactions of products.  Figure 2. The process of the multimodal interaction design method.

Results
A social application design project was conducted to validate the effectiveness of the multimodal interaction design approach. The multimodal interaction design method and the other compared design method are respectively implemented by four designers (two groups). Four metrics are used to measure the methods qualitatively and quantitatively, which includes: • Group 1 (G1)-group using other design methods, • Group 2 (G2)-group using multimodal interaction design method.  This experiment evaluates the contribution of the multimodal interaction design method for the design of a social application. The initial activities of the design are shared by other design methods and the multimodal interaction design method. Twenty users filled out a questionnaire about the needs of social applications and the analysis of their answers allows highlighting a requirement list that includes 28 user needs.
Now the procedures diverge. The tools used by G1 in this project are User-Centered Design (UCD) [15] and Participatory Design [15]. A total of 32 solution concepts and 7 evaluation methods were generated by them. After corrections are applied, 28 final solution concepts are generated. Participatory Design is used to validate the solution concepts. After some problems arise again, are corrected, 23 solution concepts are available for designers. It is done in two weeks.
In the G2, the number of resulting solution concepts is equal to 56, among them, Nielsen's Heuristics and 40-IP generated 47 solution concepts. The multimethod proposed by the Laws of Interaction Design and the Usability contains 11 evaluation methods. The 7 revisions constitute the outcome of this activity. After this reviewing process, 40 solution concepts are available to be exploited. It is done in one week. Table 3 summarizes the comparison.

Discussion
The experimental results validate that the multimodal interaction design method provides solution concepts more than the compared method. It also shows the advantages of tools synergy. The multimodal interaction design method takes a shorter time to process it, less or equal than a half. Moreover, the better quality of the solution concepts is verified by fewer criticisms during the reviewing phase. In addition, in this activity, the multimodal interaction design method group used TRIZ IFR to assist in defining the functions of the product, where the flexibility of tools shows that they are not linked anymore to the field they belong to, but to the moment they are needed and exploited during the design process. This indicates the proposed method has better generalization ability. Thus, the proposed multimodal interaction design method can be applied in wider applications and situations.

Conclusions
This paper proposes the multimodal interaction method to help researchers and designers implement more holistic tools in all stages of the product innovation process. It is based on the advantage of the synergy of HCI and TRIZ. This synergism started by highlighting the tools that could allow the synergy between HCI and TRIZ. These tools are allowed to be classified into three categories: analysis, creation, and evaluation to establish the process of the multimodal interaction design method. Many pieces of information have been merged into the design process and this implied a strong reduction of redundancy. A social application project validates the effectiveness and the added value of the multimodal interaction design method. The number of solution concepts has increased and the multimodal interaction design method requires fewer resources to be applied, which will be a novel contribution to the multimodal interaction and product innovation.