A Framework to Identify Critical Design Parameters for Enhancing User’s Satisfaction in Human-AI Interactions

The advancements in artificial intelligence technology have made changes in how people interact with systems. Unique features and user requirements of Human-AI Interactions (HAII) need to be identified with respect to those of Human-Computer interactions (HCI). This study proposes a way to find critical parameters of interaction design for enhancing user’s satisfaction when people interact with intelligent systems through voice user interfaces. We summarised distinguished user requirements for intelligent products identified from previous researches. Then match them with design parameters in terms of performance indexes that will make differences in the user’s satisfaction. The interaction scenario was set as users ask simple questions with their own voices to the system and the system answer to the questions with synthesized voices after it got to the answer by AI function. The critical performance indexes derived are the number of trials to get the right answer for a question, response time to get to the next interaction, sentence structures of the answer, and pace of the answer. An experimental setup is ready to evaluate user’s satisfaction among different levels of the above performance indexes by Wizard of Oz design method applied on a voice user interface we implement. We are going to validate the effects of performance indexes in HAII on the user’s satisfaction, which will be measured in terms of verbal and non-verbal measures.


Introduction
Through the development of artificial intelligence technology, various intelligent products such as AI speakers (Google's Google Home®, Amazon's Echo®) and virtual assistant systems (Samsung's Bixby, Apple's Siri) on mobile devices are being released [1], resulting in interaction between users and artificial intelligence [2]. Those interactions are called as Advanced Human-Computer Interaction or Human-AI Interaction (HAII). There is a tendency to recognize and understand them as a new version of Human-Computer Interaction (HCI) [3]. But Terry Winograd (2006) argued they differ from the existing Human-Computer Interaction (HCI) [4]. The intelligent products require users to interact with them in new ways. Although HAII has not yet fully established, it is necessary to check whether the user will be satisfactory while interacting with these brand new systems. It is also to check whether there is a need for establishing new criteria to evaluate the goodness of HAII design rather than utilizing traditional way of HCI evaluation method [5]. In this study, a few design parameters that may affect the user's satisfaction are identified by analysing a user's requirements for intelligent products. Finally, we devise and implement an evaluation system for checking out the effect of those design parameters on user's satisfaction while interacting with the intelligent system through the VUI (Voice User Interface).

Method
The overall study flow is shown in Figure 1 below. (1) In order to identify a few critical design parameters of intelligent products, user requirements were gathered by reviewing previous studies. Critical design parameters that can affect the satisfaction of intelligent product are drown out by mapping these user requirements with performance indexes. Commercialized intelligent products such as Samsung's Bixby or Apple's Siri were investigated as the source of variations for the design parameters.
(2) A model HAII system was designed and implemented to conduct experiments for evaluating the effects of identified design parameters. Since it is not easy to control the design parameters of the commercialized products, we took the WoZ (Wizard of Oz) evaluation method. It means that this model HAII system was designed as it can provide different interactions with modified design parameters in random order by researchers in the rear side, but the subjects are feeling and believing that they interact with the real AI system [6].

Definitions for Critical Design Parameters
We collected user requirements in two previous studies related to user requirements for intelligent products. User requirements are shown in Table 1 below. With these chosen user requirements, we analyse the functional aspects of the intelligent systems in terms of design parameters so that we can map the user requirement and the design parameters. Eight user requirements were selected and then mapped with four design parameters as shown in Table 2 below. The four design parameters derived from user requirements are defined as follows: • Response time to get to next interaction -The time it takes for an intelligent product to respond to a user's question. • Number of trials to get the right answer for a question -The number of the trial that an intelligent product challenged to give the right answer. • Pace of the answer -The pace of answers that users can easily recognize.
• Sentence structures of the answer -The amount of information in the answer presented by the intelligent product after the user requests the information. In order to design satisfactory HAII, an evaluation system is to develop for identifying the effect and proper values of the design variables. Before starting the evaluation experiments we have to specify a range and levels for each design parameter considered as independent variables. On the basis of commercial system values and the previous studies, the range of design parameters to be tested are determined as shown in Table 3 below. The response time to get to next interaction is 2 seconds on average for the released intelligent product (Bixby&Siri), and a range of 1 to 5 seconds was conceived as a good control group to test its effect on the user satisfaction. Regarding the number of trials to get the right answer for a question, based on a result from a study that VUI is excusable for up to two errors [9], three levels from first to third trials were considered as a good range for the test. Pace of the answer was set at 6 Syllable/sec, which is the average response speed of Bixby as middle levels and set 4 Syllable/sec and 8 Syllable/sec as slow and fast paces. Lastly, the sentence structure of the answer was set as three different cases: feeds the answer back only, repeat the given question before the answer to make the user sure that the system understood the question correctly, and repeat the given questions before the answer with clear and convincing reference source.

Design for HAII Evaluation System
The HAII evaluation system was designed and implemented so that the system can emulate varying interaction conditions by combining different design parameter values described in the previous section. The system was devised to be good at conducting mixed within-subject design since the time and cost of the testing experiments as well as the fatigue of subjects will be burdensome if all the design parameters are considered as within-subject variables. Response time to get to next interaction and pace of the answer were considered as between-subject variables, and the remaining two parameters (number of trials to get the right answer for a question, sentence structures of the answer) were considered as within-subject variables. By this design, any subject can finish their tasks for only 9 trials. As an experimental session begins, the subject is given a question represented on a computer screen. The subject needs to initiate the interaction by asking the question to the HAII with their own voices. Then the HAII system feeds the answer back depends on the experimental protocol. The overall flow of the interaction scenario is shown in Figure 2 below.   In the given example above, a subject is given a question that "Who is the 16th President of the United States?" The subject asks to the evaluation system through the VUI for searching the answer for the question, and the evaluation system randomly feeds one of the nine scenarios back. The interface of the HAII evaluation system is shown in Figure 4 below.

Discussion
In this study, we derived four critical design parameters of the HAII that can affect user satisfaction during interacting under VUI environment. These design parameters were carefully identified by mapping the selected user requirements on functional performance indexes of HAII. We focused on 8 out of 25 user requirements that were dealt in previous studies. We took the 8 by eliminating those ones that are redundant or does not directly related to the quality of answers from the system. To get more thorough information about how to design satisfactory HAII, it is supposed to collect the user requirements more diversely in further studies including direct surveys on experienced users of the system.
The current version of HAII evaluation system implemented in this study considered the 4 design parameters with specified ranges, but the system is devised as flexible for the changes of parameters as well as their ranges. Since we adapt the Wizard of Oz method, it is easy to add or remove a set of design parameters having a variety of ranges. The evaluation system was developed as an embedded software so that it has another advantage of portability makes it free to apply on a variety of environments using VUI.
To validate the criticality of the four parameters in the HAII design, it is mandatory to proceed with further studies regarding real evaluation of user satisfaction. User satisfaction is a subjective matter and that is used to measure through verbal responses by rating scales or semantic differential method [10]. Non-verbal evaluation protocols utilizing bio-signals such as EEG, heart rate, facial expressions are suggested recently as alternative ways to supplement the verbal protocols [11]. In the next study, we are going to utilize both protocols to find the effects and the proper values of the four parameters derived in this study.