Study About Creation of “Maai” Involving Intention Using Rhythm Controller

Abstract

In this paper, we developed a “Maai” creating agent that predicts human’s intention from the changes in his/her controller manipulation method (i.e., relationship between controller input and avatar motion) of a rhythm controller. And, we examined whether the human’s intention is reflected in the controller manipulation method in the creation of Maai involving the human’s intention. From results, we showed that the human’s intention to create Maai with the opponent leads to the potential for changes in the human’s controller manipulation method. Furthermore, we showed that such an intention of a human as to whether he/she intends to maintain or collapse Maai is expressed in his/her controller manipulation method. Consequently, our research shows that an unconscious process plays a role in the creation of Maai involving the human’s intention.

1 Introduction

A human, within a complex environment, is capable of performing spontaneous actions that suit the situation and maintains an appropriate distance (“Maai”) from another person nearby. The Japanese concept of Maai is closely related to interpersonal distance, which is determined by individual feelings. Anthropologist E.T. Hall’s proxemics [1] suggests that such distances are produced in response to social and physiological relationships, surroundings, and so on. To study such Maai, it is insufficient to consider action as only externally expressed motion. We must also consider the mental and embodied functions that create that action. However, how the mental and embodied functions work is generally very difficult to observe from the outside.

By including both mental and embodied functions in our input interface, we believe that we have solved this problem of Maai when operating avatars in virtual space. Furthermore, our past research has included the development of a “rhythm controller” [2–4] that has the following two characteristics.

• 1. The rhythmic operation of the rhythm controller precedes the avatar motion.

• 2. There is no one-to-one mapping between the controller input and avatar movement.

Characteristic 1 enables handling of an avatar as if it was a part of one’s own body. As a result, it becomes possible for an operator of the controller to create Maai with the other person on the display screen through an embodied interaction. Because of characteristic 2, we believe that the mental and embodied functions are reflected in the controller manipulation method (i.e., relationship between controller input and avatar motion). With this premise, we examine the controller manipulation method when two human test subjects improvizationally create Maai with their opponents using the rhythm controller. Our research [5–7] revealed the following:

• When the constant Maai is maintained, a human test subject did not change his/her controller manipulation method, and this method was same and consistent across two human test subjects.

• In the “Kendo” (Japanese fencing) match through avatars, the controller manipulation method when Maai is maintained differs from when Maai is collapsed.

The feature of Maai in the Kendo match is that humans intentionally collapse Maai and create new Maai, whereas Maai discussed in proxemics is desired to be maintained [8]. These results indicate that the intention of a human as to whether he/she intends to maintain or collapse Maai may be reflected in his/her controller manipulation method.

In this paper, as the first step towards addressing the issue described above, we ex-amine whether a human’s intention is reflected in the controller manipulation method in the creation of Maai involving the human’s intention. Specifically, we conduct experiments where a human test subject (i.e., human operator) and an agent that controls its avatar in such a way as to maintain a constant distant from a human avatar create Maai, and examine the difference of the human’s controller manipulation method depending on whether the human has an intention to create Maai. Additionally, if the intention of a human as to whether he/she intends to maintain or collapse Maai is reflected in his/her controller manipulation method, we believe that the agent that predicts intentions of a human operator from the changes in his/her controller manipulation method will beat him/her in the Kendo match. Therefore, we develop a Maai creating agent for achieving this, and conduct experiments where this agent and a human operator play kendo matches.

2 Rhythm Controller

Here, we describe the rhythm controller and a method to examine the controller manipulation method (i.e., relationship between the controller input and avatar motion) when an operator creates avatar motion using the rhythm controller. When a zero-cross point is created on the controller waveform, as shown in Fig. 1, the controller waveform between zero-cross points up to two points prior to this incidence was integrated; thus, the integrated value (ΔS) is used as a velocity output value for the next zero-cross point to operate the avatar. This ΔS, which is the area of a one-cycle wave form of the rhythm controller, is primarily determined by the difference in the cycle (i.e., interval between neighboring zero-cross points, ΔT) and the difference in the amplitude (ΔA). Consequently, avatar velocity V can be expressed in the following form:

$$ V = k \cdot \varDelta S\fallingdotseq f(\varDelta T,\varDelta A) $$

(1)

Fig. 1.

Transformation rule of rhythm controller

In this regard, the operator can independently change both the cycle and amplitude of the rhythm controller waveform with freedom, which means that the operator can also freely changes the relationship between the two. Consequently, the operator can temporally (dynamically) change not only the avatar velocity but also the function f that is equal to the relationship between the controller input and avatar motion—in other words, the controller manipulation method. Note that the operator can determine function f independently of avatar velocity V. In this study, in order to examine function f (i.e., controller manipulation method), (ΔT, ΔA, V) was plotted in a three-dimensional (3D) scatter plot when the zero-cross point was created on the controller waveform (Fig. 2). Furthermore, we use principal component analysis to reduce ΔT and ΔA to one principal component, which is then used as the explanatory variables in a regression analysis by which we are able to estimate function f; the dependent variable in the regression analysis is V. The parameters for determining the linear regression lines in the 3D space obtained from our analysis are the azimuth angle θ and the polar angle ϕ. θ is the variable representing a relationship between ΔT and ΔA. And, ϕ is the variable representing a relationship between V and variables affecting V other than ΔT and ΔA. However, because it is empirically demonstrated that the controller waveform created by a (human) operator is approximated by sine waves, V is determined by only ΔT and ΔA. In other words, ϕ does not change. In this paper, therefore, the differences between the linear regression lines (i.e., differences in the controller manipulation method) are evaluated using the azimuth angle θ.

Fig. 2.

Estimation method of controller manipulation method

3 Maai Creating Agent

The following describes the three design requirements for a Maai creating agent developed in this paper.

• The Maai creating agent uses Eq. (1) to control the movement of its avatar and to create and maintains an appropriate distance (Maai) from the avatar controlled by a (human) controller operator.

• In the same manner as with a (human) operator, the Maai creating agent generates a sound at the zero crossing point of the controller waveform.

• The Maai creating agent evaluates the controller manipulation method of a (human) operator in real-time, and from the changes in the human’s controller manipulation method, predicts the intention of the (human) operator and creates avatar movements.

In order to satisfy the above design requirements, we developed a Maai creating agent (Fig. 3) with four operation modes as described below.

Fig. 3.

Maai creating agent

• Record human’s controller manipulation method while moving forward and backward in a predetermined manner (stationary operation mode)

• Using the Maai creating model by Aizawa [9], maintain an appropriate distance (Maai) from the avatar controlled by a (human) operator by creating avatar movements (Maai operation mode)

• Retreat when the avatar controlled by a (human) operator approached the avatar controlled by the agent using the human’s controller manipulation method associated with collapsing Maai (avoidance operation mode)

• Advance and swing a sword at the avatar controlled by a (human) operator when the avatar controlled by a (human) operator approached the avatar controlled by the agent using the human’s controller manipulation method associated with maintaining Maai (attack operation mode)

In the kendo match experiments to be described later in this paper, the Maai creating agent first runs in stationary operation mode and then starts playing kendo matches against an avatar controlled by a (human) operator. While in a kendo match, the agent operates in one of other three modes other than stationary operation mode. The details of these four operation modes are explained below.

3.1 Stationary Operation Mode

In stationary operation mode, the agent records the human’s controller manipulation method in the following manner. First, a (human) operator moves its avatar while maintaining a constant distance (Maai) from the agent’s avatar that moves forward and backward in a predetermined manner. During this time interval, for each frame (frame rate of 50 [fps]), the azimuth angle θ of the human’s controller manipulation method is calculated from ΔT and ΔA data of the (human) operator in the most recent 0.5 [s] interval using the least-squares method. Then, the maximum θ _max , minimum θ _min and standard deviation θ _SD is calculated for the θ value for 1 [s] interval. When the value of θ _SD becomes below 0.1, the human’s controller manipulation method are considered stable, and the corresponding θ _max and θ _min are recorded as the human’s controller manipulation method associated with maintaining Maai.

Next, the (human) operator, after maintaining a distance (Maai) from the agent’s avatar moving back and forth in a predetermined manner, collapses the distance (Maai), approaches and swings his sword at agent’s avatar at the timing of his choice. At this time, the azimuth angle θ _c of the human’s controller manipulation method is calculated, using the least-squares method, from ΔT and ΔA data of the (human) operator in the 1 [s] immediately before approaching the agent’s avatar. Then, the value of θ _c is recorded as the human’s controller manipulation method associated with collapsing Maai.

3.2 Maai Operation Mode

In Maai operation mode, using a Maai creating model by Aizawa, the agent creates and maintains a distance (Maai) between its aviator and the avatar controlled by a (human) operator according to the procedure shown in Fig. 4. First, the agent estimates the distance ΔX between human operator’s and agent’s avatars at time $t + \hat{T}$ of the next zero crossing point of the agent’s controller waveform from the positions and speeds of these two avatars at current time t (Fig. 4 (a)). Next, the agent determines the speed of its avatar ($\hat{V}_{com}$) at time $t + \hat{T}$ using the Eq. (2) shown in Fig. 4 (b). Note that, when the avatar moves according to Eq. (2), agent’s avatar will move away from the human operator’s avatar when the estimated distance ΔX between avatars is small, and it will approach the human operator’s avatar, when ΔX is large. Therefore, agent’s avatar adjusts its location such that its distance from the human operator’s avatar becomes closer to the ΔX value that makes the right side of Eq. (2) 0. Further, the agent determines the agent’s controller waveform ${\hat{\varDelta }}T$, ${\hat{\varDelta }}A$, $\hat{T}$, and $\hat{A}$ (Fig. 4 (c)) from the predetermined values of the azimuth angle θ and polar angle ϕ of agent’s controller manipulation method and $\hat{V}$ _com determined using Eq. (2). Finally, from these values, using the method shown in Fig. 4, the agent generates its controller waveform for the time interval from t to $t + \hat{T}$. Note that, in Maai operation mode, the agents calculates, for each frame, the azimuth angle θ _r of the human’s controller manipulation method, using the least-squares method, from ΔT and ΔA data of the human operator in the most recent 0.1 [s] interval. Based on this calculated azimuth angle value θ _r , the agent determines whether to transition to avoidance operation mode or attack operation mode. This is discussed in more detail below.

Fig. 4.

Maai operation mode

3.3 Avoidance Operation Mode

In this paper, the agent assumes that the human operator intends to collapse Maai, when the azimuth angle θ _r of the human’s controller manipulation method is within the range of θ _c  ± 1.5 [deg], and when the human operator’s avatar moves forward. In this case, agent’s avatar transitions to avoidance operation mode and retreats to the outside of the kendo match field (i.e., outside of the display screen) at avatar’s maximum speed V _max that the human controller operator can achieve using the rhythm controller. Note that, in the kendo match between the agent and the human operator, if the human controller operator strikes agent’s avatar with his/her sword, the agent assumes that the human operator changed his/her controller manipulation method associated with collapsing Maai and updates the value of θ _c to the value used in this case.

3.4 Attack Operation Mode

In this paper, the agent considers that it is likely to succeed in striking the human operator’s avatar with its sword, when human operator intends to maintain Maai with the distance between avatars small. Consequently, when these four conditions are all satisfied, the agent transitions to attack operation mode, advances its avatar at the maximum speed V _max and swings its sword.

• A) The value of θ _r lies between θ _max and θ _min .

• B) The human avatar is moving forward.

• C) The distance between agent’s and human avatars is within 300 (the size of the kendo match field is 1000).

• D) 8 [s] or more has passed since agent’s moving from avoidance operation mode to Maai operation mode.

Note that condition D) is introduced to prevent frequent transition to attack operation mode.

4 Experiment Results

Through the use of a Maai creating agent developed in this study, we researched the difference of the controller manipulation method depending on whether humans have an intention to create Maai. First, we investigated the movement of agent’s avatar when the human operator freely controls its avatar without considering that he/she attempts to maintain an appropriate distance (Maai) from agent’s avatar that only operates in Maai operation mode. The results showed that the cross-correlation coefficient of the changes in agent’s and human avatar positions was 0.98 with a delay time of 21 [ms] (Fig. 5(a)). From this, we conclude that it was possible for the agent in Maai operation mode to synchronize the movement of its avatar to that of the human avatar. Note that, in this experiment, when the azimuth angle of agent’s controller manipulation method was initially set to 35 [deg], a value that is different from the initial azimuth angle of the human’s controller manipulation method (54 [deg]), the azimuth angle of the human’s controller manipulation method keeps 54 [deg], and two azimuth angle values never converged to the single common value (Fig. 5(b)).

Fig. 5.

Results of experiment between human and agent

Next, we instructed the human operator to move his/her avatar freely while creating constant Maai with agent’s avatar and conducted an experiment to investigate interactions between the human operator and the agent that only operates in Maai operation mode. In the experiments, the azimuth angle of the agent’s controller manipulation method changed from 50 [deg] in the first 30 [s] of the experiment to 70 [deg] in the second 30 [s]. The results showed that the cross-correlation coefficient of the changes in agent’s and human avatar positions was 0.98 with a delay time of 15 [ms] (Fig. 6(a)). This result shows that agent and human operator create the synchronized motion between mutual avatars before as well as after experiment. In addition, from examining the azimuth of the human’s controller manipulation method, we confirmed that the human operator adjusted the azimuth angle of his/her controller manipulation method to match that of the agent’s controller manipulation method, resulting in different azimuth angles between the first and the second half of the experiment (Fig. 6 (b)).

Fig. 6.

Results of cooperative move experiment between human and agent

Next, we conducted experiments where a Maai creating agent and a human operator played 20 kendo matches. The human test subjects were all proficient with rhythm controller operations and skilled in kendo matches using the controller. Further, in the experiments, for all agent operation modes, the azimuth angle of the agent’s controller manipulation method was set to the average azimuth value (48 [deg]) of five human test subjects when they were maintaining Maai. In the experiments, the human test subjects were found to continue to lose a kendo match, and this was not only because the agent was able to predict the intention of the human test subjects to collapse Maai and successfully avoided being hit by the sword, but also because the human test subjects were completely unable to avoid agent’s sword. For the first time in the ninth kendo match, the human test subject was finally able to create new controller manipulation method that were different from θ _c recorded in stationary operation mode and won the match over the agent (Fig. 7(b)). However, winning percentage of the human test subjects was only 20 [%].

Fig. 7.

Results of Kendo match between human and agent

5 Discussions

Results in Fig. 5 in the previous section show that when a human test subject had no intention of maintaining Maai with his opponent (agent), he/she did not adjust his/her controller manipulation method to match those of the opponent (agent), even if the avatars’ movements were synchronized. On the other hand, when a human test subject had intention of maintaining Maai with his opponent (agent), he/she adjusted his/her controller manipulation method to match those of the opponent (agent), even if the opponent (agent) changed its controller manipulation method in the middle of the experiment (Fig. 6). These results show that the human’s intention to create Maai with the opponent leads to the potential for changes in the human’s controller manipulation method. Furthermore, a human test subject played kendo matches against a Maai creating agent that predicts the intension of a human test subject. We observed that the winning percentage of the agent reached 80 [%]. Human test subjects who competed against the agent commented that “the agent successfully retreated as soon as I started thinking of narrowing Maai to agent’s avatar” and “the agent seized a chance to attack and beat me”. These results indicate that the intention of a human as to whether he/she intends to maintain or collapse Maai is reflected in his/her controller manipulation method. In a creation of an avatar movement using the rhythm controller, the controller manipulation method does not surface to the conscious mind. In other words, the changes in the controller manipulation method are realized by means of an unconscious process that does not directly involve the conscious mind. Consequently, our research shows that an unconscious process plays a role in the creation of Maai involving the human’s intention. Furthermore, our research shows that an unconscious process like the changes in the controller manipulation method contributes to the emergence of the intension of a human as to whether he/she intends to maintain or collapse Maai in the Kendo match where humans intentionally collapse Maai. However, it is not clear yet that the changes in the controller manipulation method as the unconscious process precede the conscious perception of such an intention of a human. We plan to research this problem in the future.

Note that a Maai creating agent developed in this study has the following problems. First, this agent can create only constant Maai with a human. In other words, this agent cannot create various Maai like the kendo match between humans. Results of our previous studies [6] show that entrainment [10–12] in multiple cycles is created in rhythm controller waveform of human operators when Maai is improvisationally created between two avatars in a Kendo match. However, a controller waveform of this agent does not change through the interaction with human operators. Furthermore, the controller manipulation method of this agent does not change in the Kendo match. Therefore, this agent cannot realize the emergence of the new controller manipulation method as shown in Fig. 7. We plan to study a way to solve the above-mentioned problems.

6 Summary

In our research, we developed a Maai creating agent that predicts human’s intention from the changes in human’s controller manipulation method (i.e., relationship between controller input and avatar motion) of a rhythm controller and determines its movement accordingly. And, we examined whether the human’s intention is reflected in the controller manipulation method in the creation of Maai involving the human’s intention. From results, we showed that the human’s intention to create Maai with the opponent leads to the potential for changes in the human’s controller manipulation method. Furthermore, we demonstrated that the agent can predict intentions of a human operator as to whether he/she intends to maintain or collapse Maai from the changes in his/her controller manipulation method. In other words, we showed that such an intention of a human is expressed in his/her controller manipulation method. Consequently, our research shows that an unconscious process plays a role in the creation of Maai involving the human’s intention.