CamTroller: An Auxiliary Tool for Controlling Your Avatar in PC Games Using Natural Motion Mapping

1.1 limitations in Current Commercial PC Games

Video games hold immense significance in the gaming market, providing immersive and interactive experiences that foster social connectivity and artistic expression. Upon analyzing the distribution of video game sales by genre, it becomes evident that action games dominate the market with a significant share of 26.9%, followed by shooting games at 20.9% and role-playing games at 11.3% [59].

In order to involve more naturalness in the interaction system of current commercial PC games, we focus on the PC games in which one player controls only one role. This is because this one-player-to-one-avatar mapping is the most suitable and direct situation for natural mapping, which means what the player does could direct the map to what the avatar does. By finding the most comfortable and natural way for players, we are exploring how the technology could adapt to player needs and preferences with NUI [13].

In the early stage, role-playing shooting games on PC like Quake [18] with limited fundamental actions like "WASD" are easy to learn while fingers are adequate for controlling; there is no demand for involving extra input methods such as motion control. However, not only the newly released shooting games but also other role-playing and action games aim to create a more realistic experience by adding a lot of actions—the complexity of action control pleasing game players and winning success in marketing. However, a strong conflict exists between complicated actions and the limited interaction system in which people can only control the avatar through the mouse and keyboard. This conflict creates a barrier for general players to enjoy the free combination of different actions because of the related memory burden, physical finger shortage, learning cost, and related health problems when using a keyboard and mouse.

1.2 Why Camtroller?

Memory burden of remembering keybindings, difficulties in forming muscle memory for ’touch typing’ in games, as well as the ’finger shortage’ problem, resulting in a high entry difficulty and steep learning curves, significantly affecting the gaming experience of beginners and ordinary players. These obstacles turned out that current commercial one-to-one avatar mapping PC games lack naturalness in their user interaction system. Therefore, it is where the NUI should be brought in to overcome these drawbacks and improve intuitiveness to benefit a large group of players. Trying to reduce the mentioned limitations, we developed the concept of Camtroller, an auxiliary tool, by natural mapping control as supplemental input to closed-source commercial games. Offering options for players to trigger some actions of the avatar by conducting the same or similar movements with their body will be effective in helping those who haven’t memorized the keybindings. Naturally mapping players’ motion to drive the avatar also enables them to conduct actions without looking down at the keyboard for the keys to be pressed with ’touch typing’ when fighting against the enemy in games, avoiding the risk of being defeated because of losing the view. Moreover, utilizing motion control can turn body parts besides the hand, such as the head, torso, and feet, into ’extra fingers’ for input, addressing the problem of ’finger shortage’ in an intuitive way.

Based on the detailed investigation of game players, this work introduced Camtroller, an auxiliary tool following the NUI concept for involving natural mapping control in current one-to-one avatar mapping PC games to overcome the aforementioned problems. Also, this work validated this concept by developing a system in one certain game through the combination of a motion-tracking solution and a mouse/keystroke emulator.

For the structure of this work, we first carefully illustrated the advantage of NUI and natural mapping control, investigated the gaming experience limitation of commercial one-to-one avatar mapping PC games, and introduced the NUI concept of Camtroller to enhance player experiences in terms of decreasing memory burden, eliminating finger shortage, reducing leaning cost and allowing body movement to avoid muscle fatigue from the longtime keyboard-mouse usage. Then we reviewed the existing works in the field of motion controllers and compared different technical routes in the section of related works. Motion tracking algorithms through a commodity RGB camera are selected since it has the least requirement for hardware and does not need to wear anything on the body. We chose PUBG as the testing platform for verifying our concept since this game is phenomenal and has a tremendous amount of players. In the concept design part, we conducted a survey to find out players’ habits to confirm the player needs and then analyzed which actions need to be mapped and how they can be mapped naturally. Then, we introduced the technical implementations from the 3D position and rotation measurement with accuracy test to the detailed posture calculation and triggering criteria of each selected motion in the technical implementation section. All technical algorithms of CamTroller were firstly confirmed by a feasibility test and then followed by a user study with objective performance measurement and subjective intuitiveness assessment. The results were discussed, and we conclude this work with current limitations and potential extensions.

2.1 Wearable Motion Controller

Many researchers and developers have attempted to develop small-size wearable motion controllers that can be categorized into passive and active tracking devices.

2.2 Non-physical Motion Controller

Non-physical motion-sensing devices, such as the Kinect, use the depth sensor and the RGB camera to detect players’ actions, allowing direct game playing with body motion. Much research has been done to study the enhancement in the game experience of utilizing the Kinect [8, 35, 40]. However, the extra costs of Kinect and some other mature commercial products, like TrackHat, prevent wider use by players due to their complex hardware [61].

Motion-sensing applications only with a commodity camera should be easily expanded according to economic reasons. Wang et al. used a commodity webcam tracking the face position to improve role-playing as well as presence effectively [57]. However, only 2D information about the center position of the face could be retrieved, which made the head orientation angle inaccurate. Moreover, their face recognition method ran lower than 10 FPS, which is insufficient during gaming. Moreover, the motion control was only in a specific prototype game, which constrained the versatility. This problem also happened to the research of Sko Torben et al.[47, 48]. Their approach is only executable for a self-developed game or a game with an adjustable gaming engine. The stability was also problematic when the scenario moved from a clean lab background to the home with more interfering objects [47]. Moreover, the FaceAPI used in the research is a commercial product that is difficult to access. In this study, only open-source toolkits are considered to provide posture recognition.

3.1 Player Survey

As PUBG is selected to validate our concept of CamTroller, we first conducted an online survey. This questionnaire was launched on an online questionnaire platform (Sojump) and was mainly spread among college student players of PUBG. There are a total of 28 multiple-choice questions in the questionnaire. Questions are set to understand several general questions: 1. the players’ gaming habits on directional moving, peeking, and item consumption; 2. why they assign their fingers in a certain way and whether they explored some new ways; 3. Also, their experience with different approaches to item taking is also asked to figure out any possible pain point. A participant will answer a maximum of 18 questions since there are dependent relationships between selected options and the following questions. The details of each question and the dependent relationship can be viewed in the appendix.

There are 190 responses in total, with 122 valid responses and 68 non-players rejected at the beginning.

For performing ’WASD’, the result turned out that the 92% and 87% players used the middle finger to move forward and backward. 88% of them use their ring finger to move left and index finger to move right. None of them use the customized key to replace ’WASD’.

For peeking, the response turned out that 11% of players never used the operation of peeking. Among the players that perform peeking in games, 80% of them use the ring finger for left peeking and the index finger for right finger (basic action). 8% and 12% of them use the middle finger for left peeking and right peeking, respectively (pro action). 57% of players that used peeking never tried to move and peek simultaneously. For why they perform peeking in this way, the two main reasons they never try this is that they never think about this operation (53%) and think the same finger can not press two keys simultaneously (42%). These results confirm our mentioned insight that most amateur players use the same finger of performing left and right walking to perform left and right peeking, which leads to a ’finger shortage’ and the situation that more than half of them did not even have a try to perform them simultaneously.

For the approaches of taking boost items, 58% of players use open the inventory, which is the basic approach from the initial PUBG tutorial. Meanwhile, 24% use the wheel menu, and only 18% use corresponding keys. It turned out that most of the PUBG players chose more complex approaches to lighten memory and keystroke burdens. However, when responding to the frequency that they feel they are unable to use boosting items when engaging with enemies, half of them often feel stuck in this problem. Only 2% of the players were never disturbed by this problem.

As a conclusion, rare players remap the key to certain actions. For peeking and moving, as the main part players are using the same finger for these two actions, they don’t even think about performing them simultaneously. In other words, the physical lack of operational freedom (finger shortage) directly limits their ability to try new operational combinations at a cognition level. Hence, it is predictable that if CamTroller could involve a higher degree of freedom for action control, the way players interact with the gaming platform could be changed from a fundamental cognitive level. Also, the survey turned out that the boost-item taking is also a pain point in the gaming experiences, which CamTroller could also try to deal with.

3.2 Motion Selection and Analysis

Based on the survey, it is confirmed that there are some issues that the concept of CamTroller could deal with. However, when digging into the design details of how CamTroller maps to certain avatar actions, many details still need to be discussed. In this section, in the first step, we selected the avatar motions that could be enhanced from the perspective of using a keyboard and mouse. In the second step, we try to extract detectable features from the avatar motions. Finally, we considered the basic posture of the computer game player and tried to find player motion with similar detectable features of the avatar motion to maintain a high intuitiveness level.

3.2.2 Step2: Analysis features of avatar motion.

After determining the avatar’s actions that need to be mapped, the following step is analyzing the detectable features of avatar motion, such as angles, distances, and positions For the purpose of finding proper features of the body suitable for motion sensing, we first analyzed how humans perform these avatar motions in the physical world Figure 2. The participants were asked to imitate five movements of the avatar in the game, including peeking, crouching, and consuming three types of boost items. Neutral posture was also performed, acting as the ’reference zero’. To observe the peeking and crouching motion, participants were holding a toy pistol to simulate the scenario in the game. As for the boost item consumption observation, vitamin pills were used to simulate the painkiller, a can of cola was used to simulate the energy drink, and a pen was used to simulate the syringe of adrenaline. In total, five participants took part in the observation activity.

Figure 2:

View Figure

The result of observation indicated that peeking is usually conducted by leaning the torso and accompanied by the head lateral bend, as shown in Figure 2 (c). Meanwhile, crouching (shown in Figure2 (a)) causes the whole body to move down. Therefore, head orientation angle and position can be used as the detection parameters. For the boosting items, using energy drinks, painkiller pills, and adrenaline syringes in the real world is usually accompanied by various gestures. Fingers flex at a large angle while holding an energy drink bottle. On the other hand, taking pills keeps the fingers straight or slightly curved. Both of these motions bring the hand closer to the mouth, which means the distance between the hand and mouth will be significantly smaller than normal. These two can be viewed in Figure 2 (d) and (e). Conducting adrenaline injections, as shown in Figure 2 (f), requires hand gestures similar to drinking, with fingers wrapped around the needle, but the hand position differs as it is applied to the heart. Hence, the flexion angle of the finger, as well as the distance between the hand to the mouth, can be utilized as the parameter for detecting.

3.2.3 Step3: Mapping player motion to avatar.

Then, consider how similar actions could be used to drive the avatar with the intuitiveness principles [3] under the game-playing situations. Based on the reality that all motions are performed while the players are sitting on a chair and using the keyboard and mouse for basic actions (WASD and aiming), which means they cannot exactly do crouching or peeking as the avatar during game playing. Hence, player motions with the corresponding detectable features are adopted for mapping player motion to avatars to achieve the principle of ’natural’ or intuitive [60].

Ideally, the avatar should simultaneously mimic the player’s movements accurately. When the player leans or moves down the head for a certain degree or distance, the avatar should perform the corresponding action. However, in the game, peeking and crouching are usually treated as state transitions, meaning that the avatar switches to another state when the corresponding key is pressed, and there is no half-peeking or half-crouching. Thus, a proper threshold is needed for triggering peeking and crouching. When the player lateral bends their head left or right above the threshold value, the keys for left or right peeking, typically Q and E, will be pressed. Similarly, when the player moves down or up their head over a specific distance, it should trigger the crouch or jump of avatars in the game.

As for using boost items in the game, pressing related keys will be triggered by hand gestures and positions. When the hand is close enough to the mouth, and the distance is below the critical value, the flat hand will result in pressing the key for painkillers, whereas the flexed hand will result in drinking the energy drink. Curved fingers will result in key pressing for adrenaline injection when the head is closer to the heart.

Figure 3:

View Figure

When looking around without moving the torso (free-look), the change of view is mainly accomplished by the axial rotation of the head. Free-looking in the game is a continuous process that allows the avatar to follow the head rotation. The rotation degree of the avatars’ view is linearly related to the mouse moving distance, so when the player presses the key for the free-looking, CamTroller will track the player’s head’s axial rotation and flexion-extension Angle, as shown in Figure 3 (a), to move the view correspondingly by simulating the movement of the mouse.

4.1 Head Position Estimation

The planar head position can be represented by the position of the nose tip x_nose and y_nose. Since the sitting posture will gradually change, it is unsuitable only to use the nose coordinates. It is better to depend on a reference. Under this application, the shoulder is chosen. The position of both shoulders can be retrieved with the MediaPipe ’Pose’ solution. The landmarks of the left and shoulder, landmarks 11 and 12, are shown in Figure 7 (b). The coordinate of the left and right shoulder, which are \((x_{shoulder_L}, y_{shoulder_L})\) and \((x_{shoulder_R}, y_{shoulder_R})\), can calculate the middle point’s coordinate, \((x_{shoulder_M}, y_{shoulder_M})\). Then, the head relative position is \(x_{s2n}=x_{nose}-x_{shoulder_M}\) and \(y_{s2n}=y_{nose}-y_{shoulder_M}\). The vertical distance between the nose and the shoulder is L_s2n = |y_s2n|

4.2 Head Orientation Estimation

Due to the complex structure of the cervical spine, head motion is deeply coupled. To have a straightforward estimation method, angles must be separated. Since flexion-extension mainly depends on the joint between the C0 (Occipital Bone, a bone of the head connecting the cervical spine) and the C1 (Atlas, the first cervical vertebra), while the C1-C2 (Atlas-Axis) joint, the one between the first and the second vertebra, contributes most to the axial rotation and the lateral bending relies more on the inferior vertebrae [4], we may simplify the head motion is carried out in the sequence of lateral bending, axial rotation, and flexion-extension.

Since MediaPipe generates data under a coordinate system different from what people generally utilize, we first introduced the angle calculation in a general coordination system and then mapped MediaPipe’s coordination to it.

4.2.1 Angles Calculation Under General Coordinate.

In a general coordination system for head pose, the center of the head is the original point with a right-hand coordinate system that takes the left-hand direction as the X-Axis, the facing direction of the head can be described through Euler Angle in the sequence of ZYX (Figure 3 (b)).

Figure 4:

View Figure

Lateral bending is only related to the planar coordinate data x and y. Two symmetric mesh points along the middle plane of the head (the Left point and Right point in Figure 4) are able to calculate the lateral bending Angle. The left point coordinate data is with a subscript L, and the same for the right one. The lateral bending Angle can be derived by: (1) \(\begin{equation} \theta _{LB}=atan\left(\frac{y_L-\ y_R\ }{x_L-\ x_R}\right) \end{equation} \)

Axial rotation depends not only on the coordinate data x and y but also on the depth z. Two symmetric mesh points are still adequate for calculating it. Included the lateral bending, the equation is: (2) \(\begin{equation} \theta _{AR}=atan\left(\frac{z_R-\ z_L\ }{\left|\left(x_R-\ x_L\right)/cos\left(\theta _{LB}\right)\right|}\right) \end{equation} \)

Flexion-extension can be determined by two mesh points, which are located at the upper part of the head and the lower part. Considering the previous two angles, it can be calculated by: (3) \(\begin{equation} \theta _{FE}=atan\left(\frac{\left(z_{Upper}-\ z_{Lower}\right)/cos\left(\theta _{AR}\right)}{\left|\left(y_{Upper}-\ y_{Lower}\right)/cos\left(\theta _{LB}\right)\right|}\right) \end{equation} \)

4.3 Accuracy Evaluation Test

Since the estimation method for head orientation angles is based on a machine learning solution whose accuracy has not been tested and validated, we conduct a test to evaluate its accuracy.

4.3.1 Apparatus.

Due to the difficulty and the unavoidable errors of measuring the angle of a real human, a head model is more suitable for accuracy validation. A mock head model is installed on a tripod with two degrees of freedom (DOF) (CIMAPRO LD-2R) that can pan and tilt to simulate the motion of a human head. The degree of freedom from panning is for axial rotation, while the one from tilting is for lateral bending and flexion-extension. There are scales on the tripod head for obtaining angles. A webcam (Rapoo C270AF) working as an image-capturing device is connected to a laptop (Zephyrus G14 2022) where the program is running. A grid transparency sheet with the scale is snapped to the desk and used as the camera distance reference. Figure 5 shows a photo of all the apparatus.

Figure 5:

View Figure

4.3.3 Results.

The error percentages for lateral bending, axial rotation, and flexion-extension are 3.88%, 5.37%, and 6.7%, respectively. Head posture angle results are also displayed in Figure 6. As seen in the figure, lateral bending is the most accurate since it is only related to the planar coordinate data x and y, while no estimated depth data z is involved. The axial rotation estimation is of minor error within 40 deg. The potential reason for the error is that the landmarks used for angle calculation are invisible over 40 degrees, and MediaPipe predicts their location. Also, it can be observed that the estimation of Extension is better than the Flexion. Generally speaking, the accuracy is adequate for motion detection.

Figure 6:

View Figure

4.4 Hand Gesture and Position Estimation

Figure 7:

View Figure

The coordinate data of the middle finger of the right hand are utilized for estimating the hand gesture. To figure out the different boosting items, the curving condition of the hand is required. The coordinate data of hand landmarks 9, 10, and 11(Figure 7 (a)) is used for calculating the angle between line segments from hand landmarks 9 to 10 and from 10 to 11. Since the depth data z is not provided, the projected angle of the middle finger will be applied: (7) \(\begin{equation} \theta _{proj}=\left|atan2\left(y_9-y_{10},\ x_9-x_{10}\right)-atan2\left(y_{11}-y_{10},\ x_{11}-x_{10}\right)\right| \end{equation} \) The palm is deemed flat when the θ_proj is smaller than 20 deg. Also, it will be considered curved when the θ_projected is greater than 30 deg.

For the right hand position(x_hand, y_hand), it is represented by the coordinates of the landmark 9 (x₉, y₉). However, only knowing the hand’s position is not enough to detect different boosting items. The position of the mouth(x_mouth, y_mouth) as well as the heart should also be retrieved. As for the mouth, it can be retrieved from the landmarks of Face Mesh, while the heart has to use another solution in the MediaPipe, which will be the ’Pose’. Although there are no landmarks for the heart, it does have one, landmark 11, for the left shoulder(x_shoulder, y_shoulder), which is close to the heart, so it will be utilized to approximate the heart. Then, the distance between hand to mouth and hand to shoulder will be: (8) \(\begin{equation} d_{h2m}=\sqrt {(x_{hand}-x_{mouth})^2+(y_{hand}-y_{mouth})^2} \end{equation} \) (9) \(\begin{equation} d_{h2s}=\sqrt {(x_{hand}-x_{shoulder})^2+(y_{hand}-y_{shoulder})^2} \end{equation} \)

4.5 Program Structure

4.5.2 Motion Detection Criteria.

All selected motions in the shooting game are related to keyboard interaction except for the free-looking. The initial triggering criteria are based on the angle or the displacement, but this method causes a noticeable lag when the threshold value is large to avoid unwanted triggering since the head is not fixed at a position in a steady state but gently twitches randomly. More details can be viewed in Figure 8.

Figure 8:

View Figure

Also, setting the threshold to small values causes system instability, while a proper value is hard to determine and difficult to fit for a large group of people. So, we will use angular velocities of lateral bending \(\dot{\theta }_{LB}\) and linear velocities of head movement along Y-Axis \(\dot{y}_{nose}\), which are the derivatives of angles and displacements. The setting of ’Dead-zone’ commonly applied on the game controller will be used to enhance stability and avoid wrong action recognition in the format of angle θ_DZ, position y_DZ, and factor k_DZ. The criteria values for motions to be considered happening based on the collectible data are listed in Table 1. Figure 9 and Figure 10 are also presented for a better illustration.

Table 1:

Motion	Criteria
Left Peeking	\({\dot{\theta }_{LB}}\lt -\dot{\theta }_{thre}\) & θLB < − θDZ
Release Left Peeking	(\({\dot{\theta }_{LB}}\gt \dot{\theta }_{thre}\) & θLB < − θDZ) or (θLB > − θDZ)
Right Peeking	\({\dot{\theta }_{LB}}\gt \dot{\theta }_{thre}\) & θLB > θDZ
Release Right Peeking	(\({\dot{\theta }_{LB}}\lt -\dot{\theta }_{thre}\) & θLB > θDZ) or (θLB < θDZ)
Crouching	(\(\dot{y}_{nose}\gt \dot{y}_{thre}\) & \(k_{thre}\times L_{s2n_0}\lt L_{s2n}\lt k_{DZ}\times L_{s2n_0}\)) or (\(L_{s2n}\lt k_{thre}\times L_{s2n_0}\))
Release Crouching	(\(\dot{y}_{nose}\lt -\dot{y}_{thre}\) & \(k_{thre}\times L_{s2n_0}\lt L_{s2n}\lt k_{DZ}\times L_{s2n_0}\)) or (\(L_{s2n}\gt k_{DZ}\times L_{s2n_0}\))
Jumping	\(y_{shoulder_M}\lt y_{DZ}\) & \(\dot{y}_{nose}\lt -\dot{y}_{thre}\)
Drinking Energy Drink	dh2m < Δdthre & θproj > αlarge
Drinking Painkiller Pills	dh2m < Δdthre & θproj < αsmall
Injecting Adrenaline	dh2s < Δdthre & θproj > αlarge

View Table

Table 1: Criteria for Different Motions

Figure 9:

View Figure

Figure 10:

View Figure

The free-looking is related to mouse interaction. The axial rotation of the head triggers the horizontal mouse movement, while the flexion-extension of the head leads to the vertical one. When the free-looking key, which is usually ’Alt’, is pressed, free-looking is considered to be occurring, and the values of head axial rotation angle θ_AR and flexion-extension angle θ_FE drive the view to change.

4.6 Feasibility Validation Test

We conducted one pilot test to verify the CamTroller’s feasibility and one user study in the real gaming scenario (PUBG). Both the CamTroller software and game PUBG are running on the laptop ROG Zephyrus G14 (Processor: AMD Ryzen™ 7 6800HS, Display Card: AMD Radeon™ RX 6700S 8GB, Memory: 8GB DDR5) plugged with an external commodity webcam Rapoo C270AF (FOV: 85.8deg, Resolution: 1920*1080p) for image capturing. The game’s content is shown on an external monitor (Xiaomi Curved Gaming Monitor 30", model number: RMMNT30HFCW). A wireless gaming mouse (Logitech G304) and a wireless mechanical keyboard (Dareu EK810) are used as input devices for the tests. The participants sit in chairs facing the monitor, about 50 cm away.

Nine participants attended the feasibility test and were asked to perform eleven motions to drive the avatar in the game, which are: 1) Head Moving Up for jumping, 2) Head Moving Down for crouching, 3) Head Left Bending for left peeking, 4) Head Right Bending for right peeking, 5) Head Left Rotation for free looking left, 6) Head Right Rotation for free looking right, 7) Head Flexion for free looking down, 8) Head Extension for free looking up, 9) Drink Energy Drink, 10) Take Painkiller, and 11) Inject Adrenaline. Each motion was performed three times in total by each participant, which means each participant was required to perform 33 motions in total. During the test, each participant was required to follow the experimenter’s instructions to perform all 33 motions one by one in a fully random order. If the avatar performed the correct motion, it was considered successful.

The success rates for eleven motions are shown in Table 2. The overall success rate is 99.33%. This high success indicates that it is feasible for gaming from the perspective of success rate.

4.7 Computation Power Occupation and Performance Test

We also conducted a test in PUBG to determine how much computation resources will be taken when running the CamTroller on the ROG Zephyrus G14 laptop. When playing in the training mode, the CamTroller took only 5% to 10% of the CPU without causing a noticeable frame-rate drop (average within 5 FPS). We also conducted a test on Kinect as a comparison. After connecting the Kinect to the same computer as an input device, the essential software for data collecting, Azure Kinect Viewer, occupied 30% to 40% of CPU usage and caused a large frame rate drop for over 30 FPS. The study by Shengmei Liu et al.[29] pointed out that the quality of experience (QoE) decreases when the frame rate drops. Moreover, the CamTroller was able to run at 25-30 FPS when playing PUBG, which means the response time of the CamTroller is about 0.033 s to 0.04s. Meanwhile, the Azure Kinect Viewer could only run within 5 FPS, which suggested that its response time is larger than 0.2s and would influence the gaming experience.

We also obtained the time for players to trigger different motions. The apparatus is the same as the feasibility test. When participants performed different motions in the test, a camera was set to record both the motion of the players and the screen. The video was used to measure the difference between the start time of players’ motion and the start time of the avatar’s motion, which was considered as the trigger time. Five participants attended this test and performed five types of motion (peeking, crouching, drinking energy drinks, taking painkillers, and injecting adrenaline) three times. The trigger time for peeking and crouching mostly falls into the range between 0.3s and 0.5s. The mean value for peeking and crouching is 0.39s (std=0.056) and 0.41s (std=0.059). As for boost item consumption, most participants trigger the motions of drinking energy, taking painkillers, and injecting adrenaline in the range from 0.5s to 0.8s. The mean value for these three motions is 0.68s (std=0.088), 0.66s (std=0.069), and 0.69s (std=0.083). The variation is relatively large since the trigger time is highly dependent on the participant’s motion speed. The boost item consumption motions have a longer trigger time than peeking and crouching since participants have to move their hands from the desk to the mouth or heart, costing more time.

Figure 11:

View Figure

Figure 12:

View Figure

5.1 Apparatus and Participants

The apparatus is the same as the Feasibility Validation Test. A total of 18 participants without cervical or upper limb problems were recruited to attend the user study, and they were all students or employees from the university. All participants had experience playing PUBG or similar shooting games but were not professional players. Before participating in the experiment, they were required to pass a small test of the basic understanding of shooting game operations, such as using WASD keys to move and using the mouse to aim and shoot.

5.2 Procedure

5.3 Data Analysis

All statistical tests are executed by SPSS 27. The QUESI results with five subscales on three approaches on boot item task, and peeking were statistically analyzed through a repeated measure ANOVA with a Greenhouse–Geisser correction and following post hoc pairwise comparisons with Bonferroni adjustment.

For the peeking performance, as in the whole 90-second video clip, there usually are a few errors, such as the avatar not being able to reach out to see the enemy. An error-free section with a length of 15 seconds with relatively stable performance was retrieved to measure the time of peeking on both sides. To estimate the time more precisely, the loop of peeking starts from the view, reaches the most left side to the right side, and goes back to the most left side view as an end as shown in Figure 12. The mean loop time is calculated as an indication of the mean performance of each participant. The mean performance time of each approach is tested through an independent-sample Kruskal-Wallis test with post hoc tests.

5.4 Results

Figure 13:

View Figure

For boost item tasks, the within-subject effects of repeated measure ANOVA on QUESI average score of each subscale turned out that the approaches (F(1.69, 28.80)=16.79; p<0.001, partial eta squared = 0.496) are significant. The pairwise comparisons of approaches show that the CamTroller (M=4.596, SE=0.064) has a significantly higher mean intuitiveness score than Basic (M=3.220, SE=0.281, p<0.001) and Pro(M=3.472, SE=0.247, p<0.001), while the mean QUESI scores are not different between Basic and Pro (p=0.692). For each subscale, the mean score keeps the same significance structures between different approaches. The result shown in Figure 13 (a) indicated that the intuitiveness of the ’CamTroller’ operation is significantly higher than the ’Basic’ and ’Pro’ operations in all five sub-scales.

For the peeking task, the statistical results are similar to the boosting item task (Figure 13(b)). The repeated measure ANOVA also revealed significant within-subject effects on the average score of different approaches (F(1.916, 32.57)=16.362; p<0.001, partial eta squared = 0.490). The pairwise comparisons among the approaches showed that the CamTroller approach (M=4.678, SE=0.089) had a significantly higher mean QUESI score compared to the Basic approach (M=3.067, SE=0.299, p<0.001) and the Pro approach (M=3.341, SE=0.269, p<0.001). There was no significant difference in the mean QUESI scores between the Basic and Pro approaches (p=0.276). These findings were consistent across all five subscales, indicating that the CamTroller approach demonstrated significantly higher intuitiveness in operation compared to both the Basic and Pro approaches.

The loop time of each approach is shown in Figure 13(c). The independent-sample Kruskal-Wallis test showed that the approaches on time for performing a peeking loop have a significant effect (H=32.407, p<0.001). The following pairwise comparison with the Bonferroni correction showed that the CamTroller is comparable to the Pro operation (p=0.116). The mean loop time of Pro and CamTroller are significantly higher than the ’Basic’ operation (p<0.001 and p=0.001, respectively).

6.1 User Study Results

The pairwise comparison on the QUESI questionnaire showed that the natural mapping using Camtroller not only performed better in the overall intuitiveness but also in each subscale of intuiveness: subjective mental workload, perceived achievement of goals, perceived effort of learning, familiarity, and perceived error rate. Based on these results, we believe the CamTroller could help players perform better when stuck in severe competition situations that require high levels of attention, focus, and decision-making in their brains. In such a situation, they need to rapidly assess the situation, evaluate multiple options, and choose the most appropriate actions to take, and even a little bit of release of mental workload can make a difference. Hence, although the CamTroller has no advantage over the Pro in simple keeping tasks, in complicated playing conditions, it is hypothesized that human performance with a CamTroller can be better than the professional player’s approach. Due to the difficulty of experimental control and performance measurement in a complicated playing situation, we did not achieve the data in such an environment. We would further enhance and evaluate people’s performance with CamTroller under more complicated playing conditions.

Gaming with CamTroller may also be beneficial for neck health. Long-time playing computer games with a keyboard and mouse may also cause health issues. Multiple research point out that the longtime usage of a keyboard and mouse can lead to discomfort, pain [7], and even injuries such as orthopedic [51] and tendinopathies [31]. A review has pointed out a negative impact of video game playtime on the musculoskeletal system in about 92% of studies [50]. Fine motor strain when performing repetitive actions is considered one reason behind the musculoskeletal hazards, while another is sitting for an extended period of time, often with poor posture. Hence, frequently stretching and conducting flexion, extension, and rotation of the cervical, thoracic, and lumbar spine are strongly recommended in Ahmed K. Emara et al.’s paper [6]. Therefore, it will be helpful if the players need to combine a motion controller for extra neck, arm, and body movement through a keyboard and mouse during the long period of game-playing.

6.2 The Generalizability of CamTroller for Other Games

Generally speaking, based on the design concept, CamTroller is applicable for all one-to-one avatar mapping PC games. For the specially designed CamTroller system for the PUBG game, a typical hardcore shooting game, we discussed here its generalizability to other games.

The easiest transplant games are other hardcore shooting games sharing similar avatar actions, like Arma 3, Insurgency, Escape from Tarkov, and the shooting controls in Cyberpunk 2077. As for other one-to-one avatar mapping PC games, for instance, in action role-play games, like Elden Ring, and Sekiro: Shadows Die Twice, dodging in different directions is commonly seen. CamTroller can be modified into the tool of trigger dodging to the direction in which the head is quickly moving. To be more specific, when the avatar needs to dodge right, the player can trigger it with a sudden move of the head towards the right. Also, using consumables is common in various types of games, and using gestures of hand to trigger the consumption will be easier and faster than traditional methods. Overall, among the recently released games, more of them are integrating modules that were previously available in different games. Just like Cyberpunk 2077, you can find features from traditional shooting games, action games, and role-play games. In other words, large-scale games are becoming freer and more complex. The gaming experience brought about by the complexity of this free combination of different game features is restricted by the limited interaction methods (keyboard and mouse) of computer games, so we introduce CamTroller to help lift this restriction.

6.3 The Comparison between CamTroller and Existing Alternatives

In different types of games, not only shooting games but also some action or role-playing games, developers have attempted to alleviate the high burden of remembering complicated key bindingsby indicating the keybindings on the UI and adding pop-up hints inside the games. For example, when the avatar is close to a door or a closet that can be opened, a tip will pop up to remind the player which key is for the corresponding action. Also, when the avatar is at low HP, a pop-up reminds the player which key is for consuming the healing items. This method has a good effect, except for the intense battle scenario. When engaging with enemies, players have no time to look at the indications on the UI to figure out which key should be pressed to perform a required action. Different from the layout and structure of buttons on the controller that can be easily distinguished from each other, keys on the keyboard have much higher similarity, making it extremely challenging to operate without looking at the keyboard especially for the players who are not proficient in touch-typing. Looking down to find the key in an intense battle will put the player in great danger since they lose the view, possibly leading to failure. Getting familiar with the keys and forming muscle memory to operate without looking down at the keyboard through repetitive practice is tedious and time-consuming. Compared to the current existing solution adopted by professional players, CamTroller is easier to learn and execute for novices.

Also, for the gamepad, which is typically used in action games. Gamepad does a good job of increasing the control ability of limited fingers. For example, only one thumb is enough to move the avatar in all four directions by the gamepad. In the keyboard, three fingers are needed for WASD control. However, as its control is not fast and accurate enough to aim a target by either the bottom or joystick compared to the mouse, the gamepad is not compatible with the shooting game. Besides, the gamepad occupies two hands most of the time, so once using the gamepad, players cannot use the keyboard and mouse simultaneously. They are forced to select one of them. CamTroller is an auxiliary tool. Although we only discuss it under the condition of gaming with keyboard and mouse input, it is possible to use the CamTroller to add more degree of freedom by head or some hand motions for the video games with gamepad input.

6.4 Limitations

While the CamTroller has demonstrated its potential to assist players in a more efficient and intuitive game experience, it still has certain limitations as a technical tool. One limitation is that the CamTroller operates as an open-loop control and does not receive any feedback from the gaming system. This means that it is unable to determine whether the motion is being initiated correctly or not. Sometimes the emulated input generated by CamTroller interferes with the input sent by the keyboard and gets ignored by the game. The approach proposed by Yu-Hsin Lin et al. for detecting video game events by monitoring the video, audio, and controller I/O [28] can be learned and migrated to the CamTroller to solve this problem. In addition, self- or auto-adjusted thresholds will be considered in further works.

Besides, for the validation part, more objective performance estimation and user experience questions could be involved. For example, the learning curve of learning using CamTroller and Pro actions; and the willingness to adopt CamTroller when they are playing games.