MouseRing: Always-available Touchpad Interaction with IMU Rings

2.1 Always-available Pointing Technique

The emergence of various always-available pointing techniques aims to strike a balance between convenience and performance to meet requirements in different scenarios, such as AR/VR and large displays. The most prevalent approach relies on cameras and CV algorithms to recognize finger-pointing[44, 57]. Finger-pointing in mid-air has already become the standard paradigm for AR/VR interaction[15]. To accommodate hands-free situations, head movement[13, 58] and eye-tracking[10, 23, 56, 61] for cursor control have also been proposed. Researchers also combined cameras with other sensors such as EMG[52], IMU[21], and touchpads[55] to improve input efficiency. Camera-based solutions offer high precision but require sensors deployed in the environment or on the headset. They also present privacy problems[22]. Consequently, researchers have begun exploring the use of other smart devices for cursor control tasks. The first category includes dedicated devices like laser pointers[25, 31] and remote controllers[3, 43], which offer good precision but are not easily portable for users.

The second category involves using daily-carry smart devices, such as smartphones[2, 5, 17, 26] and smartwatches[27, 28], to facilitate pointing input. Smart rings also fall into this category. Due to their small size and portability, smart rings can free users from handheld devices like controllers and environment-deployed sensors such as cameras, offering a universally applicable pointing technique. Additionally, smart rings stand out in certain scenarios compared to smartphones and smartwatches as they are always worn and do not need to be adjusted or taken from pockets. At the same time, we aim for the ring device to maintain high input efficiency comparable to traditional pointing input methods like touchpads and mice. In summary, our research on MouseRing aims at delivering an always-available pointing interface with greater convenience while maintaining satisfactory usability.

2.2 Ring-based Interactions

The primary usability goal of MouseRing is to enable efficient and accurate input on diverse surfaces while ensuring comfort and convenience through its always-available nature. To summarize the previous work, we have listed prior works that utilize ring-shaped devices for finger-tracking or target selection tasks in Table 1. We have outlined them based on three usability aspects: the comfort and portability of the setup, the interaction paradigm, and the sensing capability.

Finexus[9] and AuraRing[48] utilize multiple electromagnetic sensors to position the fingertip relative to the wrist nodes, thereby supporting in-air pointing for target selection. However, this necessitates wearing additional wristbands or watches and at least four sensors for positioning based on their relative distances. Finexus and AuraRing utilize an in-air pointing interaction paradigm, where users control the cursor by adjusting the orientation of their wrist or fingers. In contrast, MouseRing offers an interaction mode similar to traditional mouse devices, where users slide their fingers on flat surfaces. Both paradigms are easy to use, but prolonged in-air interaction may quickly lead to fatigue. Yuki Kubo[32] has implemented a touchpad-like interaction by installing a small pressure plate on the side of the ring. Magic Finger[59] achieves mouse-like functionality by using a camera on the finger pad to detect relative movement between the finger and the surface. While these interactions are highly intuitive, placing sensors on the finger pad or touchpads on the side of the finger significantly compromises comfort, impacting everyday activities.

IMU rings offer the advantage of being lighter, featuring a low-power, small-size setup, ensuring users can comfortably wear and use smart rings for extended periods. However, their sensing capabilities are somewhat limited. Despite considerable research implementing various gesture-based interactions[11, 16, 24, 39, 41] and Bayesian inference-based typing inputs[20, 30, 38] through IMU rings, there have been few efforts to track the fingertip for continuous 2D cursor control directly. LightRing[29] first proposed using an IMU to estimate lateral finger movements and an infrared sensor to perceive finger bending to estimate forward and backward movements. However, this approximation lacks precision and requires a complex calibration phase. Mouse on a Ring[62] does not directly track finger movement but controls cursor movement through an airmouse mechanism, using the tilt and acceleration changes of the IMU ring in the air. AnywhereTouch[47] uses the attitude angle of the fingertip IMU, calculates the speed of each finger joint through inverse kinematics, and achieves a 93% accuracy rate in uni-stroke three-classification tasks. This approach has been inspiring for our research approach to physics-informed machine learning. However, the finger-tracking results obtained directly from inverse kinematics are limited and do not support target selection tasks. Placing an IMU on the fingertip can also interfere with daily activities. Our work has systematically studied and modeled fingertip motion behavior, enhancing the fingertip tracking precision against prior work through the use of physics-informed machine learning. This has allowed us to realize the concept of precise touchpad interaction everywhere. Furthermore, the requirement of being always available also brings forth the usability demands of comfort and convenience. MouseRing meets these criteria by being lightweight, calibration-free, and robust to different surfaces, thereby ensuring a positive wearing and using experience in real-world scenarios.

Figure 2:

View Figure

2.3 Hand Modelling

In Human-Computer Interaction (HCI) and Computer Vision (CV), human hands are often modeled as articulated objects in three-dimensional space with interconnected joints[12, 49]. Kuch et al.[33] and Lee et al.[36, 37] have proposed simplified hand skeleton models with 26 and 27 degrees of freedom (DOF) that systematically describe the movements of individual joints. Many studies on gesture recognition[1, 50, 51, 54] followed their works and simplified models for specific tasks to achieve state-of-the-art performances. For example, Ahmad et al.[1] achieved 30 FPS gesture recognition using a 19-DOF simplified model. Spurr et al.[54] integrated hand constraints to achieve weakly-supervised gesture recognition.

We believe incorporating physical knowledge into the pipeline of sensing finger-sliding behaviors in MouseRing can also benefit tracking accuracy. However, several challenges exist. Firstly, the pressure between fingertips and surfaces can cause forced bending of finger joints and ligament deformation[34], rendering Lee’s constraints[36] for relaxed hands no longer valid. Secondly, the impact of skin deformation must be considered when we observe finger movements using an IMU ring fixed on the skin. Lastly, finger sliding only represents a small subspace of hand movement, potentially resulting in additional beneficial physical motion relationships between joints.

Our study builds upon previously proposed 3D hand models[37] and focuses on finger-sliding input tasks. We employ data-driven research methods to propose new motion relationships and disprove invalid physical constraints. The physical knowledge can provide assistance in developing stable and accurate MouseRing fingertip tracking algorithms.

3.1 Ring Number and Position

MouseRing has three ring-wearing configurations, including double-ring and two single-ring configurations. Under the double-ring configuration (Fig. 2(c)), users wear two rings with 6-axis IMU sensors on their index finger’s intermediate phalanx and proximal phalanx. Under the single-ring configuration (Fig. 2(a)-(b)), users wear a ring on either phalanx.

For the ring position, previous research[18] has shown that users prefer to use their index finger for touch input. Therefore, we chose to place rings on the index finger. Although placing the IMU sensor on the distal phalanx provides the richest information due to its proximity to the fingertip[53], it would significantly impact users’ daily activities. Therefore, we only consider the phalanxes, which are further back in position.

Regarding the number, placing more rings on one finger may help better estimate the angle between finger bones. We conducted a pilot study and interviewed 12 users to determine the maximum number of rings they could tolerate for an extended period. All users agreed that one ring was acceptable, and most users (9 in 12) considered that two rings were acceptable despite having a minor impact on daily activities. Most users rejected more rings due to interference with daily activities, comfort, and aesthetics. Given that the pilot study is preliminary and solely based on intuition, we will further evaluate the comfort level of wearing varying numbers of rings under real-world conditions.

3.2 Finger-sliding Mode

IMU rings predict finger movements by observing the local accelerations of the skeletal region where they are worn and establishing a relationship with the corresponding fingertip movements. Therefore, the prediction accuracy is greatly influenced by the chosen finger-sliding mode. It’s obvious that the level of hand restriction presents a trade-off between accuracy and comfort. Performing finger sliding when the entire hand is suspended in the air without any constraints (Fig. 2(g)) is quite effortless and free for users. However, at this point, IMU-based tracking is impractical in principle, because the uniform motion of the entire hand cannot be perceived by accelerometers or gyroscopes. Prior work has explored finger-sliding under the fixation of the middle finger and thumb (LightRing[29]) or the entire hand (Anywhere Touch[47]). In our work, we propose three different finger-sliding modes (Fig. 2(d)-(f)) with varying degrees of restriction, and systematically compare their algorithm tracking accuracy in subsequent research, aiming to find an interaction method that satisfies both accurate tracking and user-friendliness.

4.1 Apparatus

The experimental apparatus is shown in Fig. 3. Participants wore two IMU rings on their index finger’s intermediate and proximal phalange. Participants were instructed to wear the IMU sensors on the dorsal side of their fingers. In addition, four 2mm-diameter infrared reflective spheres were attached to the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), and nail tip of the index finger. The four markers were pre-labeled in the Optitrack system, allowing us to identify which joint each marker corresponds to. Participants were asked to slide their index fingers on a pressure touchpad, ensuring constant contact between the fingertip and the touchpad. The touchpad recorded the position of the contact point between the fingertip and the surface. During the experiment, we sampled the IMU’s acceleration and angular acceleration data, the position of the four key points of the finger tracked by Optitrack, and the pressure array data from the touchpad.

Each IMU sensor was a 9-axis accelerometer MPU9250. We recorded 6-axis acceleration and angular velocity data from the sensor and simultaneously logged the IMU’s attitude angles calculated in real time by the attitude estimation algorithm (section 6.2). We only used the six-axis data of acceleration and angular acceleration because we discovered that the uneven magnetic field variations in indoor spaces resulted in higher accuracy in 6-axis attitude estimation compared to the 9-axis (mentioned in section 6.2). The IMU frame rate was 200Hz. Each IMU was fixed on an adjustable iron ring and connected to an Arduino UNO via DuPont wires. During data collection, wired data transmission ensured high data quality. However, in real-world user experiments (section 9), we utilized a wireless transmission MouseRing prototype to ensure a more realistic user experience. The touchpad was the Morph Sensal touchpad, which can sense the pressure peak point of the touch and return the pressure force and peak point coordinates. The touchpad frame rate was 80Hz. We used eight OptiTrack Prime 13 motion capture cameras to capture the three-dimensional coordinates of the index finger’s joints and the nail tip. The camera array was placed 1-2m from the finger to ensure high-precision capture. The frame rate of OptiTrack Motive was 200Hz.

4.2 Participants

We recruited 12 participants (5 females, aged 20-26, M = 23.6) from the university campus. All participants were right-handed and used their right index fingers to input. The average length of index fingers was respectively 75.7mm(std=1.60) and 79.2 millimeters (std=2.79) for females and males, which were close to the existing literature[40]. We concluded that our participants’ finger lengths can represent the majority of users.

4.3 Design and Procedure

Data collection included 2postures*3modes = 6sessions. Participants were required to perform finger-sliding input on a horizontally positioned touchpad while sitting and a vertically positioned touchpad while standing. For each touchpad orientation, participants were required to perform data collection sessions using the three finger-sliding modes: Rested Wrist mode, Rested Thumb & Middle Finger mode, and Rested Palm mode. During each session, participants completed 20 different one-stroke finger-sliding movements.

Fig. 3(c) shows the 20 strokes. The 1^st-16^th one-stroke movements are straight lines in various directions. We divided 360° into 16 equal parts, and the angle between the i^th line and the positive x-axis was 22.5(i − 1)°. These data were used to simulate the user moving the cursor in various directions. The 17^th and 18^th one-stroke movements are clockwise/counter-clockwise circles, which simulate the user making small-radius turns and slight adjustments. For the 19^th and 20^th one-stroke movements, the participant’s index finger remained stationary, and these data were used as negative examples for ML learning. We encouraged participants to perform input with various initial postures to cover the entire posture space in our dataset.

In each session, assisted by paper-printed one-stroke images, the participants controlled their index fingers to slide 10 times along their imaginary direction. Before starting each one-stroke movement, participants needed to tap the touchpad twice quickly. These double taps were only used to align the data from different sensors. It is worth noting that the stroke directions on paper were only provided as a reference for the participants. During algorithm optimization, we used the actual trajectory captured by the pressure sensors and cameras as the ground truth. Each participant completed a total of 2(postures) × 3(modes) × 20(strokes) × 10(times) = 1200 finger-sliding strokes. They were allowed to rest for 5 minutes between sessions. The entire experiment lasted approximately 100 minutes.

4.4 Data Pre-processing

We first unified the data from all channels to 200Hz using linear interpolation. Due to the time delay in transmitting OptiTrack data, we developed a graphical interface to manually align the OptiTrack signal with the other two channels. The double taps made by the participants during the experiment left two peaks in the z-axis data from the accelerators and the OptiTrack. Two annotators independently adjusted the time deviation to align the signals, and the labeled time deviation differences between both annotators were less than or equal to 2 frames (≤ 0.01s). Each segment of aligned data has a recording duration of approximately 10 minutes (around 2000 frames). Each data segment includes about 200 instances of finger-sliding data, where the finger performs uni-stroke gestures on the touchpad.

4.5 Data Segmentation

5.1 Modeling Index Finger Movement

We simplified the physical model of the various bones and joints of the index finger using a kinematic chain(Fig. 4(a)), referencing modeling methods from computer vision[36, 37] and surgical medicine[34]. The three joints of the index finger (DIP, PIP, MP) connect the three bones of the finger (distal phalanx, intermediate phalanx, proximal phalanx) and the carpal bones of the hand. In addition, we also introduced the tip of the fingernail, as well as the contact point between the fingertip and the surface, because they are also crucial in finger sliding. The nail tip and DIP jointly model the vector corresponding to the distal phalanx, whose displacement is strongly correlated with the displacement of the contact point. The tactile sensation brought by the contact point and surface is the most intuitive way for users to perceive finger sliding.

5.2 Constraints on Joint Motion during Finger Sliding

Based on the aforementioned modeling, we propose six hypotheses regarding motion constraints, which are based on existing literature[37] and observations. To validate or falsify these hypotheses, we employed hypothesis testing on statistical measures and visualized the results in Figure 4(b)-(j). We found that several assertions regarding finger joint constraints in the natural state may not hold when the finger is in a tense state, as the joints and ligaments can be passively pulled. Conversely, due to the small-scale movements of the joints involved in finger sliding, complex hand mechanical movements can be approximated by simpler models in local motion spaces. We summarize the conclusions here, while the detailed data analysis is presented in Appendix A.

6.1 Overview

The overall goal of MouseRing is to achieve precise fingertip tracking through IMU sensing. We employ an intuitive approach that predicts the real-time velocity of the fingertip, accumulates these velocities, and updates the sliding trajectory in real-time. As shown in Fig. 5, we introduce the algorithm pipeline. We use the orientation of the IMUs worn on the intermediate and proximal phalanx of the index finger to represent the spatial orientation of the two finger phalanxes. We estimate each bone’s attitude by processing the continuously read acceleration and angular acceleration data streams. Since the accuracy of the attitude is crucial for calculating the ML model and physical constraints, we have carefully optimized the algorithm for hand interaction in attitude estimation. Next, we train an RNN-based model to learn from features such as finger skeleton attitudes and ring accelerations and to predict fingertip velocity. However, black-box probabilistic models suffer from unstable predictions and poor interpretability. Therefore, we establish several physical constraints based on the attitudes and velocity, judge the degree of compliance between the predicted instantaneous velocities and physical constraints, and correct the velocity. In addition, we also implement touch state detection and cursor smoothing to achieve a complete mouse-like target selection interaction experience.

6.2 Attitude Estimation

We placed an OptiTrack marker on an IMU sensor under rotational motion to collect ground truth data. We employed direct integration, standard 6-axis complementary filtering, and 9-axis complementary filtering methods to estimate the attitude, resulting in average errors of 6.63° , 3.58° , and 9.25°. Direct integration of angular accelerations led to significant attitude drift due to random environmental noise and sensor system bias. On the other hand, indoor environments exhibit pronounced and non-uniform variations in magnetic fields, with the orientation of the magnetic vector deviating up to 40° within a 1m × 1m area. This significant deviation severely affects the accuracy of magnetometer data. Therefore, considering the aforementioned challenges, we optimized our system within the framework of 6-axis attitude estimation.

We find that as a musculoskeletal movement mechanism, the movement of the index finger is smooth. There is no high-frequency component in its acceleration and angular acceleration. Thus, we recommend using a 1Hz-5Hz Butterworth bandpass filter to filter the acceleration signal and a 1Hz-10Hz Butterworth bandpass filter for the angular acceleration signal. After removing high-frequency noise, we compare the measured acceleration with the ground truth obtained by differentiating 3D positions. The error is reduced by over 20%.

Secondly, we apply a passive complementary filter method to calculate the attitude of the ring. Similar algorithms[14] were initially used for attitude estimation of large-scale, high-speed objects such as aircraft. We redesign the controller parameters for small-scale finger movements. We use the Mahony algorithm[14] as the framework. Its complementary filter algorithm can be regarded as a second-order control system with the characteristic polynomial S² + K_pS + K_i. K_p and K_i can be expressed as K_i = ω², K_p = 2ζω , where ω and ζ represent the cutoff frequency and damping coefficient. In this control system, the second-order control system has the best response when the damping ratio ζ = 0.707[60]. When ω is around 1 rad/s, the finger IMU’s attitude estimation has a faster response speed and more accurate results. Too small ω causes an attitude drift, while too large ω results in a significant fluctuation. Therefore, we set the parameters as ω =1.5rad/s and ζ =0.707. We use Euler angles (roll, pitch, and yaw) to measure the average error of the algorithm. The mean errors are 0.55° , 3.12° , and 1.65°.

6.3 Machine Learning for Speed Prediction

We design an RNN model to predict the index finger’s real-time velocities of the four key joint points (NailTip, DIP, PIP, and MP). We use the Nail Tip instead of the Contact Point of the fingertip as the optimization target (Conclusion 4 in Section 5.2). The model includes predictions of the MP joint for all input modes because its displacement cannot be ignored (Conclusion 5 in Section 5.2).

We select the filtered acceleration, the filtered angular acceleration, and the attitude obtained from the Mahony algorithm as the input features. We use quaternions Q as input features for the IMU attitude, which provides better prediction results than Euler angles and direction vectors. Instead of frame-to-frame prediction, we use the signals within a 20-frame (0.1s) time window. Thus, the model can learn both the current motion state and the recent motion trends. Depending on the number of rings, either or both of the ring information are utilized.

The model’s output predicts the real-time velocities of the index fingers’ four key points (Nail Tip, DIP, PIP, and MP). The projected velocity of the Nail Tip represents the user’s input. The velocity predictions of other key points can help correct the velocity of the Nail Tip in physical constraints. We average the velocities of the last five frames to smooth the displacement jumps caused by Optitrack cameras.

Our model consists of a single-layer LSTM with a hidden state size of 32 and two linear layers, followed by RELU as the activation function before each linear layer. Let y be the velocity vector of the key joints. We design our loss function as: \(L = 0.2*(1 - Cos\_Similarity(y_{pred}, y_{true})) + 0.8*MSE({y_{pred}},{y_{true}})\) to increase the weight of the accuracy of velocity direction prediction relative to the accuracy of velocity magnitude prediction because, in target selection tasks, people are more sensitive to inconsistent directions. We implement the above model in Python based on the PyTorch framework, with a batch size of 32, and train the model to the best using cross-validation.

6.4 Physics-constrained velocity correction

The predicted velocities of key points from the ML model are independent. The independency leads to inconsistency in velocities among key points. The predictions between consecutive frames are also independent, which leads to instability in the predicted velocity.

The attitude of the skeletons is a slowly changing and relatively stable quantity. Using it to establish physical constraint relationships can help establish connections between predicted velocities of different joints among frames. Our idea of velocity correction is to quantify the degree of conformity between the current predicted velocity and the physical constraints with a confidence score. The confidence score serves as a weight to update the correction value of the current velocity. The current velocity combines with the historical one to achieve smoother and more stable velocity prediction. We find the following physics constraints to be effective.

Directional consistency: Confidence score C₁ characterizes the consistency of the velocity directions of different joints(Conclusion 6 in 5.2.6). If the consistency is greater than a threshold, the confidence score of the predicted value will be reduced. \(\begin{equation*} \left\lbrace \begin{array}{llll}\alpha _1 = \arccos \langle \mathbf {\hat{v}_{nailtip\parallel A}}, \mathbf {\hat{v}_{DIP_n\parallel A}}\rangle \\ \alpha _2 = \arccos \langle \mathbf {\hat{v}_{DIP\parallel A}}, \mathbf {\hat{v}_{PIP_n\parallel A}}\rangle \\ \alpha _3 = \arccos \langle \mathbf {\hat{v}_{nailtip\parallel A}}, \mathbf {\hat{v}_{PIP_n\parallel A}}\rangle \\ C_1 = \frac{1}{3}\sum _{i=1}^3\mathbb {I}(\alpha _i \le \alpha _{i_{thres}}) +\frac{1}{3}\sum _{i=1}^3\mathbb {I}(\alpha _i \gt \alpha _{i_{thres}})\cos ^2(\alpha _i) \end{array}\right. \end{equation*} \) ⟨v₁, v₂⟩ represents the dot product. Hat denotes the unit vector. A is the given horizontal/vertical plane where the index finger is sliding. \(\hat{v}_{Joint_n\parallel A}\) is the projection of the velocity vector on plane A. The values of \(\alpha _{i_{thres}}\) are also referenced from 5.2.6. \(\alpha _{1_{thres}}=13^\circ\) \(\alpha _{2_{thres}}=15^\circ\) \(\alpha _{3_{thres}}=30^\circ\)

Co-planarity: Confidence score C₂ characterizes co-planarity. The four key points (NailTip, DIP, PIP, and MP) are always on the same plane (Conclusion 2 in 5.2.2). Therefore, their instantaneous normal velocity vectors relative to their common plane are also coplanar. We have: \(\begin{equation*} C_2 = (\mathbf {\hat{v}_{DIP_n}} - \mathbf {\hat{v}_{PIP_n}})\times (\mathbf {\hat{v}_{PIP_n}} - \mathbf {\hat{v}_{MP_n}}) \cdot (\mathbf {\hat{v}_{nailtip_n}} - \mathbf {\hat{v}_{DIP_n}}) \end{equation*} \) Hat denotes the unit vector. The subscript “n” represents the component of the vector in the normal direction.

Length consistency: Confidence score C3 characterizes length consistency. The points on the same bone have equal instantaneous radial velocities due to the rigid body constraint (Conclusion 3 in 5.2.3). \(\begin{equation*} C_3 = min (\frac{\mathbf {v_{MP\parallel L_1}}}{2\mathbf {v_{DIP_\parallel L_1}}},\frac{\mathbf {v_{DIP\parallel L_1}}}{2\mathbf {v_{MP_\parallel L_1}}}) + min (\frac{\mathbf {v_{DIP\parallel L_2}}}{2\mathbf {v_{PIP_\parallel L_2}}},\frac{\mathbf {v_{PIP\parallel L_2}}}{2\mathbf {v_{DIP_\parallel L_2}}}) \end{equation*} \)

L1 and L2 are the directional vectors of the intermediate and proximal phalanx. They are calculated from the attitude angles of the key points in the attitude estimation.

The overall confidence score, denoted as C, is calculated as the product of individual constraint confidence scores C_i. As different constraint equations have varying refinement effects on the accuracy of velocity estimation, we introduce an exponent parameter for each C_i, which is explored by traversing all possible values to achieve the global optimum for velocity prediction. The final confidence score and velocity correction formula are as follows. \(\tilde{v}^t\) is the model’s predicted value at time t, and v^t is the correction value. \(\begin{equation*} \left\lbrace \begin{array}{ll}C = C_1^3 \cdot C_2^2 \cdot C_3 \\ \mathbf {\tilde{v}_{nailtip}^t} = C \cdot \mathbf {v_{nailtip}^t} + (1-C)\cdot \cdot \mathbf {\tilde{v}_{nailtip}^{t-1}} \end{array}\right. \end{equation*} \)

6.5 Touch State Detection

To enable tracking the sliding of the finger on the physical surface, the system needs to detect in real-time whether the user’s index finger is in contact with the surface to exclude the case where the finger is hovering in the air. We refer to previous works[18], which achieved a 99% accuracy in touch-down event detection. For each axis of the 6-axis IMU, we computed the maximum, minimum, mean, skewness, and kurtosis within a 10-frame time window. These features, totaling 30 dimensions, were used to classify the data from each time window into four categories: touch-down, touch-up, in-air, and touching, using an SVM model.

We then employed a state machine to detect the touch state in real-time. When the touch state is false, if the touch detector detects a touch-down event or continuous touching for five consecutive time windows (with 80% overlap between adjacent windows), the touch state transitions to true. Conversely, when the touch state is true, if the touch detector detects a touch-up event or continuous in-air state for five consecutive time windows, the touch state transitions to false. We conducted tests on our dataset. 95.5% of finger-sliding interactions were accurately identified in their entirety. When participants’ fingers continuously interacted during the user experiment, only 11.8 seconds per hour of fingertip in-air state were mistakenly recognized as touch state. The algorithm demonstrated robust performance as a switch for touch input.

6.6 Scaling the Velocities in Different Directions

For right-handed users, sliding towards the lower left and upper right is more effortless than sliding towards the upper left or lower right because the natural rotation of the right hand around the wrist causes the fingers to move in these two directions. To address this issue, we amplify the speed amplitude of the more difficult sliding directions. With this optimization, users have a similar subjective sliding experience when moving the cursor in all directions.

6.7 Cursor Filtering

We further implement the 1€filter[8] for the corrected velocity, which performs well in smoothing mouse input. The filter has two parameters: the minimum cutoff frequency \(f_{C_{min}}\) and the speed coefficient β. Reducing the minimum cutoff frequency will reduce slow speed jitter, while increasing the speed coefficient will reduce speed lag. We select \(f_{C_{min}}=0.004\) and β = 0.08 to filter the corrected mouse displacement in the model calculation.

7.1 Simulation Set-up

We utilized data from all 12 participants, trained the model using the leave-one-out cross-validation method, and predicted the velocity within each time window for each user during the uni-stroke process of finger-sliding. By integrating the velocities, we simulated the predicted fingertip sliding trajectory. This section compared the predicted trajectories against the ground truth trajectories from Optitrack using the following five metrics in Figure 6:

Table 2 presents the simulation results obtained under different datasets, ring configurations, and model settings. For the MouseRing algorithm, we separately trained the models using three different finger-sliding modes’ data. For simulation groups without annotated datasets, we report the average error of the three action modes (RW, RTM, RP). We also compared the effect of ring position and quantity on tracking accuracy by training the models using single-ring and dual-ring data.

As a baseline algorithm, we replicated the kinetic-based model from AnywhereTouch[47], an existing work that supports finger tracking based on finger-worn IMUs. It uses the change in pitch angle and the relation formula \(\theta _{DIP}= \frac{2}{3}\theta _{PIP}\) to estimate forward and backward displacement. It predicts the left and right movement by mapping changes in yaw angle to the displacement. Furthermore, we conducted a simple ablation study where we removed the physical-constrained correction from the MouseRing algorithm and evaluated the performance of the end-to-end RNN. Additionally, during the training process of the RNN model, we selectively masked the components of each axis of the 6-axis IMU to assess the utility of each axis.

7.2 Finger-sliding Modes

We separately trained models for the three sliding modes (Rested Wrist, Rested Thumb & Middle Finger, and Rested Palm). We initially expected that the RP and RTM modes, which have stronger hand constraints and fewer degrees of freedom, would be more accurately predicted. However, the simulation results only partially met our expectations.

Considering both the wearing & input experience and fingertip tracking accuracy, RW (Rested Wrist) is the best input mode for overall interaction. Among the three modes, the RP mode has the smallest θ_lerror of 5.34° , while RW and RTM have similar θ_lerror of 7.51° and 7.36°. On the one hand, fixing the entire palm on the surface does make the prediction more accurate. However, the improvement in accuracy is less significant compared to the loss of interaction comfort. On the other hand, while placing the thumb and index finger on the surface causes less loss of interaction comfort, it does not provide much help to the model’s prediction. We conclude that the RW mode allows for natural input and maintains accuracy similar to the other two input modes.

7.3 Ring Number and Position

The accuracy of the single-ring configurations is lower than the double-ring configuration, with displacement angle errors of 13.23° and 12.33° , respectively, compared to the 6.61° of the double-ring sensor configuration. For the proximal phalanx ring, y_error increased significantly from 12% to 22.3%, resulting in a l_error of 24.01%. This can be attributed to the fact that, due to its longer distance from the fingertip, the proximal ring exhibits smaller variations in orientation when the fingertip moves forward or backward. Thus, the information from the IMU is not sufficient to predict the y-direction displacement.

In conclusion, for the single-ring configuration, it is more suitable to wear the ring on the intermediate phalanx. Despite the decreased accuracy, we argue the accuracy is enough for non-fine-grained target selection. In daily-life scenarios, the single-ring configurations are more comfortable to use due to their lighter wear.

7.4 Machine Learning vs. Kinematic-based Modelling

The kinematic-based method has a significant systematic bias in the prediction direction for two reasons. Firstly, the \(\theta _{DIP}= \frac{2}{3}\theta _{PIP}\) relationship does not hold when the finger slides, leading to a highly inaccurate forward and backward displacement prediction. Secondly, differences in the wearing position of the ring, finger length, and finger sliding habits among different users can result in significant prediction errors during angle mapping. RNN model can learn the sliding action mode well, achieving a θ_lerror of 8.75° and a l_error of 13.94%. However, its θ_verror reaches 32.8° , indicating that the instantaneous speed is very inaccurate on occasion. After incorporating the physical constraints into the MouseRing algorithm, the physical constraints reject inferior speed prediction. The θ_verror is significantly reduced to 14.53°. The θ_lerror also benefits from more stable velocity prediction. The general results demonstrate that physics-based knowledge can assist machine learning, making finger-sliding predictions more stable and accurate.

By comparing models with and without removing 6-axis features, we also analyzed the useful information provided by each axis of the IMU sensor. a_x and ω_z contribute significantly to the prediction of displacement in the x-direction. We attribute this to the integration of x-axis acceleration related to x-axis displacement. Also, the z-axis angular velocity strongly correlates with fingertip movement when the palm rotates around the wrist. After removing these features from the model, the error in x-axis displacement increased from 6.80% to 8.01% and 11.70%.

Surprisingly, for the prediction of y-axis displacement, the acceleration in the y-direction is not the most important. Instead, a_z, ω_x, and ω_z contribute more. These features determine the angle between the index finger phalanxes and their respective postures. This indicates that the y-axis displacement of the fingertip is mainly influenced by finger bending and changes in hand posture rather than the translational motion of the index finger. Even though removing information from other axes did not significantly increase the errors in the x or y-directions, θ_lerror still increased significantly. All 6-axis data is helpful for accurately predicting the velocity.

7.5 Summary

For RQ1, we find that the prediction accuracy of different motion modes is similar, thus rejecting the previous hypothesis. RW (Rested Wrist) is an input mode that balances free interactive motion and good accuracy. For RQ2, while the double-ring configuration (θ_lerror=6.61°) can achieve higher prediction accuracy, the single-ring configuration still has considerable prediction performance (θ_lerror=12.33°). It potentially supports simple cursor control tasks in mobile scenarios. For RQ3, a pure RNN can learn the pattern of finger motion well, while physical knowledge connects the estimated finger attitude with velocity predictions and refines the prediction results. The combination of the two can achieve high-precision and stable trajectory prediction.

8.1 Input Methods

We compared three input methods. In addition to MouseRing, we chose two target selection methods based on the mouse selection paradigm as baselines. For MouseRing, we tested both double-ring and single-ring setups.

Figure 7:

View Figure

8.2 Apparatus

We ran the Fitts Law study’s JavaScript program on a Dell G3 laptop. The laptop’s touchpad was used as the TouchPad baseline. We used the Deli 2803 flying mouse as the AirMouse input device. For MouseRing, participants wore the same prototype device as in the data collection section. We removed the display-to-control ratio of the Windows system by reverse-engineering[7]. In this way, we eliminated the potential impact of the display-to-control ratio on different input methods.

8.3 Participants

We recruited 12 participants (6 females, aged 20 to 25, M = 24.0) from the campus. All participants were right-handed. All participants were very familiar with TouchPad input. Four participants had previous experience with Air Mouse. None had used MouseRing before.

8.4 Design and Procedure

The experiment was based on a Fitts’ Law target selection GUI (Fig. 7(b)). Each time, two yellow buttons, S (Start) and E (End), appeared on the screen. The buttons were randomly generated with diameters ranging from 12mm to 30mm and distances ranging from 90mm to 400mm. Participants first moved the cursor to the Start button. After 1 second, the Start button turned green. They then moved the cursor to the End button. After a 250 ms pause, the End button turned green, indicating the completion of a target selection. The time interval between the color changes of the two buttons was recorded as the selection time.

Each participant had to complete the task under four different input methods or configurations. Participants had to complete 5 rounds * 10 times/round = 50 selections for each input method. Before using each input method, participants could practice freely until they felt they had mastered the input method. Each participant performed the tasks in a different order to eliminate the effects of fatigue and learning.

Figure 8:

View Figure

8.5 Results

We used linear regression in the Fitts’ Law study to fit the relationship between the average selection time and index of difficulty (\(ID = log_2 \frac{2D}{W}\)). We ran one-way RM-ANOVA and Friedman tests for different input methods to test the significance of differences between the average time.

The Friedman test revealed significant differences in the mean selection time between different input methods (χ² = 26.51, p < 0.001). The input efficiency of dual MouseRing (MT = 658.1ms, STD = 45.1ms) was only slightly slower than TouchPad (MT = 629.1ms, STD = 41.5ms). In contrast, AirMouse and single MouseRing were significantly slower than TouchPad (F = 5.9, 12.2, 15.0, p < 0.05). We found that the double-ring configuration of MouseRing was a fast, anywhere-available input method compared to AirMouse.

Furthermore, we fitted the relationship between MeanTime and ID, where the slope of the line represents the cursor movement rate, and the intercept reflects the target-locking speed (Fig. 8(b)). TouchPad (k = 83.47) had the fastest cursor movement speed, followed closely by MouseRing (k = 94.07) and AirMouse (k = 93.54). The model’s accuracy decreased in the single-ring configuration and decreased cursor movement speed. AirMouse had the largest intercept because the participant’s suspended hand was prone to shaking, making target locking difficult.

For RQ4, we found that the MouseRing with a double-ring configuration can achieve comparable input speed (629 ms vs. 658 ms) to the TouchPad and is faster than the anywhere-available baseline (AirMouse). With a more lightweight wearing experience, as a tradeoff, the MouseRing in the single-ring configuration exhibits approximately 20% higher completion times compared to the touchpad. Participants reported that target selection becomes challenging when the targets are small. Therefore, in real-world tasks, it is necessary to investigate how much MouseRing can support fine-grained target selection tasks in the single-ring configuration.

9.1 Setup

MouseRing has the potential advantage of providing always-available interaction, so we asked participants to complete target selection tasks in both standing and sitting postures on different surfaces. We evaluated the usability of MouseRing in both dual-ring and single-ring configurations on four different surfaces. The desktop, sofa, wall, and thigh surfaces cover different plane orientations, hardness, and flatness levels. The desktop and wall are hard and flat surfaces, while the sofa and thigh are soft and uneven. Participants interacted with the desktop and sofa (horizontal surfaces) while sitting and with the wall and thigh (vertical surfaces) while standing.

We chose the mouse and AirMouse as baselines, with the mouse replacing the touchpad used in the lab-condition study, as it would be difficult for participants to constantly hold a touchpad while standing. Participants used the mouse on a desktop and wall surface when sitting and standing, respectively. We conducted 12 within-subject studies (2 Mouse + 2 AirMouse + 4 Dual-Ring + 4 Single-Ring), with the input method, posture, input surface, and ring number as factors.

9.2 Apparatus

We used the Samsung UA65JU5900JXXZ as the large-screen device. The screen size is 65 inches, with participants inputting from a distance of 3-5m. The user experiment script and GUI were run on a laptop and projected onto the screen via an HDMI cable. We used the Logitech M186 as the mouse device and the Deli 2803 flying mouse as the AirMouse input device.

We integrated the Bluetooth and IMU sensor modules into a small wireless ring to provide a more realistic wearing and usage experience closer to real-life scenarios (Figure 9). This ring has the same sensing modules as the previous prototype device and sends data to the computer via Bluetooth at 200 Hz. A more detailed design of the wireless ring is presented in Appendix B.

9.3 Participants

We recruited 12 participants from the campus (7 females, aged 18 to 25, M=22.4). All participants were right-handed. All participants were very familiar with the mouse device. None had previous experience with either the AirMouse or MouseRing before.

Figure 9:

View Figure

9.4 Design & Procedure

We designed a user experiment for large-screen device interaction. Participants played the role of a museum guide and used the large-screen device to introduce exhibits to visitors. Participants were required to sequentially move the mouse and select buttons on the large screen, controlling the detailed descriptions of the exhibits to pop up individually and then read them aloud. Afterward, the participants clicked the page-turning button and continued introducing the following exhibit. Each page contained five buttons that could trigger events. The size and position of the buttons on the page were designed in advance to cover different sizes and distances. We provided the participants with a script to guide them on the order of clicking buttons and reading text. Screenshots of the interactive pages are presented in Figure 13 in Appendix C.

Before the experiment began, participants had 5 minutes to learn and familiarize themselves with controlling the cursor using the AirMouse and MouseRing. Each participant needed to complete one round of the experiment under 12 sets of settings (2 Mouse + 2 AirMouse + 4 Dual-MouseRing + 4 Single-MouseRing). Under each setting, participants completed 10 pages * 5 times/page = 50 selections. We recorded the time taken for each target selection, as well as the distance to and size of the target. Each participant completed the tasks in a different order to eliminate the effects of fatigue and learning. After the experiment, participants filled out a subjective questionnaire and briefly talked about their feelings. The experiment lasted approximately two hours.

Figure 10:

View Figure

9.5 Results

We ran one-way RM-ANOVA and Friedman tests for different input settings to test the significance of differences between the average time. The significance of subjective ratings was tested using the Mann-Whitney-Wilcoxon rank test.

10.1 AR/VR Input

In scenarios involving visual occlusion and mobile interactions, MouseRing facilitates user manipulation of graphical interfaces in AR/VR environments. Unlike hand-held controllers, MouseRing is considerably smaller and can be worn daily. Additionally, MouseRing does not rely on HMDs to be equipped with cameras, offering a low-power sensing solution that can potentially reduce the weight and size of future HMDs. We have implemented two VR applications: a voice and video calling application (Fig 11(a)) and a video player (Fig 11(b)). These applications utilize touchpad interactions supported by MouseRing to enable convenient cursor control in VR for button selections.

Figure 11:

View Figure

10.2 Large screen Displays

In conference and smart home scenarios, MouseRing can serve as a substitute for remote controllers and air mice to control projectors, smart TVs, and large-screen displays efficiently. We have implemented a series of cursor interactions and shortcut commands based on continuous mouse control (Fig 11(c)). Speakers can utilize MouseRing to highlight key points in presentation slides and switch content, even while standing or walking away from the screen.

10.3 Mobile and Sports Scenarios

In mobile or sports scenarios, we designed FaST Sliders[42], enabling MouseRing to support sending shortcut commands by sliding in different directions on any surface. During sports, users are constrained in body posture and input capabilities. In commuting scenarios, MouseRing facilitates subtle and rapid input. MouseRing imposes minimal physical effort demands and liberates users from needing additional devices. Based on MouseRing’s FaST Sliders interaction, we have implemented the control of a music player and remote control functionality for a fitness instructional video app on a tablet (Fig 11(d),(e)).

11.1 Stronger Sensing Capability with Physically-informed Models

Utilizing IMU sensors for body tracking is challenging, not solely due to sensor noise leading to inaccurate pose estimation but also because sparse pose information is inadequate to recover the full spectrum of body movements. In our work, additional prior information can benefit finger tracking by providing helpful information gain in hand kinetics for ML models. We contend that the inherent knowledge within the physical structure and movement patterns of the human body can be modeled as priors, assisting in achieving robust perception from weak sensor signals. Similar body constraint modeling has already been employed in full-body pose estimation studies. We hope our approach can inspire more work in HCI to develop novel motion and behavior recognition techniques.

11.2 Towards Personalized Online Calibration

Although there are many common physical laws governing the movement of index fingers, the differences in finger length ratios and movement styles among individuals are difficult to avoid. Setting hyperparameters in the model to represent these individual differences and dynamically learning these parameters during user use can effectively improve the recognition accuracy of MouseRing. One possible approach is to fit the predicted fingertip trajectory of the model and the line between the initial cursor position and the user’s following selected target (ground truth of trajectory). Compared to a series of calibrations in advance, online personalized calibration is iterated in the background without occupying additional user attention and achieves better results with the larger amount of online data.

11.3 Sensing Ability

Due to the interference of indoor magnetic fields, we utilized signals from a 6-axis IMU sensor and abandoned potentially helpful information from the magnetometer. This approach was feasible for our research, as each user experiment lasted only about 10 minutes. After that, the initial attitude of the IMU was recalibrated to eliminate the impact of attitude angle drift. For long-term continuous use, the MouseRing algorithm also needs to be improved. Magnetometer information can complement accelerometer information in a clean outdoor space. For indoor circumstances, prior estimation of spatial magnetic fields could enhance the sensing capability of MouseRing. In the experiment, signals were collected at a frequency of 200 Hz. This frequency is sufficient for fingertip motion tracking tasks. However, a higher sampling frequency[35] may improve the accuracy of touch-down and touch-up event detection, as these events utilize the frequency domain characteristics of the signal. Nevertheless, there is a trade-off between the ring device’s sensing capability and power consumption.

11.4 Long-term Wearing & Remounting

Although a single-ring setup can support simple input interactions, higher precision and accuracy of MouseRing sensing require wearing two rings. The long-term wearing of a single ring on the intermediate phalanx may cause slight discomfort. One possible solution is to wear the intermediate phalanx ring on the base of the other fingers for improved comfort during daily wear. The MouseRing requires users to position the IMU sensor on the backside of the finger when wearing the ring. Still, we did not explicitly require precise angle alignment during data collection or user evaluations. Participants also did not report any noticeable impact on accuracy due to the remounting process, which assures position adjustments while using MouseRing. The wearing status can be recognized by calculating the relative posture between the rings. Various dual-ring wearing methods among different fingers can potentially provide independent and richer input methods.