iPose: Interactive Human Pose Reconstruction from Video

5.1 Camera

When a user modifies the camera via the bounding box (Figure 3(a)), the corresponding operation produces a displacement vector (Δx, Δy) in the screen space (Figure 3(b)(c)). Δx and Δy produce new camera parameters \((t^{\prime }_{x}, s^{\prime }_{x})\) and \((t^{\prime }_{y}, s^{\prime }_{y})\) along both dimensions independently. Without loss of generality, we give the deduction of x dimension of the left bottom control point. The deduction processes for the y dimension and other control points on the bounding box are similar, and thus omitted here.

Dragging. When a user drags the bounding box (Figure 3(b)), Δx only modifies the translation parameter t_x. Specifically, given the weak perspective projection x_i = [(X_i + t_x) · s_x + 1] · w/2, where w is the width of the video frame, the new projection after dragging is: \(x_i+\Delta x=[(X_i+t^{\prime }_{x})\cdot s_x + 1] \cdot w/2\). Jointly solving these two equations gives the new translation parameter \(t^{\prime }_{x}\): (1) \(\begin{equation} t^{\prime }_{x}=t_x + \frac{2}{w} \times \frac{\Delta x}{s_x} \end{equation} \)

Scaling. When a user scales the bounding box by dragging a control point (Figure 3(c)), Δx modifies both the translation parameter t_x and scale parameter s_x. According to the weak perspective projection, the projection of two points X1 and X2 after scaling in the case of Figure 3(c) is \(x_1 + \Delta x=[(X_1 + t^{\prime }_{x}) \cdot s^{\prime }_{x} + 1] \cdot w/2\) and \(x_2=[(X_2 + t^{\prime }_{x}) \cdot s^{\prime }_{x} + 1] \cdot w/2\), respectively. Jointly solving these two equations gives the new scale parameter \(s^{\prime }_{x}\): (2) \(\begin{equation} s^{\prime }_{x}=\frac{x_2 - x^{\prime }_{1}}{x_2-x_1} \cdot s_x \end{equation} \) Denote the coefficient in Equation (2) \(\frac{x_2 - x^{\prime }_{1}}{x_2-x_1}\) as r. Then according to the equivalence of a random point X_i:  \(x_2 + r\cdot (x_i-x_2)=[(X_i + t^{\prime }_{x}) \cdot s^{\prime }_{x} + 1] \cdot w/2\), the new translation parameter is: \(\begin{equation*} t^{\prime }_{x} = t_x + \frac{1}{s_x} + \frac{(1-r)\cdot x_2}{r\cdot s_x\cdot \frac{w}{2}} - \frac{1}{r\cdot s_x} \end{equation*} \)

5.2 Pose

In the pose mode, the change in 2D position (Δx, Δy) of the i-th joint is mapped to its change in 3D rotation Δθ. Specifically, as shown in Figure 3(d), the new position of a joint in the screen space \((x^{\prime }_{i}, y^{\prime }_{i})\) is first inversely projected into \((X^{\prime }_{i}, Y^{\prime }_{i}, Z^{\prime }_{i})\) in the 3D world space under joint kinematic constraints. Then the angle between the vectors of the old and new joint positions with the parent joint position, along with their normal directions, formed the incremental axis-angle 3D rotation Δθ. To model kinematic constraints, we categorize the joints on the SMPL model into the following four types according to mobility (see Figure 5(a)).

Figure 5:

View Figure

Fixed-based joints are those considered less movable due to their stability requirements, e.g., shoulder collars and hips. They can be indirectly manipulated through global translations or orientations of the torso, which helps in maintaining overall rigidity.

First-level joints are joints that directly connect to fixed-based joints, e.g., elbows, knees, and chest. A fixed-based joint, a first-level joint, and a second-level joint form a two-bone structure (Figure 5(b)), connected with two bone segments. When a user drags a first-level joint handle on the video frame, the positions of the fixed-based and second-level joints remain unchanged. The trajectory of the first-level joint in 3D is thus constrained by a 3D ring to maintain constant bone lengths.

Second-level joints are those connected to a fixed-based joint via first-level joints, e.g., the wrist, ankle, and neck. Their position changes often largely influence the kinematic chain. When a user specifies the target position of a second-level joint on the video frame, its target position in 3D is determined by inverse projecting the 2D position into a minimum skew viewplane [11](Figure 5(d)). Then the 3D positions of the first-level and second-level joints are both updated according to the two-bone IK algorithm [9] (Figure 5(e)).

End-effectors are joints whose movements are relatively more independent and do not largely propagate back along kinematics chains, such as head, hand, and foot. End-effectors connect to their parent joints with a single bone segment (Figure 5(c)) and are thus constrained by a 3D sphere to maintain constant bone length.

There still exist ambiguities despite the above-mentioned geometry constraints. Specifically, for both the first-level joints and end-effectors, the 3D joint can move towards two possible directions along the 3D ring or sphere [9]. We address this ambiguity with a heuristic that a user’s manual intervention fine-tunes the initial pose and should choose a side that retains the adduction or abduction movement tendency. The algorithm thus checks the relative depth between the manipulated joint and its parent joint in the initial pose, and selects a side that makes the absolute relative depth increase. In case the heuristic fails in specific scenarios, the user can seek to the traditional 3D rotation widget (Figure 2(d)) to fix the errors. For the second-level joints, an inherent ambiguity for two-bone IK is the uncertain orientation of the first-level joint during the update. We empirically set the forward orientation as the transformed forward axis of a first-level joint under T-pose. A user can overwrite the forward orientation via the first-level joint handle, which explicitly specifies the orientation of the two-bone kinematics chain.

Figure 6:

View Figure

5.3 Pose Refinement

This section introduces the method for using video frame constraints to automatically refine the 3D pose. To maintain controllability, the refinement is conducted locally at body-part level instead of globally to the whole body pose. The main idea is to first find the transformation via the pixel-level mismatch between the projection of a body part and its segmentation in the background video frame, and then use the transformation to refine the foreground 3D pose.

We first use the projected foreground body parts to parse video frames. As shown in Figure 6(a), the projection of the 3D vertices of the body part (the orange point clouds) is used as an initial hint to retrieve the segmentation of the body part in the background video frame (the blue and cyan masks). We implement the video frame body part segmentation with Segment Anything [23], using the bounding box of projected vertices and the 2D position of the bone center as prompts. However, the shapes of masks can be noisy due to the influence of clothes and uncertain body part boundaries, and thus they do not perfectly match the shapes of projected vertices (Figure 6(b)). We thus seek to use a robust alignment method to find an optimal alignment that best snaps the body part projection to the mask. Specifically, the projected body part vertices and the segmentations are both converted to contour representations (Figure 6(b)(d)). Then we run a RANSAC [15] on both contours to compute robust alignment with inliers identified by the algorithm. The results are the optimal 2D translation and rotation, which are applied to the projected joint positions to produce two displacement vectors at both the parent joint and the user-specified joint (Figure 6(c)(e)).

Then the displacements are used as constraints from video frames to update the pose (Figure 6(f)(g)) and meanwhile satisfying the kinematic constraints using the same method as in Section 5.2. The only difference is that the 2D displacements are automatically computed instead of specified by a user. Note that the pose refinement process includes the video frame segmentation and the optimal alignment that are not fully controllable. In case the algorithm produces errors, a user can decline the snapping result via the undo button (Figure 2(a)) and disable the automatic snapping (Figure 2(f)) to retain the manual correction results.

5.4 Temporal Propagation

The goal of temporal propagation is to automatically update future τ frames upon a user’s specification at frame t_f by using the user’s specifications and the temporal consistency of video frames. Since the errors in the camera and pose parameters do not follow temporal consistency and are random in future frames, we formulate temporal propagation as an optimization problem that allows constraints to influence each other. We use user-specified 2D joint handles \((x_{t_f}, y_{t_f})\) as prompts to predict the landmark positions in the future frames (x_p, y_p), p ∈ [t_f + 1, t_f + τ] using a point tracking method (coTracker [22] in our implementation). Denote the new pose as \(\theta ^{\prime }_{p} = \theta _p+\Delta \theta _p\). The objective is to find pose corrections Δθ_p, p ∈ [t_f + 1, t_f + τ] and camera corrections (Δs_p, Δt_p), p ∈ [t_f + 1, t_f + τ] in future frames, so that the projections of the 3D joints under new parameters best fit (x_p, y_p). The objective function also includes a temporal smoothness term on joint rotations, and constraints on the magnitude of Δs and Δt: (3) \(\begin{equation} \begin{split} \mathop {\arg \min }_{\Delta s,\Delta t,\Delta \theta } & \sum _{p=t_f+1}^{t_f+\tau } \sum _{i=1}^{n} \Vert \Pi \left(J\left[M(\beta ,\theta ^{\prime }_{p})\right]_i\right)-(x_{p,i},y_{p,i})\Vert ^2 \\ & + \lambda _{smooth} \sum _{p=t_f+1}^{t_f+\tau } \sum _{i=1}^{n} \Vert \theta ^{\prime }_{p,i} - \theta ^{\prime }_{p+1,i}\Vert ^2 \\ & + \lambda _{s} \sum _{p=t_f+1}^{t_f+\tau } \Vert \Delta s_p\Vert ^2 + \lambda _{t} \sum _{p=t_f+1}^{t_f+\tau } \Vert \Delta t_p\Vert ^2\\ \end{split} \end{equation} \)

Both joint position tracking and the optimization can also produce errors, where users can intervene manually again at specific errors.

6.1 User Study

Procedure. We recruited 8 participants (A1 ∼ A8, aged 21 ∼ 33, three female). A1 and A3 had experience in 3D modeling, A2 had background in monocular human pose reconstruction, A4, A7 and A8 had background in pose analysis, and A5 and A6 were novice in all of these fields. In order to quantitatively measure the human pose reconstruction accuracy by iPose, we use the test set of the 3DPW dataset [38], which provides ground-truth 3D poses for 24 in-the-wild videos, with duration ranging from 387 to 1,824 frames. See Section B.1 for details on dataset processing. We obtained the initial parameters with a state-of-the-art automatic human pose reconstruction method (ROMP [32]) and let participants correct errors with iPose. We did not compare our method against traditional pose editing methods (e.g., using Blender) because they have a significantly steeper learning curve.

In each session, a participant was first trained on two sample videos outside the 3DPW dataset, and then corrected three videos in a think-aloud manner. There was no limitation on editing time for each video; the participants ended editing when they were satisfied with the results. We recorded user intervention statistics, such as editing time and usage of functionalities, as well as participants’ comments. See Table 2 for detailed statistics of the 24 videos. At the end of the session, the participant completed a standard System Usability Scale (SUS) questionnaire [7]. Each session took about four hours, and the training session took about 20 minutes on average.

Figure 7:

View Figure

Accuracy. We adopt the conventional metrics of pose accuracy, including mean per joint position error (MPJPE), Procrustes-aligned MPJPE (PA-MPJPE), and mean per vertex position error (MPVPE). Table 1 shows the accuracy achieved by iPose within the editing time as listed in Table 2. We compared with automatic computer vision methods, whose performance is derived from the original papers.

Our method achieved significant accuracy improvements over existing automatic methods. In addition, we observed scenarios where human perception is advantageous at addressing issues challenging for automatic methods. For example, as shown in Figure 7(a), when a subject is walking towards the camera, the aperture problem makes the poses of limbs difficult to recover, but can be addressed with human’s prior knowledge of walking movements. The video subject’s left arm was heavily occluded in the Figure 7(b). However, with the understanding of the action that the video subject is pointing to a distant area, A1 easily recognized that the left arm should be straight instead of curled. Figure 7(c) contains challenging lighting conditions with heavy occlusion, which is solved by a hybrid of partial body parts observation and the knowledge of the getting off car action. Figure 7(d) is an example of camera perspective. A7 explicitly commented that it is difficult to compromise the right foot to align with the frame, because it would result in the toe pointing upwards and not matching reality. The camera distortion also caused the video subject’s left leg to appear longer than the right one. A7 edited referring to the 3D viewer instead of fixing the alignment on the video frame.

We also inspected video frames with large errors after corrections with iPose and have identified the following main sources of errors. First of all, the 3DPW dataset [38] in essence provides “pseudo” ground truth 3D poses, and thus the precision of ground truth is also limited. We observed many artifacts in ground truth 3D poses, such as the left arm in Figure 7(c) and the unstable leg poses in Figure 9(d). In these cases, the poses resulting from iPose are better than ground truth regarding physical plausibility. Besides, a general type of error is caused by the mismatch between the estimated body shape and the actual video subjects’ body shape. Even though iPose supports adjusting body shapes via sliders, there is an ambiguity in editing the global scale of video subjects. As illustrated in Figure 8, when there is a misalignment between the projection and the video subject, a user can either modify the camera scale parameter (Figure 8(d)) or the body height parameter (Figure 8(e)). Despite the alignment on the video frame, there remains a near-constant error in joint positions in the model space. In order to verify the influence of shape difference, we applied the ground-truth poses in the 3DPW test set onto SMPL with body shapes after editing with iPose, resulting in an average MPJPE 40.15mm, PA-MPJPE 20.71mm, and MPVPE 51.83mm (corresponding to \(51.55\%\), \(43.58\%\), and \(54.27\%\) of the total errors in Table 2). The third type of error comes from camera models. We adopt a weak perspective camera model to be consistent with computer vision methods, but this introduces an approximation error with the perspective projection [42]. Besides, the orientation of the camera often causes failures in recovering the video subject’s global orientation (e.g., Figure 9(b)). Finally, the depth ambiguity and occlusions remain challenging to solve when no precise semantics exist. For example, the bending of the upper body in Figure 9(a) and the right hand in Figure 9(c) remain uncertain.

Figure 8:

View Figure

Figure 9:

View Figure

Usability. The average SUS score was 80.93 (SD=11.38). While all participants found iPose easy to learn, they agreed that it is specific-purpose and not designed for the general public users. All participants agreed or strongly agreed that functions in iPose are well-integrated, and we observed that all functions were covered during their editing. For each video, apart from correction operations, the participants spent large portion of editing time previewing the action, identifying the errors and considering about the corrections. Since all frame contain certain amount of errors, the participants focused on major errors to trade-off between accuracy and editing time. Participants switched from 2D joint handles to the 3D rotation widget mainly to move joints perpendicular to the video frame (e.g., limbs in Figure 7(a)) and adjust the orientation of the head, indicating using one 2D joint handle to control the head is insufficient. Participants disabled auto-snapping for end-effectors because subjects in the 3DPW dataset wore hats and IMUs, making the snapping of heads and hands unstable. Participants often used the temporal propagation to reuse the correction results mainly when the issues in the following frames were similar, and when the corrections at a frame would be relatively complex but propagated from a previous frame and then refined were easier. The temporal propagation failed when movements were perpendicular to video frames, such as the walking towards camera action (e.g., Figure 7(a)).

Design Reflections. Through the user study, we identified the following limitations in the current design of iPose. Even though the 3D viewer provides alternative viewpoints to videos, the participants skipped checking under other viewpoints when there were no large misalignments on the video. This is problematic when the depth ambiguity is hardly noticeable. It is thus helpful to simultaneously visualize a few viewpoints to help identify the errors. Besides, we provide one joint handle for manipulating the head following the SMPL model definition, but this is often insufficient, so participants had to use the 3D rotation widget for finer adjustment. It is worth a special design for the head joint, such as manipulating look-at. Finally, the temporal propagation automatically corrects a fixed number of future frames. However, since there are large variations in video subjects’ action pace, it would be more desirable to adopt a flexible temporal propagation scheme, where a user can dynamically decide how many frames to propagate according to action complexity and number of consecutive frames with similar errors.

6.2 Expert Interviews

To understand the effectiveness of iPose in practice, we conducted semi-structured interviews with a researcher in rehabilitation (E1) and a practitioner in sports performance analysis (E2). We prepared video footage of rehabilitation exercises and sports actions. During the interview, we first demonstrated our interface and let the experts try it freely. Then we discuss revolving questions such as how they feel about iPose, how they position its role in their practice, and suggestions on improvements. Each session lasted about one hour.

Both experts confirmed the usefulness of iPose in their practice. E1: “Current clinical gait analysis relies heavily on marker-based capture. Markerless methods are very sensitive to lighting conditions. This tool demonstrates a promising future trend.” E2: “iPose will be especially useful in two cases: the training of action reproducibility for beginners and the analysis of an elite athlete template [4]. These two cases require accurate poses but currently they can only be conducted empirically by biomechanical analysts.” E1 suggests that it would be useful to enable pose correction with events since the requirements for accuracy differ with focus. For example, the range of motion is a key attribute in evaluating a gait cycle, and it would be desirable if a user can quickly locate the timing when a thigh reaches its extreme position and correct the pose at that moment. E2 mentions that when target users of the system are non-professional athletes, it is recommended to trade some accuracy for speed in displaying pose correction results.

A IMPLEMENTATION DETAILS

iPose runs on a PC running Ubuntu20.04 (Intel i7 @3.6GHz, 12GB RAM) with an Nvidia GeForce RTX 3080Ti GPU. The videos used in the user study are of resolution 1080p, with a frame rate of 25 fps. Video frames are cropped and scaled according to the initial projection of poses to best fit the video viewer window in the interface. Video frame processing with Segment Anything [23] ran at 0.5 fps on this PC in a separate thread from the main interface. To avoid drifting, we empirically set temporal propagation within five future frames, with duration about 10 seconds. λ_smooth, λ_s and λ_t in Equation (3) are all set to 100.

B RESULTS

B.2 Qualitative Results

We compare the reconstruction results from iPose with state-of-the-art automatic computer vision methods, including three image-based methods (ROMP [32], PyMAF [41], and BEV [33]) and two video-based methods (VIBE [24] and CycleAdapt [28]). We used video footage from the Internet with challenging sports poses. The results are shown in Figure 10. For automatic pose reconstruction, there is no significant difference in temporal consistency nor regularities in artifacts between image-based and video-based methods. While no automatic computer vision method can reconstruct human poses perfectly, iPose can effectively address the visible artifacts.

Figure 10:

View Figure

B.3 Comparison with Alternative Approach

3D body landmark detection is an orthogonal and complementary task with respect to pose reconstruction. Specifically, more accurate landmark detection could provide better handles for users to manipulate poses. To explore the possibility of an alternative approach that first regresses 3D joint positions and then fits a 3D body model to the joint positions with inverse kinematics, we compare the results from iPose with a state-of-the-art image-based 3D landmark detection method MediaPipe BlazePose [5]. As shown in Figure 11, except for the occluded legs Figure 11(d), BlazePose can accurately locate the joint positions from the video viewpoint. However, it still suffers from depth ambiguity, such as the errors in the elbow pose in Figure 11(e) and the right knee in Figure 11(f). The misinterpretation of the video subject’s center of gravity in Figure 11(a) and (f) also makes the poses hard to refine with manual interventions. Thus, there is still a gap in incorporating 3D joint positions in the pipeline.

Figure 11:

View Figure