Learning 3-D Human Pose Estimation from Catadioptric Videos

1 Introduction

Accurately reconstructing the human pose in 3-D from real images in a variety of indoor and outdoor scenarios, has a wide range of interesting applications in emerging fields such as virtual and augmented reality, human computer interaction, humanoid robotics and monitoring mobility. In the past 10 years, many 3-D pose-related datasets have emerged, which has greatly promoted the development of this field.

Existing 3-D pose datasets can be roughly divided into three categories. The first type collects 3-D pose data using a marker-based motion capture system synchronized with video, such as HumanEva [Sigal et al., 2010], Human3.6M [Ionescu et al., 2014] and GPA [Wang et al., 2019]. This type of data is mostly collected indoors, and restricts recording to skin-tight clothing. To overcome the limitations of marker-based data collection, several marker-less approaches have also been used. 3DPW [von Marcard et al., 2018] and MPI-INF-3DHP [Mehta et al., 2017] propose marker-less capturing system (based on inertial measurement sensors and recording in green screen studio with multiple cameras in a dome system). CMU Panoptic Studio [Joo et al., 2019] creates a panoptic studio and capture poses with 10 pre-calibrated RGBD cameras. This marker-less system enables diverse clothing but requires an expensive studio setup. The third is synthetic data which can be generated by retargeting MoCap sequences to 3-D avatars [Chen et al., 2016]. However the results lack realism. Trained using the synthetic data, learning based methods often suffer from the peculiarities of the rendering, leading to poor generalization to real images.

∗Corresponding author

In acquiring above data, either high-accuracy sensors or multiple-camera system are costly and have limited capturing scenarios. In this paper, we propose to exploit catadioptric videos for collecting 3-D pose. In conventional catadioptric systems (an optical system that combines refraction-based lenses and reflection-based mirrors), one can treat each mirror as a virtual camera. For example, [Gluckman and Nayar, 2001] designed and implemented a real-time catadioptric stereo system which uses only a single camera and two planar mirrors. There are two major challenges for the utilization of catadioptric videos. First, as pointed out by [Gluckman and Nayar, 2001], at least two mirrors are required for camera calibration and 3-D estimation, which is infeasible in socially-shared catadioptric videos (e.g., YouTube videos themed dancing tutorial with a mirror). Secondly, catadioptric videos are often captured in realistic scenarios, which significantly complicates reliable estimation of human keypoints. Unlike the marked-based systems, the initial pose estimation is often noisy and demands further refinement.

To tackle above challenges, we harness the following insight: different from generic objects, human body has many strong structure priors, such as the symmetry of the left and right bone length, the degree of freedom of keypoints, etc. We prove that, for one-mirror system that dominates online catadioptric videos, using the human priors can significantly reduce the ambiguity of camera's interinsic parameter calibration, and serves as a guiding objective in refining noisy human keypoints.

Our main contributions are three-folds: 1) We propose a large-scale Dance-before-Mirror (DBM) video dataset, which is more than 24 times larger than existing largest Human3.6M [Ionescu et al., 2014]. Overall 6,922 dance-beforemirror videos are crawled from video-sharing websites, and 175,652 20-second clips are extracted. The DBM data can serve as a new benchmark for 3-D human pose estimation; 2) We devise a novel algorithm for intelligently estimating highquality 3-D human keypoints from online catadioptric videos. Briefly, we boil it down to an optimization problem between two sets of mirrored 2-D human keypoints. The key trait of the proposed algorithm is the joint use of epipolar geometric formulas and human skeleton priors. In particular, the symmetry prior of left / right human body is utilized for inferring camera intrinsics and recovering occluded human keypoints; 3) Comprehensive cross-data transfer experiments are conducted together with other three datasets (Human3.6M, MPIINF-3DHP, 3DPW). The experimental results clearly demonstrate the usefulness of our data.

2 RelatedWorks

Human Pose Estimation For 2-D human pose estimation [Fang et al., 2017; Chen et al., 2018; Cao et al., 2017], most prevailing methods adopt an encoder-decoder architecture to predict a heatmap for each keypoint, from which the keypoint position is further inferred. Exemplar methods include Simple baseline [Xiao et al., 2018] and CPN [Chen et al., 2018]. More recently, HR-Net [Sun et al., 2019] uses the repeated multi-scale fusions across high-to-low and low-to-high sub-networks to exchange the multi-resolution information. For 3-D human pose estimation, some traditional methods like [Wei and Chai, 2010] propose a generative method using physics priors to get the 3-D pose in a monocular view. Others [Guan et al., 2009; Jain et al., 2010] introduce semi-automatic analysis-by-synthesis fitting of parametric body models. Modern deep network based methods [Pavlakos et al., 2017; Sun et al., 2018] can broadly be classified into two classes: direct regression and 'lifting' based approaches. The former requires lots of 3-D-pose labelled images and predict the 3-D location straightly from the image. But such datasets are either captured in studio scenarios with limited pose and appearance diversity [Ionescu et al., 2014] or contains lots of synthetic imagery [Chen et al., 2016]. While 'Lifting' based approaches predict the 3D pose from a separately detected 2-D pose [Martinez et al., 2017]. More recently, some literature [Qiu et al., 2019; Remelli et al., 2020] leverage the multi-view informantion, which further improve the performance.

Learning from Catedioptric Systems [Mariottini et al., 2012] proposes a new image-based camera localization and 3-D scene reconstruction algorithm by observing a scene being reflected on two (or more) planar mirrors. [Zhou et al., 2016] proposes an omnidirectional stereo vision sensor based on one single camera and catadioptric system. However, existing works are mostly geometry-oriented, not fully harnessing the learning approach and human skeleton priors, which inspires our work.

3 Dance-before-Mirror (DBM) Video Dataset

This section elaborates on the crawling / processing of largescale Dance-before-Mirror (DBM) video dataset, particularly a novel algorithm that extracts 3-D human keypoints from videos without human annotations.

3.1 Video Crawling and Clip Generation

Motivating the construction of DBM videos we consider several desiderata, including the comprehensive coverage of human poses, clothes, and background visual appearance. Furthermore, camera motion and shot change should be maximally excluded for easing the estimation of camera intrinsics. We observed in the past few years tremendous dancingteaching videos have been shared at social media websites. In particular, we choose Bilibili 1 , a popular video-sharing website in Asian countries themed around animation, comic, and games, and YouTube 2 as two main resources. On both sites we search videos with queries "dance tutorial mirror" and its several variants. The top-ranked videos in the search results are mostly records of dancing tutorials. Mirrors are generally positioned in front of the dancing tutors during the video recording in order for a non-occlusion viewing of the dancing actions. We manually filter out videos without mirrors from the returned results. This totals 6,922 videos (5,840 from Bilibili and 1,082 from Youtube) with durations from half a minute to longer than an hour, all of which we suppose are amenable to further processing. Figure 1 illustrates a few randomly-drawn frames from these videos. As we can see, DBM videos span a wide spectrum of key factors, e.g., human poses, races, genders, etc.

For data cleaning, a human detector (we use pretrained FPN [Lin et al., 2017] with ResNet-101 [He et al., 2016] on MS-COCO [Lin et al., 2014]) first scans video frames at 5 fps (frames per second). Only frames with two persons (tutor and the correspondence in mirror) are kept. Continuous 20 seconds (100 key frames) are trimmed out as clips. For diversity, at most 100 clips are reserved for each video.

Next, we use HR-Net [Sun et al., 2019] pretrained on MSCOCO and MPII [Andriluka et al., 2014] for detecting keypoints from each person. For each keypoint pi = (xi, yi, si), where (xi, yi) is the pixel coordinates, and si is the confidence score. In DBM videos, self-occlusion is commonly observed. We simply use the confidence scores of keypoints for filtering purpose. When si < τ, a keypoint is marked as occluded. When the number of occluded points in the entire clip is greater than N, the video is judged to be severely occluded and can not be used in subsequent algorithm. In practice, τ and N are empirically set to 0.7 and 1500, respectively.

3.2 Geometric Model

The geometry of our proposed model is depicted in Figure 2. Let O be the optical center and π be the single mirror in the catadioptric video. Conventionally, it is a dominating practice to add a virtual camera for each mirror (e.g., the mirrored optical center O′ in Figure 2).

1 https://www.bilibili.com/

2 https://www.youtube.com/

With rare exception a dancing mirror can be modeled as a planar. Suppose the world coordinate system is aligned with the camera's. A planar mirror π can be uniquely determined by a normal vector n (n ∈ R3 and ‖n‖2 = 1) and a nonnegative scalar d ≥ 0 for representing the orthogonal distance between mirror π and the optical center O. Let P ∈ R3 be an arbitrary space point in the scene which is not on the mirror. P ′ is its reflected point. The relationship between P, P ′ is specified by:

with two key elements calculated as:

which can be trivially verified by connectingP, P ′ in Figure 2 and integrating the connection [Gluckman and Nayar, 2001]. E is an identity matrix whose size can be inferred from its context. The matrix S = (R t; 0 1) is called the reflection matrix in mirror systems.

To obtain a Euclidean-sense estimation of human keypoints, it is necessary to determine the camera intrinsics. Generally, it is challenging to perform self-calibration on DBM videos due to the lack of informative clues. We simplify it through the following approximation as the work [Gluckman and Nayar, 2001] did: it is often the case that the skew is zero, the optical center is roughly above the image center. The remaining key variable to be learned is the focal length. In specific, our approximation leads to the following camera intrinsic matrix:

Use p, p′ to denote the pixels that correspond to P, P ′ , respectively. For a pinhole camera model, their relationship can be represented as

where Z,Z ′ are the values on the z-axis for P, P ′ respectively.

3.3 3-DKeypoint Estimation

The geometric model is parameterized by the mirror normal n and focal length f. For right now we assume f is known and defer its optimization in Section 3.5. To learn n, we have the following observation (detailed derivation is found in the supplemental material):

Theorem 1 (Fundamental matrix for 1-mirror system). For an arbitrary space point P ∈ R3 , the following holds

Eq. 5 paves the way of computing n. From all matched points (p, p′ ) in a clip, we use the RANSAC [Fischler and Bolles, 1981] algorithm to estimate an optimal n. R, t follow by Eq. 2. Next, to estimate the depth of human keypoints, we apply the standard triangulation technique [Hartley and Zisserman, 2003; Ma et al., 2010] to Eq. 4, returning two depth values Z, Z ′ .

3.4 3-DKeypoint Refinement

Since our algorithm solves the 3-D keypoint estimation from matched 2-D points, the position accuracy of the 2-D keypoints becomes crucial. Unfortunately, in most DBM video, the self-occlusion inevitably appears and causes unreliable 2D estimation.

To recover the occluded 2-D keypoints, we first divide the human keypoints into two categories: torso keypoints and the limbs keypoints. The keypoints of the torso mainly include head, neck, shoulder and hip, and the keypoints of the limbs include elbow, wrist, knee and ankle. When human moves, the motion of the torso is typically slow, yet the limbs often swing quickly. Based on this observation, we propose different strategies for refining the two sets of keypoints, respectively.

For slow-moving torso keypoints, we use temporal filtering to refine the occluded keypoints. As mentioned before, when the confidence of a keypoint estimation is below τ, it is regarded as an occluded point. For a torso keypoint, represent its point-sequence along time as {.., hi-1, hi, hi+1, ...}. When hi is occluded (i.e., with low confidence) yet neither of hi-1, hi+1 is, we simply use 0.5 · (hi-1 + hi+1) to replace the original hi. A sequential scan of all such sequences completes the refinement.

For the limbs keypoints, we refine them by jointly using epipolar geometry and human skeleton priors. In the singlemirror catadioptric system, point e in Figure 2 is known to be an epipole. According to the nature of the catadioptric system and epipoles, all lines that connect two matched points (i.e., any pair (p, p′ )) will intersect at the epipole e. If the camera that captures the video is visuable, the epipole will be right on where the camera locates. An example is shown in Figure 3.

To refine occluded points, it is important to first estimate the pixel position of the epipole. For a clip captured by a fixed camera, the epipole stays unchanged in all frames. In practice, we choose a few matched points of highest confidence scores. The line connecting each pair is estimated. We calculate the intersection between any two lines, obtaining a candidate point of the epipole. Finally, the median of all candidate points is calculated to serve as a robust estimation of the epipole's position. See Figure 3 for an illustration. Let e = (xe, ye) be the pixel coordinate of the epipole point. As shown in Figure 3, let (amr, aml, ar, al) be the elbow points and (wmr,wml,wr,wl) be the wrist points. Without loss of generality, assume wr is occluded and suffers from inaccurate 2-D estimation. Since wmr is not occluded, and wr is located on the epipolar line ---→ ewmr. We have the epipolar constraint

where (xwr , ywr ) is the candidate pixel coordinate of wr.

For point pairs (wl, wml), (al, aml), and (ar, amr) that are not occluded, we can use the triangulation method in Section 3.3 to find their corresponding 3-D coordinates Wl,Al and Ar. As a skeleton prior, the length of the left and right arms of a human should be equal. This prior leads to the following equation:‖Wl - Al‖2 = ‖Wr - Ar‖2, (7)

where ‖ · ‖2 is the Euclidean distance. Wr is the 3-D coordinate of wr, which can be inferred via triangulation after knowing (xwr , ywr ). In practice, we can sample a few candidates for (xwr , ywr ), and keep the one maximally satisfying Equations 6 and 7 as the best guess.

Figure 4 shows the all six possible occlusion cases of the shoulder-eblow pair. The joint optimization of Equations 6 and 7 solves Case 2 in Figure 4. Indeed, we claim that Cases 3 and 4 are also solvable by reducing them to a conditional version of Case 2. The details are deferred to the supplementary materials. To tackle other human keypoints, we associate the human body keypoints into shoulder-eblow, elbow-wrist, hip-knee and knee-ankle. The occlusion in other configurations can be restored likewise.

3.5 Determination of Optimal Focal Length

All above computations assume a known focal length. Since f is usually in a moderate value range, we adopt an exhaustive search scheme for finding the optimal focal length. In specific, for each candidate f, we estimate all 3-D keypoints as aforementioned. Next, for each f a skeleton symmetry score is computed over all left-right limb pairs, such as min(‖Wl - Al‖2, ‖Wr -Ar‖2)/max(‖Wl -Al‖2, ‖Wr -Ar‖2). The f that attains best skeleton symmetry will be kept. Figure 5 shows the mean / std of estimated focal lengths over all clips in a video. In practice we apply the mean to all clips.

4 Evaluations

4.1 Investigation of DBMData

After all above data filtering operatioins, we divide the train/val/test set according to the ratio of 7:1:2, and get 124243/18114/33295 clips respectively. During the division process, we ensure that the clips of the same video do not appear in different sets. Table 1 contrasts the key statistics of mainstream 3-D human pose benchmarks. Our intelligentlycollected DBM surpasses all others in term of data scale.

We next evaluate the quality of epipole estimation, which is crucial for 3-D keypoint refinement. Geometrically, in the mirrored image, the position of the camera is the position of the epipole point, as the case in Figure 3. We manually pick 100 videos that have the camera visible, and then mark the position of the camera as the groundtruth of epipole point. Table 2 summarizes the Euclidean distance between the estimated pole coordinates and the true pole coordinates by varying point pairs involved in the computation. More point pairs are observed to generate more robust estimation.

Others about the distribution of the quality of 3-D keypoints in human skeleton prior (such as human skeleton symmetry), and the influence of refinement on the quality of 3-D keypoints are also important. We present more details in the supplemental materials.

4.2 3-D Pose EstimationModel

Following [Pavllo et al., 2019], we use a fully-connected network with residual connections for 3-D pose estimation. The architecture is shown in Figure 6. The input layer takes the concatenated (x, y) coordinates of J joints for each frame and outputs a 1024-dimensional feature. This is followed by two ResNet-style blocks which are surrounded by a skipconnection. Each block performs two fully-connected layers and they are followed by batch normalization, rectified linear units, and dropout. Finally, the last layer outputs a prediction of the 3-D poses for the input frame.

4.3 Datasets for Comparison

Human3.6M (H36M) [Ionescu et al., 2014] is a popular motion capture dataset containing 3.6 million video frames for 11 subjects, of which 7 are annotated with 3-D poses. Following [Pavllo et al., 2019], we train on 5 subjects (S1, S5, S6, S7, S8) and test on 2 subjects (S9, S11). 3DPW [von Marcard et al., 2018] is a wild video dataset taken from a moving phone camera. It contains 60 videos and about 68k frames covering multiple scenarios and actions, with the annotated 3-D positions of 24 keypoints. 3DHP [Mehta et al., 2017] is a human pose dataset captured in a multi-camera studio, covering a wide range of viewpoints and actions. It contains about 1.8 million frames recorded by 14 cameras, including 8 actors and 8 activity sets. Each frame has the annotation of 3-D keypoint positions (28 for train and 17 for test). We train and test our model on the original train/test split for 3DPW and 3DHP.

4.4 Data Distribution Analysis and Visualization

To study the human pose coverage of the proposed DBM data, we adopted the t-SNE [Maaten and Hinton, 2008] to visualize the pose distribution in the dataset. Specifically, we randomly select 1000 poses from the training set of DBM and H36M respectively, and project them into the 2-dimensional diagram, as shown in Figure 7. Critically, since most DBM videos contain standing-like poses, poses like "sitting" and "sitting down" are inadequate in DBM in comparison with H36M, as seen in the top-right zone. Otherwise, DBM covers larger spectrum of human poses, such as the pose of raising both hands, and the pose of separating the hands and legs together. We highlight some zones with exemplar poses in Figure 7. This shows the DBM dataset can enrich existing data, thereby promoting the development of the entire community.

4.5 Cross-Dataset Generalization

A good 3-D pose dataset shall generalize well on other crossscenario datasets. To validate the generalization capability of different 3-D pose datasets, we train our 3-D pose estimation model separately on four 3-D pose estimation datasets, DBM, H36M, 3DPW and 3DHP, and conduct cross-dataset evaluation by testing these models on the rest of the datasets.

We consider two metrics in our cross-dataset evaluation, following common practice. The first is mean per-joint position error (MPJPE) in millimeters, calculating the mean error between the predicted joint positions and the ground truth. The second is procrustes analysis MPJPE (P-MPJPE) in millimeters, calculating the MPJPE after alignment with the ground truth in rigid transformation, rotation and scale. For the whole four datasets, we train the same 3-D pose estimation model as specified in Figure 6. Hyperparameters such as learning rate and dropout ratio are the same as those in [Pavllo et al., 2019]. In order to make a fair comparison on different datasets, the input of the 3-D pose estimation model is the ground truth 2-D keypoints. Following [Wang et al., 2020], we choose the mutual 14 joints of DBM, H36M, SURREAL, 3DPW and 3DHP. Because the bone lengths of the human in each dataset are different, we use the average bone length of the target data to normalize the predicted 3-D coordinates before performing cross-data tests.

The cross-dataset evaluation results are shown in Table 3. In most settings, DBM and 3DHP are two best performers. DBM shows some more extraordinary results in terms of PMPJPE. For instance, model trained on DBM achieves the optimal P-MPJPE when transferred to H36M and 3DPW compared to other datasets. We would emphasize that data in H36M and 3DHP are obtained by sensors, and the way we obtain DBM data is much cheaper in contrast. Nonetheless, the cheaply obtained DBM shows a generalization ability comparable or better to H36M and 3DHP. When compared to 3DPW, whose data is also obtained in a cheap way using IMU, model trained on DBM performs much better generalization ability on H36M (83.9 v.s. 157.7) and 3DHP (109.8 v.s. 158.1) test sets.

To further conduct fine-grained study, we test our DBM model on different kind of actions in H36M and compare them to the results trained on 3DHP. The results are shown in Table 4. Our main observation is that our performance in many actions is better than 3DHP except for sitting-related actions. DBM is a dataset related to dancing movements, thus the sitting scenario is relatively rare. Therefore, we conduct another experiment to complement 280K sitting poses from 3DHP to DBM, forming a new data DBM+. With the supplied sitting data, DBM almost dominates all action categories. In addition, we compare the model trained on H36M or H36M+DBM. The observed improvement (e.g., 32.7 v.s. 30.7 in P-MPJPE) proves the DBM is strongly complementary to existing datasets.

4.6 Some Examples

In Figure 8, we illustrate some examples of the DBM dataset. The first row is the original video frames, and the second and third rows are the 2-D and 3-D pose estimation results of the persons obtained by our proposed method.

5 Concluding Remarks

This work constructs a new large-scale DBM benchmark for 3-D human pose estimation. We explore a new method of collecting 3-D pose from webly-available catedioptric videos, and propose a novel algorithm that combines geometry and human skeleton priors. In future we will include active annotation for further improving the data quality.

Acknowledgments

This work is supported by National Key R&D Program of China (2018AAA0100702), National Natural Science Foundation of China (61772037) and Beijing Natural Science Foundation (Z190001).