Human Motion Generation via Cross-Space Constrained Sampling

1 Introduction

Video generation, especially human motion video generation, has been attracting increasing research attention. Early methods [Vondrick et al., 2016; Villegas et al., 2017a] directly apply/extend conventional 2D GAN (i.e., used to deal with 2D image generation) to generate 3D spatio-temporal video. However, these approaches usually yield low video quality (non-realistic looking) due to the high dimensional searching space. To this end, some recent methods [Yan et al., 2017; Walker et al., 2017] have attempted to constrain the generator with human skeleton information (e.g., skeleton figures or joint position maps), thus to output more realistic articulated human motions. However, these methods also have significant limitations. First, most of these algorithms require for each frame a corresponding specific skeleton pattern for image synthesis. In other words, a sequence of skeleton representation vectors (or joint positions) should be given in prior for video generation. However, in most cases, it is very hard to get this information, which greatly limits its application. Second, these methods always require pairs of image frames with same background and identical person for supervised training. However, to obtain such strong supervised training data is very expensive, which in turn forbids further scale up of algorithmic training.

∗Corresponding Author: Bingbing Ni.

†Shanghai Institute for Advanced Communication and Data Science, Shanghai Key Laboratory of Digital Media Processing and Transmission, Shanghai Jiao Tong University, Shanghai 200240, China

To explicitly address these issues, this work proposes a new problem setting. Namely, given a SINGLE static image (with a human figure inside, denoted by "input person") and a video dataset of a specific human motion of some other persons (e.g., walking, dancing, denoted by "target motion"), we aim to generate a novel video sequence of the input person acting some similar motion out. Note that the synthesized motion images should not follow exactly the same motion (i.e., the same joint position movements), and we DO allow randomness in terms of the motion. In other words, we shall first sample a proper sequence of (articulated) motion representations from the specific motion representation space of the target action type, and then according to these generated motion representations and the input human figure to further synthesize the full sequence of motion images.

It is therefore observed that for each time stamp, simultaneous sampling/generation of a particle in the motion representation space as well as an corresponding image in the appearance space is required, and moreover, the pair of samples in both space should be constrained by each other (i.e., make both sides compatible). Motivated by this observation, in this work we propose a cross-space human motion video generation network which features two paths: a forward path that first samples/generates a sequence of low dimensional motion vectors based on Gaussian Process (GP) (as GP is an effective latent space method modeling human motion), which is paired with the input person image to form a moving human figure sequence; and a backward path based on the predicted human images to re-extract the corresponding latent motion representations. As lack of supervision, the reconstructed latent motion representations are expected to be as close as possible to the GP sampled ones, thus yielding a cyclic objective function for cross-space (i.e., motion and appearance) mutual constrained generation. We further propose an alternative sampling/generation algorithm with respect to constraints from both spaces. As a form of self-supervision, the above framework NO LONGER needs pair of groundtruth image and input frame sharing same background and identical person for model training, makes the approach very flexible. Extensive experimental results show that the proposed framework successfully generates novel human motion sequences with reasonable visual quality.

2 RelatedWork

Gaussian Process. [Lawrence, 2004] proposes GPLVM that apply Gaussian Process [Seeger, 2004] to model articulated motion. [Wang et al., 2008] further extends GPLVM to a dynamic version which models temporal dependent data. [Bui et al., 2016] models the multiple properties of single object with hierarchical Gaussian Process. Many researchers utilize Gaussian Process for various applications like 3D human tracking [Sedai et al., 2013] and human action classification [Li and Marlin, 2016]. However, traditional Gaussian Process can only sample low dimensional data like skeletons, and not high dimensional data like videos. To address this issue, our work generates video sequence using GP-GAN cross-space constrained sampling.

Video Generation. The early methods [Sch¨odl et al., 2000; Agarwala et al., 2005] utilize video texture to generate video sequence. In the last years, there have been a number of works applying GAN on video generation. [Vondrick et al., 2016] generates a spatio-temporal cuboid with two independent streams: a moving foreground pathway and a static background pathway. [Villegas et al., 2017a] proposes a motioncontent network that apply Convolutional LSTM to capture the temporal dynamics. Some works use pose information in human video generation. [Ma et al., 2017] introduces a twostage approach to generate a new image from a person image and a novel pose. [Yan et al., 2017] further considers the continuity and smoothness of the video frames and generate a human motion video with a single iamge and human skeleton sequence. [Walker et al., 2017], [Villegas et al., 2017b] and [Denton and vighnesh Birodkar, 2017] first model possible movements of humans in the pose space, and then use the future poses to predict the future frames in the pixel space. These works require a lot of auxiliary constraint data such as the skeleton of each frame. However, our method overcomes this limitation with cross-space constraint and only needs the motion space.

3 Method

3.1 ProblemDefinition andMethod Overview

As illustrated in Section 1, we first formulate the task mathematically as follows: given a static image ( "input person"), denoted as y0 ∈ R1×C×W×H , and a specific human video sequence, denoted as Y ∈ RN×C×W×H , we aim to first sample a motion sequence, denoted as ˆS ∈ RN×D ("target motion"), then generate a novel video, denoted as ˆY ∈ RN×C×W×H , of the input person y0 acting the target motion ˆS out to some extent of motion randomness. Here C, W and H represent the image channel, width and height. N denotes the length of the sequence. D is feature dimension.

There are two long standing problems on generating videos, i.e., motion consistency as well as smoothness. As stated above, the motion randomness is another important consideration. In the majority of the previous works [Villegas et al., 2017b; Denton and vighnesh Birodkar, 2017], ground truth is treated as exclusive generation target, which significantly restricts the generation variety. So it is demanding to develop a novel generation scheme which could deal with these three aspects appropriately.

To this end, we propose a cross-space human motion video generation network. First, the proposed network generates a sequence of low dimensional motion representation vectors ˆS based on a Gaussian Process (Motion Generation Module). Second, the generated motion vectors ˆS are utilized to form a moving human figure sequence ˆY (Appearance Generation Module) based on the static image y0. Third, taking above three aspects into consideration, we propose a cyclic framework to sample instances from both spaces in a collaborative way, where sampling in one space is conditioned on the other one. Detailed architecture of our model is shown in Figure 2.

3.2 Motion GenerationModule

This module is based on Gaussian Process, which consists of a mapping from a latent space T to the motion space S and a dynamical model in the latent space. A latent variable mapping can be formulated as

where S is the motion sequence, T = {t1, ..., tN} denotes the latent representation, f is a non-linear mapping, and ϵ is a zero-mean white Gaussian noise process.

Inspired from Gaussian Process Dynamical Model (GPDM) [Wang et al., 2008], we obtain the latent prior by marginalizing out the parameters

p (T|¯α)

where KT is a kernel matrix with hyperparameters ¯α where the elements are defined by a linear and RBF kernel

Similarly, GPDM marginalizes over f in closed form and obtains a Gaussian density

where KS is a kernel matrix with hyperparameters ¯ β andW is a scaling matrix. KS is constructed by a RBF kernel

Thus, the generative model can be defined by

where the priors are p (¯α) ∝ ∏ α-1 i , p ( ¯ β) ∝ ∏ β-1 i and

We follow the strategy of [Wang et al., 2008] and use the two-stage map estimation algorithm to optimize the Gaussian Process model. We first estimate the hyperparameters Θ = {α, β,W}. Then we update T with holding Θ fixed. In the fisrt step, we optimize

Using Hamiltonian Monte Carlo (HMC) to sample T from p (T|S, ¯α, ¯ β,W), we obtain the approximation

In the second step, we optimize

The optimization procedure is illustrated in Algorithm 1.

3.3 Appearance GenerationModule

Given a static image y0 and a sequence of skeletons ˆS = {ˆs1, ..., ˆsN} produced by the motion generation module, this module aims to generate a video ˆY = {y1, ..., yN} of the input person acting the target motion out. The motion in the video should be coherent and the appearance ought to remain consistent over frames. To this end, we propose a conditional GAN, which is able to fully utilize the appearance as well as the skeleton information. On one hand, inspired from [Zhu et al., 2017], pairs of skeleton and appearance images are stacked along the channel as input. On the other hand, the discriminator is required to distinguish ground truth from generated ones conditioned on the static image y0.

Adversarial Loss. After sampling a sequence of skeletons, we use the conditional GAN above to generate a completely new video sequence. Our generative model has two inputs of an image y0 and skeletons S. We apply adversarial loss [Goodfellow et al., 2014] as follows:

where G tries to generate images G(y0, S), and D tries to distinguish between synthesized images G(y0, S) and real images Y.

3.4 Cross-space Constraint

Only using adversarial loss is not feasible to generate satisfying results, because these two modules are trained independently, i.e., the motion correspondences can not be guaranteed. To this end, we propose cross-space constraint to bridge the gap between the skeleton and video space. The proposed constraint, consisting of consistency and smoothness in both two spaces, is shown as follows.

Pose Constraint (Consistency & Smoothness). There are many possible mapping functions that the generator may learn [Zhu et al., 2017]. To pursue the most likely one, we utilize the pose information contained in the video space to ensure the pose consistency. To be specific, the poses of synthesized images should correspond to the input poses (consistency). We use a pose estimator F to extract the human skeletons from the generated images which should be near the input skeletons, and apply an L1 loss

Another constraint is that the trajectory of human motion is usually smooth, e.g. the limb swing during walking, which is also contained in the video space. Note that the MAP estimation of Gaussian Process inside our motion generation module is closely related to the motion smoothness [Wang et al., 2008]. So we use the log-likelihood to measure the smoothness of generated poses:

Appearance Constraint (Consistency & Smoothness). In the video space, we introduce some constraint that can preserve smoothness of appearance between the input and output. We make the generator to approximate an identity mapping using a pixel-wise identity loss

To keep consistency of the person when generating the images, we use a classifier C to determine whether the synthesized images belong to the same category as the input image.

Full objective. Considering all above constraints, our full objective is

where λ, γ, α, β are the weights of different objectives. We aim to solve

Optimization. During the optimization procedure, we iteratively perform sampling motion sequence and generating video sequence as follows:

• Static Image to Skeleton Sequence. Given a starting point s∗0 , the corresponding latent representation t∗0 can be estimated by

Algorithm 1 Optimization Algorithm

Input: M human images {y(n)}Mn=1 .

• Q video sequences {Y(q)}Qq=1 .
• Q skeleton sequences {S(q)}Qq=1 .

Output: parameters Θ∗ = {α∗ , β∗ ,W∗ }, matrix T∗ , generator G∗ .

• 1: Initialize ¯α⇐ (0.9, 1, 0.1, e) , ¯ β ⇐ (1, 1, e) , {wk} ⇐ 1.
• 2: Initialize G,D using xavier.
• 3: repeat
• 4: // Motion Generation (Gaussian Process)
• 5: Sample {T(r)}Rr=1 ∼ p (T|S, ¯α, ¯ β,W).
• 6: Construct {K(r) S ,K(r) T }R r=1 from {T(r)}Rr=1 .
• 7: for j = 1 : J do
• 8: for k = 1 : D do
• 9: d ⇐ [(S)1k , ..., (S)Nk ]T .

• 11: end for
• 12: { ¯α, ¯ β} ⇐ minimize Lε (Θ) (Equation 9) using SCG for K iterations.

13:

• {T} ⇐ maximize Lθ (T) (Equation 10) using SCG for K iterations.
• 14: end for
• 15: // Appearance Generation (Conditional GAN)
• 16: for i = 1 : I do
• 17: Sample ˆS.
• 18: for all batches Sb in ˆS do
• 19: Minimize L(G,D) (Equation 16).
• 20: Update G, D.
• 21: end for
• 22: end for
• 23: until Convergence

Then we use a HMC sampler to generate a new latent sequence ˆT∗ , and apply a Gaussian prediction to get the final prediction ˆS∗ :

This procedure is optimized according to Lε (Θ) and Lθ (T).

• Skeleton Sequence to Video Sequence. Given a skeleton sequence, we use the proposed generator to produce new video sequence. And here the cross-space constraint L (G, D, F, C) is applied in both motion and video space.

We summarize the whole optimization procedure in Algorithm 1.

3.5 Implementation Details

We implement Gaussian Process based on [de G. Matthews et al., 2017], and adapt the architecture for generative networks from [Zhu et al., 2017]. All networks were trained using the Adam solver with a leanrning rate as 0.0001 and a batch size of 10. We set λ = γ = 10 and α = β = 1. As shown in [Zhu et al., 2017], we apply a least-squares loss [Mao et al., 2017] rather than the negative log likelihood for higher quality results. We also update the discriminator with the 50 previously generated images [Shrivastava et al., 2017] instead of the latest one to reduce model oscillation. The skeletons are extracted by the state-of-the-art pose estimator (OpenPose) [Cao et al., 2017]. And we fine tune the ResNet-18 [He et al., 2016] pre-trained in the ImageNet as the classifier.

4 Experiment

In this section, we conduct both quantitative and qualitative experiments to evaluate the performance of proposed framework. To demonstrate the effectiveness of our model, we also conduct detailed comparison experiments with two strong baselines. Meanwhile we present in-depth analysis on the contribution of proposed cross-space constraint, which are the key component in our model. Details are given as follows.

4.1 Datasets

KTHDataset. This dataset [Schuldt et al., 2004] contains six types of human actions: walking, running, jogging, boxing, hand clapping and hand waving. There are 25 subjects with four different scenarios outdoors and indoors. And the videos are gray scale with a resolution of 120 × 160.

Human3.6MDataset. This dataset [Ionescu et al., 2014] offers poses and videos taken from 10 actors in 17 scenarios. All videos are recorded from four different views simultaneously in indoor environment. The main difficulties of generation lie in two aspects : (1) The Human3.6M dataset contains many subtle movements throughout all video sequences, for example random swing of limbs. (2) The appearance of 10 actors vary largely, which is very difficult for the model to learn.

4.2 Evaluation

Evaluation Setup. We compare the proposed model against two strong baselines: McNet [Villegas et al., 2017a] and DrNet [Denton and vighnesh Birodkar, 2017], which achieve state-of-the-art performance in human motion generation task. To fairly compare with them, we follow the evaluation setup of DrNet, where the 10th frame is used as input image y0, to generate the following 30 frames. For KTH datasets, we use person 1-15 for training and 16-25 for testing. For Human3.6M dataset, we train on subject 1, 5, 6, 7, 8 and test on subject 9, 11. We resize the frames to resolution of 256×256. Note that not all the skeletons could be successfully detected by the OpenPose. To solve this problem, we manually annotate the skeletons of all the remaining failure cases.

Qualitative Evaluation. Figure 3 demonstrates qualitative results of our method and two baselines on KTH dataset. Here we remark two observations: (1) With the time step increasing, both the proposed model and DrNet well preserve the shape completeness of human figures, while it degrades rapidly on the results of MCNet. (2) Compared to DrNet, our model further successfully preserves the color of human body and pose transition during the long-term prediction (Note the comparison with DrNet from T = 20 to T = 30). These results clearly demonstrate that the articulated information (human skeleton) boosts the generation quality of human motion by a large margin. And more importantly, with the help of cross-space constraint, our model maintains the appearance consistency and motion continuity, especially on the longterm generation task.

Quantitative Evaluation. To quantitatively evaluate the quality of generated results, we conduct the Inception Score [Salimans et al., 2016] comparison experiments with these two baselines. As commonly used evaluation metric on image generation task, it is considered more appropriate than the PSNR or SSIM on video generation domain. Following the evaluation setup of DrNet, the Inception Score is calculated for 100 timesteps. As shown in Figure 6, the Inception Score of MCNet is unstable and decays rapidly with the increasing number of generated frames, which indicates degraded video quality during long-term generation task. The performance of DrNet is close to MCNet during the first few timesteps (from T = 0 to T = 40), and keeps relatively higher scores at latter time steps(from T = 40 to T = 100). The proposed model outperforms these two baselines during the whole generation procedure. It indicates that benefiting from the cross-space constraint, the sampled clear and reasonable motion guarantees the following high quality video generation.

Human Preference Evaluation. To comprehensively evaluate the performance of our model, we also conduct human preference experiment, according to the human psychophysical evaluation metric [Vondrick et al., 2016]. We conduct 2000 comparisons in total with 50 different people (40 preference selection for each person), i.e., we collect 1000 comparisons against McNet and 1000 against DrNet. They are required to select one out of two videos they prefer under the criteria of appearance consistency and motion continuity. As shown in Table 2, our results are perceptually more realistic compared to the baselines. However, we observe nearly the same preference proportion for the hand clapping, hand waving and boxing motions. It mainly results from these actions usually involve tiny movement, which largely reduces the video generation difficulty. When generating more complicated motion like walking and running, our approach performs much better than these baselines, benefiting from the proposed cross-space constraint. Overall, out method outperforms the baselines by a large margin, i.e. 81.5% of people prefer ours over McNet and 63.6% over DrNet.

Ablation Study. For the key contribution of our work, crossspace constraint, we conduct detailed ablation study to evaluate the effectiveness of each component. As illustrated in Figure 4, we can observe that these losses have different effects on the generated videos. Specificly, removing the GAN loss leads to highly unnatural generation results which completely losses the original texture of the reference image. It indicates that the adversarial loss is important for video generation task. Removing the pose constraint, consisting of Equation 12 and 13, results in the poses nearly unchanged during the whole video sequence. It shows that pose loss mainly helps to learn the target motion transition generated by the motion generation module. The appearance constraint consists of Equation 14 and 15. Removing it makes the generator fail to preserve color consistency between the input and output. Even we input a dim image, the network generates very bright frames. Combining these three losses together, our model could largely benefit from both appearance and pose constrains as well as the cyclic training framework. The generation results are visually satisfying. Meanwhile we perform quantitative evaluations over these loss terms with results shown in Table 1. We compute Inception Score with three types of action in KTH dataset. Note that the reality and diversity are important criteria in Inception Score evaluation, which do not take the consistency into consideration. So one can notice that relatively higher scores in term of GAN+Pose, which mainly results from the lack of appearance constraint. Generally all three components have positive effect on our video generation task. The proposed cross-space constrained framework boosts the performance by a large margin.

5 Conclusion

In this work, we propose a cross-space human motion video generation model. By combining Gaussian Process and conditional GAN, we introduce a cyclic objective for cross-space mutual constrained generation. Experiments show that our model generates high quality video sequence of the input person acting the target motion out.

Acknowledgments

This work was supported by National Science Foundation of China (U1611461, 61502301, 61521062). The work was partially supported by China's Thousand Youth Talents Plan, STCSM 17511105401 and 18DZ2270700.