{"type":"ScholarlyArticle","title":"Human Motion Generation via Cross-Space Constrained Sampling","authors":[{"name":"Zhongyue Huang","slug":"zhongyue-huang","given":"Zhongyue","family":"Huang","affiliation":"Shanghai Jiao Tong University , China; Shanghai Institute for Advanced Communication and Data Sci- ence , Shanghai Key Laboratory of Digital Media Processing and Transmission , Shanghai Jiao Tong University , Shanghai 200240 , China"},{"name":"Jingwei Xu","slug":"jingwei-xu","given":"Jingwei","family":"Xu","affiliation":"Shanghai Jiao Tong University , China"},{"name":"Bingbing Ni","slug":"bingbing-ni","given":"Bingbing","family":"Ni","affiliation":"Shanghai Jiao Tong University , China"}],"abstract":"We aim to automatically generate human motion sequence from a single input person image, with some specific action label. To this end, we propose a cross-space human motion video generation network which features two paths: a forward path that first samples/generates a sequence of low dimensional motion vectors based on Gaussian Process (GP), which is paired with the input person image to form a moving human figure sequence; and a backward path based on the predicted human images to re-extract the corresponding latent motion representations. As lack of supervision, the reconstructed latent motion representations are expected to be as close as possible to the GP sampled ones, thus yielding a cyclic objective function for cross-space (i.e., motion and appearance) mutual constrained generation. We further propose an alternative sampling/generation algorithm with respect to constraints from both spaces. Extensive experimental results show that the proposed framework successfully generates novel human motion sequences with reasonable visual quality.","doi":"10.24963/ijcai.2018/105","url":"https://ijcai.org/proceedings/2018/105","pdf":"https://ijcai.org/proceedings/2018/0105.pdf","markdown":"https://ijcai.org/proceedings/2018/105.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/105","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":105,"pages":{"first":757,"last":763},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Agarwala et al., 2005] Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael F. Cohen, Brian Curless, David Salesin, and Richard Szeliski. Panoramic video textures. international conference on computer graphics and interactive techniques, 24(3):821-827, 2005.","title":"Panoramic video textures","authors":"Aseem Agarwala, Colin Ke, Chris Zheng, Maneesh Pal, Michael F Agrawala, Brian Cohen, David Curless, Richard Salesin, Szeliski","year":"2005","venue":"international conference on computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bui et al., 2016] Thang D. Bui, José Miguel Hernández- Lobato, Daniel Hernández-Lobato, Yingzhen Li, and Richard E. Turner. Deep gaussian processes for regres- sion using approximate expectation propagation. ICML, pages 1472-1481, 2016.","title":"Deep gaussian processes for regression using approximate expectation propagation","authors":"D Thang, José Bui, Daniel Miguel Hernández-Lobato, Yingzhen Hernández-Lobato, Richard E Li, Turner","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In 2017 CVPR, pages 1302-1310, 2017.","title":"Realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de G. Matthews et al., 2017] Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman. Gpflow: a gaussian process library using tensorflow. Journal of Machine Learning Research, 18(40):1-6, 2017.","title":"Gpflow: a gaussian process library using tensorflow","authors":"Alexander G De, G Matthews, Mark Van Der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, James Hensman","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Denton and vighnesh Birodkar, 2017] Emily L. Denton and vighnesh Birodkar. Unsupervised learning of disentan- gled representations from video. NIPS, pages 4417-4426, 2017.","title":"Unsupervised learning of disentangled representations from video","authors":"Emily L Denton, Birodkar","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in NIPS 27, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2014","venue":"Advances in NIPS 27","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ionescu et al., 2014] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sens- ing in natural environments. PAMI, 36(7):1325-1339, 2014.","title":"Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments","authors":"Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu","year":"2014","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lawrence, 2004] Neil D. Lawrence. Gaussian process latent variable models for visualisation of high dimensional data. In Advances in NIPS 16, pages 329-336, 2004.","title":"Gaussian process latent variable models for visualisation of high dimensional data","authors":"Neil D Lawrence","year":"2004","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Marlin, 2016] Steven Cheng-Xian Li and Ben- jamin M. Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. NIPS, pages 1804-1812, 2016.","title":"A scalable end-to-end gaussian process adapter for irregularly sampled time series classification","authors":"Steven Cheng-Xian Li, Benjamin M Marlin","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2017] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. NIPS, pages 405-415, 2017.","title":"Pose guided person image generation","authors":"Ma","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray- mond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In 2017 ICCV, pages 2813-2821, 2017.","title":"Least squares generative adversarial networks","authors":"Xudong Mao, Qing Li, Haoran Xie, Y K Raymond, Zhen Lau, Stephen Paul Wang, Smolley","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Salimans et al., 2016] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NIPS, pages 2234-2242, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schuldt et al., 2004] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th ICPR 2004., vol- ume 3, pages 32-36, 2004.","title":"Recognizing human actions: a local svm approach","authors":"Schuldt","year":"2004","venue":"Proceedings of the 17th ICPR 2004","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schödl et al., 2000] Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.","title":"Video textures","authors":"Schödl","year":"2000","venue":"Proceedings of the 27th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sedai et al., 2013] Suman Sedai, Mohammed Bennamoun, and Du Q. Huynh. A gaussian process guided particle filter for tracking 3d human pose in video. IEEE Transactions on Image Processing, 22(11):4286-4300, 2013.","title":"A gaussian process guided particle filter for tracking 3d human pose in video","authors":"Suman Sedai, Mohammed Bennamoun, Du Q Huynh","year":"2013","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seeger, 2004] Matthias W. Seeger. Gaussian processes for machine learning. International Journal of Neural Sys- tems, 14(2):69-106, 2004.","title":"Gaussian processes for machine learning","authors":"Matthias W Seeger","year":"2004","venue":"International Journal of Neural Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shrivastava et al., 2017] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In 2017 CVPR, pages 2242- 2251, 2017.","title":"Learning from simulated and unsupervised images through adversarial training","authors":"Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, Russell Webb","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Villegas et al., 2017a] Ruben Villegas, Jimei Yang, Se- unghoon Hong, Xunyu Lin, and Honglak Lee. Decom- posing motion and content for natural video sequence pre- diction. In ICLR 2017, 2017.","title":"Decomposing motion and content for natural video sequence prediction","authors":"Villegas","year":"2017","venue":"ICLR 2017","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Villegas et al., 2017b] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical pre- diction. ICML, pages 3560-3569, 2017.","title":"Learning to generate long-term future via hierarchical prediction","authors":"Villegas","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vondrick et al., 2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. NIPS, pages 613-621, 2016.","title":"Generating videos with scene dynamics","authors":"Carl Vondrick, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Walker et al., 2017] Jacob Walker, Kenneth Marino, Abhi- nav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In 2017 ICCV, pages 3352-3361, 2017.","title":"The pose knows: Video forecasting by generating pose futures","authors":"Jacob Walker, Kenneth Marino, Abhinav Gupta, Martial Hebert","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2008] Jack M. Wang, David J. Fleet, and Aaron Hertzmann. Gaussian process dynamical models for hu- man motion. pattern recognition and machine intelligence, 30(2):283-298, 2008.","title":"Gaussian process dynamical models for human motion","authors":"Jack M Wang, David J Fleet, Aaron Hertzmann","year":"2008","venue":"pattern recognition and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yan et al., 2017] Yichao Yan, Jingwei Xu, Bingbing Ni, Wendong Zhang, and Xiaokang Yang. Skeleton-aided ar- ticulated motion generation. In Proceedings of the 2017 ACMMM, pages 199-207, 2017.","title":"Skeleton-aided articulated motion generation","authors":"Yan","year":"2017","venue":"Proceedings of the 2017 ACMMM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In 2017 ICCV, pages 2242-2251, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}