{"type":"ScholarlyArticle","title":"On Retrospecting Human Dynamics with Attention","authors":[{"name":"Minjing Dong","slug":"minjing-dong","given":"Minjing","family":"Dong","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"School of Computer Science , Faculty of Engineering , University of Sydney , Australia"}],"abstract":"Deep recurrent neural networks have achieved impressive success in forecasting human motion with a sequence to sequence architecture. However, forecasting in longer time horizons often leads to implausible human poses or converges to mean poses, because of error accumulation and difficulties in keeping track of longerterm information. To address these challenges, we propose to retrospect human dynamics with attention. A retrospection module is designed upon RNN to regularly retrospect past frames and correct mistakes in time. This significantly improves the memory of RNN and provides sufficient information for the decoder networks to generate longer term prediction. Moreover, we present a spatial attention module to explore and exploit cooperation among joints in performing a particular motion. Residual connections are also included to guarantee the performance of short term prediction. We evaluate the proposed algorithm on the largest and most challenging Human 3.6M dataset in the field. Experimental results demonstrate the necessity of investigating motion prediction in a self audit manner and the effectiveness of the proposed algorithm in both short term and long term predictions.","doi":"10.24963/ijcai.2019/100","url":"https://ijcai.org/proceedings/2019/100","pdf":"https://ijcai.org/proceedings/2019/0100.pdf","markdown":"https://ijcai.org/proceedings/2019/100.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/100","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":100,"pages":{"first":708,"last":714},"keywords":["Machine Learning: Time-series;Data Streams","Computer Vision: Motion and Tracking"],"videos":[],"references":[{"position":1,"raw":"[Akhter et al., 2012] Ijaz Akhter, Tomas Simon, Sohaib Khan, Iain Matthews, and Yaser Sheikh. Bilinear spatiotemporal basis models. ACM Trans. Graph., 31(2):17:1-17:12, April 2012.","title":"Bilinear spatiotemporal basis models","authors":"Ijaz Akhter, Tomas Simon, Sohaib Khan, Iain Matthews, Yaser Sheikh","year":"2012","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barsoum et al., 2017] Emad Barsoum, John Kender, and Zicheng Liu. HP-GAN: probabilistic 3d human motion prediction via GAN. CoRR, abs/1711.09561, 2017.","title":"HP-GAN: probabilistic 3d human motion prediction via GAN","authors":"Emad Barsoum, John Kender, Zicheng Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. Attention-gan for object trans- figuration in wild images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 164-180, 2018.","title":"Attention-gan for object transfiguration in wild images","authors":"Xinyuan Chen, Chang Xu, Xiaokang Yang, Dacheng Tao","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. CoRR, abs/1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, pages 4346-4354, Washington, DC, USA, 2015. IEEE Computer Society.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Con- volutional sequence to sequence learning. CoRR, abs/1705.03122, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghosh et al., 2017] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-term predictions. CoRR, abs/1704.02827, 2017.","title":"Learning human motion models for long-term predictions","authors":"Partha Ghosh, Jie Song, Emre Aksan, Otmar Hilliges","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gui et al., 2018] Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José MF Moura. Adversarial geometry-aware human motion prediction. In European Conference on Computer Vision (ECCV). Munich, Germany (September 2018), 2018.","title":"Adversarial geometry-aware human motion prediction","authors":"Liang-Yan, Yu-Xiong Gui, Xiaodan Wang, José Mf Liang, Moura","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ionescu et al., 2014] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural en- vironments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325-1339, July 2014.","title":"Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments","authors":"C Ionescu, D Papava, V Olaru, C Sminchisescu","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jain et al., 2015] Ashesh Jain, Amir Roshan Zamir, Sil- vio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. CoRR, abs/1511.05298, 2015.","title":"Structural-rnn: Deep learning on spatio-temporal graphs","authors":"Ashesh Jain, Silvio Amir Roshan Zamir, Ashutosh Savarese, Saxena","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lehrmann et al., 2014] Andreas M. Lehrmann, Peter V. Gehler, and Sebastian Nowozin. Efficient nonlinear markov models for human motion. 2014 IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1314-1321, 2014.","title":"Efficient nonlinear markov models for human motion","authors":"Andreas M Lehrmann, Peter V Gehler, Sebastian Nowozin","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. Convolutional sequence to sequence model for human dynamics. CoRR, abs/1805.00655, 2018.","title":"Convolutional sequence to sequence model for human dynamics","authors":"Chen Li, Zhen Zhang, Wee Sun Lee, Gim Hee, Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction using recurrent neural networks. CoRR, abs/1705.02445, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nie et al., 2018] Xuecheng Nie, Jiashi Feng, Yiming Zuo, and Shuicheng Yan. Human pose estimation with parsing induced learner. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2100- 2108, 2018.","title":"Human pose estimation with parsing induced learner","authors":"Nie","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pavlovic et al., 2000] Vladimir Pavlovic, James M. Rehg, and John MacCormick. Learning switching linear mod- els of human motion. In Proceedings of the 13th Interna- tional Conference on Neural Information Processing Sys- tems, NIPS'00, pages 942-948, Cambridge, MA, USA, 2000. MIT Press.","title":"Learning switching linear models of human motion","authors":"Vladimir Pavlovic, James M Rehg, John Maccormick","year":"2000","venue":"Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS'00","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104- 3112.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Curran Associates, Inc., 2014.","title":null,"authors":"Curran Associates, Inc","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tekin et al., 2016] Bugra Tekin, Pablo Márquez-Neila, Mathieu Salzmann, and Pascal Fua. Fusing 2d uncertainty and 3d cues for monocular body pose estimation. CoRR, abs/1611.05708, 2016.","title":"Fusing 2d uncertainty and 3d cues for monocular body pose estimation","authors":"Pablo Bugra Tekin, Mathieu Márquez-Neila, Pascal Salzmann, Fua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Urtasun et al., 2008] Raquel Urtasun, David J. Fleet, An- dreas Geiger, Jovan Popović, Trevor J. Darrell, and Neil D. Lawrence. Topologically-constrained latent variable mod- els. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 1080-1087, New York, NY, USA, 2008. ACM.","title":"Topologically-constrained latent variable models","authors":"Raquel Urtasun, David J Fleet, Andreas Geiger, Jovan Popović, Trevor J Darrell, Neil D Lawrence","year":"2008","venue":"Proceedings of the 25th International Conference on Machine Learning, ICML '08","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2013] Chang Xu, Dacheng Tao, Yangxi Li, and Chao Xu. Large-margin multi-view gaussian process for image classification. In Proceedings of the Fifth Interna- tional Conference on Internet Multimedia Computing and Service, pages 7-12. ACM, 2013.","title":"Large-margin multi-view gaussian process for image classification","authors":"Chang Xu, Dacheng Tao, Yangxi Li, Chao Xu","year":"2013","venue":"Proceedings of the Fifth International Conference on Internet Multimedia Computing and Service","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}