{"type":"ScholarlyArticle","title":"A Deep Bi-directional Attention Network for Human Motion Recovery","authors":[{"name":"Qiongjie Cui","slug":"qiongjie-cui","given":"Qiongjie","family":"Cui","affiliation":"Nanjing University of Science and Technology , Nanjing , China"},{"name":"Huaijiang Sun","slug":"huaijiang-sun","given":"Huaijiang","family":"Sun","affiliation":"Nanjing University of Science and Technology , Nanjing , China"},{"name":"Yupeng Li","slug":"yupeng-li","given":"Yupeng","family":"Li","affiliation":"Nanjing University of Science and Technology , Nanjing , China"},{"name":"Yue Kong","slug":"yue-kong","given":"Yue","family":"Kong","affiliation":"Nanjing University of Science and Technology , Nanjing , China"}],"abstract":"Human motion capture (mocap) data, recording the movement of markers attached to specific joints, has gradually become the most popular solution of animation production. However, the raw motion data are often corrupted due to joint occlusion, marker shedding and the lack of equipment precision, which severely limits the performance in real-world applications. Since human motion is essentially a sequential data, the latest methods resort to variants of long short-time memory network (LSTM) to solve related problems, but most of them tend to obtain visually unreasonable results. This is mainly because these methods hardly capture long-term dependencies and cannot explicitly utilize relevant context, especially in long sequences. To address these issues, we propose a deep bi-directional attention network (BAN) which can not only capture the long-term dependencies but also adaptively extract relevant information at each time step. Moreover, the proposed model, embedded attention mechanism in the bi-directional LSTM (BLSTM) structure at the encoding and decoding stages, can decide where to borrow information and use it to recover corrupted frame effectively. Extensive experiments on CMU database demonstrate that the proposed model consistently outperforms other state-of-the-art methods in terms of recovery accuracy and visualization.","doi":"10.24963/ijcai.2019/99","url":"https://ijcai.org/proceedings/2019/99","pdf":"https://ijcai.org/proceedings/2019/0099.pdf","markdown":"https://ijcai.org/proceedings/2019/99.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/99","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":99,"pages":{"first":701,"last":707},"keywords":["Computer Vision: 2D and 3D Computer Vision","Machine Learning: Deep Learning","Computer Vision: Motion and Tracking","Humans and AI: Human-Computer Interaction","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine transla- tion by jointly learning to align and translate. CoRR, abs/1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bütepage et al., 2017] Judith Bütepage, Michael J. Black, Danica Kragic, and Hedvig Kjellström. Deep represen- tation learning for human motion prediction and classifi- cation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1591-1599, 2017.","title":"Deep representation learning for human motion prediction and classification","authors":"Judith Bütepage, Michael J Black, Danica Kragic, Hedvig Kjellström","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2019] Qiongjie Cui, Beijia Chen, and Huaijiang Sun. Nonlocal low-rank regularization for human motion recovery based on similarity analysis. Information Sci- ences, 2019.","title":"Nonlocal low-rank regularization for human motion recovery based on similarity analysis","authors":"Cui","year":"2019","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. 2015 IEEE Inter- national Conference on Computer Vision (ICCV), pages 4346-4354, 2015.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gui et al., 2018] Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José M. F. Moura. Adversarial geometry-aware human motion prediction. In ECCV, 2018.","title":"Adversarial geometry-aware human motion prediction","authors":"Liang-Yan, Yu-Xiong Gui, Xiaodan Wang, Liang, M F José, Moura","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Holden et al., 2017] Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character con- trol. ACM Trans. Graph., 36:42:1-42:13, 2017.","title":"Phase-functioned neural networks for character control","authors":"Daniel Holden, Taku Komura, Jun Saito","year":"2017","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Holden, 2018] Daniel Holden. Robust solving of optical motion capture data by denoising. ACM Trans. Graph., 37:165:1-165:12, 2018.","title":"Robust solving of optical motion capture data by denoising","authors":"Daniel Holden","year":"2018","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2018] Wenyu Hu, Zhao Wang, Shuang Liu, Xi- aosong Yang, Gaohang Yu, and Jian Jun Zhang. Motion capture data completion via truncated nuclear norm regu- larization. IEEE Signal Processing Letters, 25:258-262, 2018.","title":"Motion capture data completion via truncated nuclear norm regularization","authors":"Wenyu Hu, Zhao Wang, Shuang Liu, Xiaosong Yang, Gaohang Yu, Jian Jun Zhang","year":"2018","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kucherenko and Kjellström, 2018] Taras Kucherenko and Hedvig Kjellström. A neural network approach to miss- ing marker reconstruction. CoRR, abs/1803.02665, 2018.","title":"A neural network approach to missing marker reconstruction","authors":"Kjellström Kucherenko, Taras Kucherenko, Hedvig Kjellström","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lai et al., 2011] R. Y. Q. Lai, P. C. Yuen, and K. K. W. Lee. Motion capture data completion and denoising by singu- lar value thresholding. Proc Eurographics Association, 11(3):924-929, 2011.","title":"Motion capture data completion and denoising by singular value thresholding","authors":"R Y Q Lai, P C Yuen, K K W Lee","year":"2011","venue":"Proc Eurographics Association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2018] Xuequan Lu, Honghua Chen, Sai-Kit Ye- ung, Zhigang Deng, and Wenzhi Chen. Unsupervised ar- ticulated skeleton extraction from point set sequences cap- tured by a single depth camera. In AAAI, 2018.","title":"Unsupervised articulated skeleton extraction from point set sequences captured by a single depth camera","authors":"Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mall et al., 2017] Utkarsh Mall, G Roshan Lal, Siddhartha Chaudhuri, and Parag Chaudhuri. A deep recurrent frame- work for cleaning motion capture data. arXiv preprint arXiv:1712.03380, 2017.","title":"A deep recurrent framework for cleaning motion capture data","authors":"Utkarsh Mall, Roshan Lal, Siddhartha Chaudhuri, Parag Chaudhuri","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Martinez et al., 2017] Julieta Martinez, Michael J. Black, and Javier Romero. On human motion prediction us- ing recurrent neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4674-4683, 2017.","title":"On human motion prediction using recurrent neural networks","authors":"Julieta Martinez, Michael J Black, Javier Romero","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Cheng, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In IJCAI, 2017.","title":"A dual-stage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Cheng, Guofei Cheng, Garrison W Jiang, Cottrell","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/366"},{"position":17,"raw":"[Ruiz et al., 2018] Alejandro Hernandez Ruiz, Juergen Gall, and Francesc Moreno-Noguer. Human motion pre- diction via spatio-temporal inpainting. arXiv preprint arXiv:1812.05478, 2018.","title":"Human motion prediction via spatio-temporal inpainting","authors":"Alejandro Hernandez Ruiz, Juergen Gall, Francesc Moreno-Noguer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rumelhart et al., 1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323:533-536, 1986.","title":"Learning representations by back-propagating errors","authors":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams","year":"1986","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E. Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan et al., 2013] Cheen-Hau Tan, Junhui Hou, and Lap- Pui Chau. Human motion capture data recovery using trajectory-based matrix completion. Electronics Letters, 49(12):752-754, 2013.","title":"Human motion capture data recovery using trajectory-based matrix completion","authors":"Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau","year":"2013","venue":"Electronics Letters","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xia et al., 2018] Guiyu Xia, Huaijiang Sun, Beijia Chen, Qingshan Liu, Lei Feng, Guoqing Zhang, and Renlong Hang. Nonlinear low-rank matrix completion for human motion recovery. IEEE Transactions on Image Processing, 27:3011-3024, 2018.","title":"Nonlinear low-rank matrix completion for human motion recovery","authors":"Guiyu Xia, Huaijiang Sun, Beijia Chen, Qingshan Liu, Lei Feng, Guoqing Zhang, Renlong Hang","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao et al., 2011] Jun Xiao, Yinfu Feng, and Wenyuan Hu. Predicting missing markers in human motion capture using l1-sparse representation. Computer Animation and Virtual Worlds, 22(2-3):221-228, 2011.","title":"Predicting missing markers in human motion capture using l1-sparse representation","authors":"Xiao","year":"2011","venue":"Computer Animation and Virtual Worlds","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J. Smola, and Eduard H. Hovy. Hi- erarchical attention networks for document classification. In HLT-NAACL, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4651- 4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2016] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. Attention- based bidirectional long short-term memory networks for relation classification. In ACL, 2016.","title":"Attentionbased bidirectional long short-term memory networks for relation classification","authors":"Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, Bo Xu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2018] Xiaowei Zhou, Sikang Liu, Georgios Pavlakos, Vijay Kumar, and Kostas Daniilidis. Human mo- tion capture using a drone. 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pages 2027- 2033, 2018.","title":"Human motion capture using a drone","authors":"Xiaowei Zhou, Sikang Liu, Georgios Pavlakos, Vijay Kumar, Kostas Daniilidis","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}