{"type":"ScholarlyArticle","title":"DBDNet: Learning Bi-directional Dynamics for Early Action Prediction","authors":[{"name":"Guoliang Pang","slug":"guoliang-pang","given":"Guoliang","family":"Pang","affiliation":"Sun Yat-sen University , China"},{"name":"Xionghui Wang","slug":"xionghui-wang","given":"Xionghui","family":"Wang","affiliation":"Sun Yat-sen University , China"},{"name":"Jian-Fang Hu","slug":"jian-fang-hu","given":"Jian-Fang","family":"Hu","affiliation":"Sun Yat-sen University , China; Guangdong Province Key Laboratory of Information Security Technology , China"},{"name":"Qing Zhang","slug":"qing-zhang","given":"Qing","family":"Zhang","affiliation":"Sun Yat-sen University , China"},{"name":"Wei-Shi Zheng","slug":"wei-shi-zheng","given":"Wei-Shi","family":"Zheng","affiliation":"Sun Yat-sen University , China; Key Laboratory of Machine Intelligence and Advanced Computing , Ministry of Education , China"}],"abstract":"Predicting future actions from observed partial videos is very challenging as the missing future is uncertain and sometimes has multiple possibilities. To obtain a reliable future estimation, a novel encoder-decoder architecture is proposed for integrating the tasks of synthesizing future motions from observed videos and reconstructing observed motions from synthesized future motions in an unified framework, which can capture the bi-directional dynamics depicted in partial videos along the temporal (past-to-future) direction and reverse chronological (future-back-to-past) direction. We then employ a bi-directional long short-term memory (Bi-LSTM) architecture to exploit the learned bi-directional dynamics for predicting early actions. Our experiments on two benchmark action datasets show that learning bi-directional dynamics benefits the early action prediction and our system clearly outperforms the state-of-the-art methods.","doi":"10.24963/ijcai.2019/126","url":"https://ijcai.org/proceedings/2019/126","pdf":"https://ijcai.org/proceedings/2019/0126.pdf","markdown":"https://ijcai.org/proceedings/2019/126.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/126","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":126,"pages":{"first":897,"last":903},"keywords":["Computer Vision: Computer Vision","Computer Vision: Action Recognition","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4724-4733, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fragkiadaki et al., 2015] Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346-4354, 2015.","title":"Recurrent network models for human dynamics","authors":"Katerina Fragkiadaki, Sergey Levine, Panna Felsen, Jitendra Malik","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ghosh et al., 2017] Partha Ghosh, Jie Song, Emre Aksan, and Otmar Hilliges. Learning human motion models for long-term predictions. In International Conference on 3D Vision, 2017.","title":"Learning human motion models for long-term predictions","authors":"Partha Ghosh, Jie Song, Emre Aksan, Otmar Hilliges","year":"2017","venue":"International Conference on 3D Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Graves and Schmidhuber, 2005] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602-610, 2005.","title":"Framewise phoneme classification with bidirectional lstm and other neural network architectures","authors":"Schmidhuber Graves, Alex Graves, Jürgen Schmidhuber","year":"2005","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hara et al., 2018] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the his- tory of 2d cnns and imagenet? In IEEE Conference on Computer Vision and Pattern Recognition, pages 6546- 6555, 2018.","title":"Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?","authors":"Hara","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2018a] Jian-Fang Hu, Wei-Shi Zheng, Lianyang Ma, Gang Wang, Jianhuang Lai, and Jianguo Zhang. Early action prediction by soft regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2018.","title":"Early action prediction by soft regression","authors":"Hu","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018b] Jian-Fang Hu, Wei-Shi Zheng, Jiahui Pan, Jianhuang Lai, and Jianguo Zhang. Deep bilinear learning for rgb-d action recognition. In European Conference on Computer Vision, pages 346-362, 2018.","title":"Deep bilinear learning for rgb-d action recognition","authors":"Hu","year":"2018","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jain et al., 2016] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 5308-5317, 2016.","title":"Structural-rnn: Deep learning on spatio-temporal graphs","authors":"Ashesh Jain, Silvio Amir R Zamir, Ashutosh Savarese, Saxena","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kay et al., 2017] Will Kay, João Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kong et al., 2014] Yu Kong, Dmitry Kit, and Yun Fu. A dis- criminative model with multiple temporal scales for action prediction. In European Conference on Computer Vision, pages 596-611, 2014.","title":"A discriminative model with multiple temporal scales for action prediction","authors":"Yu Kong, Dmitry Kit, Yun Fu","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kong et al., 2017] Yu Kong, Zhiqiang Tao, and Yun Fu. Deep sequential context networks for action prediction. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1473-1481, 2017.","title":"Deep sequential context networks for action prediction","authors":"Yu Kong, Zhiqiang Tao, Yun Fu","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kong et al., 2018] Yu Kong, Shangqian Gao, Bin Sun, and Yun Fu. Action prediction from videos via memorizing hard-to-predict samples. In AAAI Conference on Artificial Intelligence, 2018.","title":"Action prediction from videos via memorizing hard-to-predict samples","authors":"Yu Kong, Shangqian Gao, Bin Sun, Yun Fu","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lai et al., 2018] Shaofan Lai, Wei-Shi Zheng, Jian-Fang Hu, and Jianguo Zhang. Global-local temporal saliency action prediction. IEEE Transactions on Image Process- ing, 27(5):2272-2285, 2018.","title":"Global-local temporal saliency action prediction","authors":"Shaofan Lai, Wei-Shi Zheng, Jian-Fang Hu, Jianguo Zhang","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lan et al., 2014] Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In European Conference on Computer Vision, pages 689-704. 2014.","title":"A hierarchical representation for future action prediction","authors":"Tian Lan, Tsung-Chuan Chen, Silvio Savarese","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Yang Li, Kan Li, and Xinxin Wang. Deeply- supervised cnn model for action recognition with trainable feature aggregation. In International Joint Conference on Artificial Intelligence, 2018.","title":"Deeplysupervised cnn model for action recognition with trainable feature aggregation","authors":"Yang Li, Kan Li, Xinxin Wang","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2016] Shugao Ma, Leonid Sigal, and Stan Sclaroff. Learning activity progression in lstms for ac- tivity detection and early detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1942- 1950, 2016.","title":"Learning activity progression in lstms for activity detection and early detection","authors":"Ma","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ryoo, 2011] Michael S Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In IEEE International Conference on Computer Vision, pages 1036-1043, 2011.","title":"Human activity prediction: Early recognition of ongoing activities from streaming videos","authors":"Michael S Ryoo","year":"2011","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural infor- mation processing systems, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tang et al., 2018] Yongyi Tang, Lin Ma, Wei Liu, and Wei- Shi Zheng. Long-term human motion prediction by mod- eling motion context and enhancing motion dynamics. In International Joint Conference on Artificial Intelligence, 2018.","title":"Long-term human motion prediction by modeling motion context and enhancing motion dynamics","authors":"Yongyi Tang, Lin Ma, Wei Liu, Wei-Shi Zheng","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/130"},{"position":20,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In IEEE International Conference on Computer Vision, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Walker et al., 2017] Jacob Walker, Kenneth Marino, Ab- hinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. CoRR, abs/1705.00053, 2017.","title":"The pose knows: Video forecasting by generating pose futures","authors":"Jacob Walker, Kenneth Marino, Abhinav Gupta, Martial Hebert","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European Con- ference on Computer Vision, pages 20-36. Springer, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2018] Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. Auto-conditioned recurrent networks for extended complex human motion synthesis. In International Conference on Learning Rep- resentations, 2018.","title":"Auto-conditioned recurrent networks for extended complex human motion synthesis","authors":"Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, Hao Li","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}