{"type":"ScholarlyArticle","title":"PredCNN: Predictive Learning with Cascade Convolutions","authors":[{"name":"Ziru Xu","slug":"ziru-xu","given":"Ziru","family":"Xu","affiliation":"School of Software , Tsinghua University , China National Engineering Laboratory for Big Data Software Beijing Key Laboratory for Industrial Big Data System and Application"},{"name":"Yunbo Wang","slug":"yunbo-wang","given":"Yunbo","family":"Wang","affiliation":"School of Software , Tsinghua University , China National Engineering Laboratory for Big Data Software Beijing Key Laboratory for Industrial Big Data System and Application"},{"name":"Mingsheng Long","slug":"mingsheng-long","given":"Mingsheng","family":"Long","affiliation":"School of Software , Tsinghua University , China National Engineering Laboratory for Big Data Software Beijing Key Laboratory for Industrial Big Data System and Application"},{"name":"Jianmin Wang","slug":"jianmin-wang","given":"Jianmin","family":"Wang","affiliation":"School of Software , Tsinghua University , China National Engineering Laboratory for Big Data Software Beijing Key Laboratory for Industrial Big Data System and Application"}],"abstract":"Predicting future frames in videos remains an unsolved but challenging problem. Mainstream recurrent models suffer from huge memory usage and computation cost, while convolutional models are unable to effectively capture the temporal dependencies between consecutive video frames. To tackle this problem, we introduce an entirely CNN-based architecture, PredCNN, that models the dependencies between the next frame and the sequential video inputs. Inspired by the core idea of recurrent models that previous states have more transition operations than future states, we design a cascade multiplicative unit (CMU) that provides relatively more operations for previous video frames. This newly proposed unit enables PredCNN to predict future spatiotemporal data without any recurrent chain structures, which eases gradient propagation and enables a fully paralleled optimization. We show that PredCNN outperforms the state-of-the-art recurrent models for video prediction on the standard Moving MNIST dataset and two challenging crowd flow prediction datasets, and achieves a faster training speed and lower memory footprint.","doi":"10.24963/ijcai.2018/408","url":"https://ijcai.org/proceedings/2018/408","pdf":"https://ijcai.org/proceedings/2018/0408.pdf","markdown":"https://ijcai.org/proceedings/2018/408.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/408","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":408,"pages":{"first":2940,"last":2947},"keywords":["Machine Learning: Deep Learning","Computer Vision: Video: Events, Activities and Surveillance"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915, 2016.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet and others, 2015] Franc ¸ois Chollet et al. Keras. https://github.com/keras-team/keras, 2015.","title":"Franc ¸ois Chollet et al. Keras","authors":"Others Chollet","year":"2015","venue":null,"doi":null,"url":"https://github.com/keras-team/keras,2015","internal_url":null},{"position":4,"raw":"[Finn et al., 2016] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, 2016.","title":"Unsupervised learning for physical interaction through video prediction","authors":"Finn","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pougetabadie, Mehdi Mirza, Bing Xu, David Warde- farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. NIPS, 3:2672-2680, 2014.","title":"Generative adversarial networks","authors":"Ian J Goodfellow, Jean Pougetabadie, Mehdi Mirza, Bing Xu, David Wardefarley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kalchbrenner et al., 2017] Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In ICML, 2017.","title":"Video pixel networks","authors":"Nal Kalchbrenner, Aaron Van Den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lotter et al., 2017] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations (ICLR), 2017.","title":"Deep predictive coding networks for video prediction and unsupervised learning","authors":"William Lotter, David Gabriel Kreiman, Cox","year":"2017","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mathieu et al., 2016] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction be- yond mean square error. In ICLR, 2016.","title":"Deep multi-scale video prediction beyond mean square error","authors":"Michael Mathieu, Camille Couprie, Yann Lecun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2010] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Re- current neural network based language model. In Eleventh Annual Conference of the International Speech Communi- cation Association, 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"Eleventh Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Patraucean et al., 2016] Viorica Patraucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differentiable memory. In ICLR Workshop, 2016.","title":"Spatio-temporal video autoencoder with differentiable memory","authors":"Ankur Viorica Patraucean, Roberto Handa, Cipolla","year":"2016","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ranzato et al., 2014] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014.","title":"MarcAurelio Ranzato","authors":"Ranzato","year":"2014","venue":"Video (language) modeling: a baseline for generative models of natural videos","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rumelhart et al., 1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1, 1988.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1988","venue":"Cognitive modeling","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi et al., 2015] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In NIPS, pages 802-810, 2015.","title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","authors":"Shi","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Srivastava et al., 2015] Nitish Srivastava, Elman Mansi- mov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using lstms. In ICML, 2015.","title":"Unsupervised learning of video representations using lstms","authors":"Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Den Oord et al., 2016a] Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Ko- ray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord Van Den","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[van den Oord et al., 2016b] Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In NIPS, pages 4790-4798, 2016.","title":"Conditional image generation with pixelcnn decoders","authors":"Oord Van Den","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Villegas et al., 2017] Ruben Villegas, Jimei Yang, Se- unghoon Hong, Xunyu Lin, and Honglak Lee. Decom- posing motion and content for natural video sequence pre- diction. In International Conference on Learning Repre- sentations (ICLR), 2017.","title":"Decomposing motion and content for natural video sequence prediction","authors":"Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, Honglak Lee","year":"2017","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. Predrnn: Recurrent neural networks for predictive learning using spatiotempo- ral lstms. In Advances in Neural Information Processing Systems, pages 879-888, 2017.","title":"Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms","authors":"Wang","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Werbos, 1990] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550-1560, 1990.","title":"Backpropagation through time: what it does and how to do it","authors":"Werbos Paul","year":"1990","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu and Koltun, 2015] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016] Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, and Xiuwen Yi. Dnn-based prediction model for spatio-temporal data. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, page 92. ACM, 2016.","title":"Dnn-based prediction model for spatio-temporal data","authors":"Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, Xiuwen Yi","year":"2016","venue":"Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In AAAI, pages 1655-1661, 2017.","title":"Deep spatio-temporal residual networks for citywide crowd flows prediction","authors":"Junbo Zhang, Yu Zheng, Dekang Qi","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng, 2015] Yu Zheng. Methodologies for cross-domain data fusion: An overview. IEEE transactions on big data, 1(1):16-34, 2015.","title":"Methodologies for cross-domain data fusion: An overview","authors":"Yu Zheng","year":"2015","venue":"IEEE transactions on big data","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}