{"type":"ScholarlyArticle","title":"Deep Optical Flow Estimation Via Multi-Scale Correspondence Structure Learning","authors":[{"name":"Shanshan Zhao","slug":"shanshan-zhao","given":"Shanshan","family":"Zhao","affiliation":"Zhejiang University , Hangzhou , China"},{"name":"Xi Li","slug":"xi-li","given":"Xi","family":"Li","affiliation":"Zhejiang University , Hangzhou , China; Alibaba-Zhejiang University Joint Institute of Frontier Technologies , Hangzhou , China"},{"name":"Omar El Farouk Bourahla","slug":"omar-el-farouk-bourahla","given":"Omar El Farouk","family":"Bourahla","affiliation":"Zhejiang University , Hangzhou , China"}],"abstract":"As an important and challenging problem in computer vision, learning based optical flow estimation aims to discover the intrinsic correspondence structure between two adjacent video frames through statistical learning. Therefore, a key issue to solve in this area is how to effectively model the multi-scale correspondence structure properties in an adaptive end-to-end learning fashion. Motivated by this observation, we propose an end-to-end multi-scale correspondence structure learning (MSCSL) approach for optical flow estimation. In principle, the proposed MSCSL approach is capable of effectively capturing the multi-scale inter-image-correlation correspondence structures within a multi-level feature space from deep learning. Moreover, the proposed MSCSL approach builds a spatial Conv-GRU neural network model to adaptively model the intrinsic dependency relationships among these multi-scale correspondence structures. Finally, the above procedures for correspondence structure learning and multi-scale dependency modeling are implemented in a unified end-to-end deep learning framework. Experimental results on several benchmark datasets demonstrate the effectiveness of the proposed approach.","doi":"10.24963/ijcai.2017/488","url":"https://ijcai.org/proceedings/2017/488","pdf":"https://ijcai.org/proceedings/2017/0488.pdf","markdown":"https://ijcai.org/proceedings/2017/488.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/488","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":488,"pages":{"first":3490,"last":3496},"keywords":["Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Ahmadi and Patras, 2016] Aria Ahmadi and Ioannis Patras. Unsupervised convolutional neural networks for motion estimation. In Image Processing (ICIP), 2016 IEEE Inter- national Conference on, pages 1629-1633. IEEE, 2016.","title":"Aria Ahmadi and Ioannis Patras. Unsupervised convolutional neural networks for motion estimation","authors":"Patras Ahmadi","year":"2016","venue":"Image Processing (ICIP), 2016 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bailer et al., 2015] Christian Bailer, Bertram Taetz, and Di- dier Stricker. Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 4015-4023, 2015.","title":"Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation","authors":"Bailer","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baker et al., 2011] Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92(1):1-31, 2011.","title":"A database and evaluation methodology for optical flow","authors":"Simon Baker, Daniel Scharstein, Stefan Lewis, Michael J Roth, Richard Black, Szeliski","year":"2011","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ballas et al., 2015] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional net- works for learning video representations. arXiv preprint arXiv:1511.06432, 2015.","title":"Delving deeper into convolutional networks for learning video representations","authors":"Nicolas Ballas, Li Yao, Chris Pal, Aaron Courville","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bao et al., 2014] Linchao Bao, Qingxiong Yang, and Hailin Jin. Fast edge-preserving patchmatch for large displace- ment optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3534- 3541, 2014.","title":"Fast edge-preserving patchmatch for large displacement optical flow","authors":"Bao","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brox and Malik, 2011] Thomas Brox and Jitendra Malik. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE transactions on pat- tern analysis and machine intelligence, 33(3):500-513, 2011.","title":"Large displacement optical flow: descriptor matching in variational motion estimation","authors":"Malik ; Brox, Thomas Brox, Jitendra Malik","year":"2011","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Butler et al., 2012] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, pages 611-625. Springer, 2012.","title":"A naturalistic open source movie for optical flow evaluation","authors":"J Daniel, Jonas Butler, Garrett B Wulff, Michael J Stanley, Black","year":"2012","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2015] Alexey Dosovitskiy, Philipp Fis- chery, Eddy Ilg, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox, et al. Flownet: Learning optical flow with convolutional net- works. In 2015 IEEE International Conference on Com- puter Vision (ICCV), pages 2758-2766. IEEE, 2015.","title":"Flownet: Learning optical flow with convolutional networks","authors":"Alexey Dosovitskiy, Philipp Fischery, Eddy Ilg, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, Thomas Brox","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3354-3361. IEEE, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer network- s. In Advances in Neural Information Processing Systems, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jason et al., 2016] J Yu Jason, Adam W Harley, and Kon- stantinos G Derpanis. Back to basics: Unsupervised learn- ing of optical flow via brightness constancy and motion smoothness. In Computer Vision-ECCV 2016 Workshops, pages 3-10. Springer, 2016.","title":"Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness","authors":"Jason Yu, Adam W Harley, Konstantinos G Derpanis","year":"2016","venue":"Computer Vision-ECCV 2016 Workshops","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolution- al architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675-678. ACM, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ranjan and Black, 2016] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. arXiv preprint arXiv:1611.00850, 2016.","title":"Optical flow estimation using a spatial pyramid network","authors":"Anurag Ranjan, Michael J Black","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Revaud et al., 2015] Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Epicflow: Edge- preserving interpolation of correspondences for optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1164-1172, 2015.","title":"Epicflow: Edgepreserving interpolation of correspondences for optical flow","authors":"Revaud","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Siam et al., 2016] Mennatullah Siam, Sepehr Valipour, Martin Jagersand, and Nilanjan Ray. Convolutional gated recurrent networks for video segmentation. arXiv preprint arXiv:1611.05435, 2016.","title":"Mennatullah Siam, Sepehr Valipour","authors":"Siam","year":"2016","venue":"Convolutional gated recurrent networks for video segmentation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Weinzaepfel et al., 2013] Philippe Weinzaepfel, Jerome Re- vaud, Zaid Harchaoui, and Cordelia Schmid. Deepflow: Large displacement optical flow with deep matching. In Proceedings of the IEEE International Conference on Computer Vision, pages 1385-1392, 2013.","title":"Deepflow: Large displacement optical flow with deep matching","authors":"Weinzaepfel","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yang and Li, 2015] Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1019-1027, 2015.","title":"Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model","authors":"Yang, Li","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhu et al., 2017] Yi Zhu, Zhenzhong Lan, Shawn Newsam, and Alexander G Hauptmann. Guided optical flow learn- ing. arXiv preprint arXiv:1702.02295, 2017.","title":"Guided optical flow learning","authors":"Yi Zhu, Zhenzhong Lan, Shawn Newsam, Alexander G Hauptmann","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}