{"type":"ScholarlyArticle","title":"Layered Optical Flow Estimation Using a Deep Neural Network with a Soft Mask","authors":[{"name":"Xi Zhang","slug":"xi-zhang","given":"Xi","family":"Zhang","affiliation":"Illinois Institute of Technology Chicago , IL 60616"},{"name":"Di Ma","slug":"di-ma","given":"Di","family":"Ma","affiliation":"Illinois Institute of Technology Chicago , IL 60616"},{"name":"Xu Ouyang","slug":"xu-ouyang","given":"Xu","family":"Ouyang","affiliation":"Illinois Institute of Technology Chicago , IL 60616"},{"name":"Shanshan Jiang","slug":"shanshan-jiang","given":"Shanshan","family":"Jiang","affiliation":"Illinois Institute of Technology Chicago , IL 60616"},{"name":"Lin Gan","slug":"lin-gan","given":"Lin","family":"Gan","affiliation":"Illinois Institute of Technology Chicago , IL 60616"},{"name":"Gady Agam","slug":"gady-agam","given":"Gady","family":"Agam","affiliation":"Illinois Institute of Technology Chicago , IL 60616"}],"abstract":"Using a layered representation for motion estimation has the advantage of being able to cope with discontinuities and occlusions. In this paper, we learn to estimate optical flow by combining a layered motion representation with deep learning. Instead of pre-segmenting the image to layers, the proposed approach automatically generates a layered representation of optical flow using the proposed soft-mask module. The essential components of the soft-mask module are maxout and fuse operations, which enable a disjoint layered representation of optical flow and more accurate flow estimation. We show that by using masks the motion estimate results in a quadratic function of input features in the output layer. The proposed soft-mask module can be added to any existing optical flow estimation networks by replacing their flow output layer. In this work, we use FlowNet as the base network to which we add the soft-mask module. The resulting network is tested on three well-known benchmarks with both supervised and unsupervised flow estimation tasks. Evaluation results show that the proposed network achieve better results compared with the original FlowNet.","doi":"10.24963/ijcai.2018/163","url":"https://ijcai.org/proceedings/2018/163","pdf":"https://ijcai.org/proceedings/2018/0163.pdf","markdown":"https://ijcai.org/proceedings/2018/163.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/163","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":163,"pages":{"first":1170,"last":1176},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Computer Vision: Motion and Tracking"],"videos":[],"references":[{"position":1,"raw":"[Ahmadi and Patras, 2016] Aria Ahmadi and Ioannis Patras. Unsupervised convolutional neural networks for motion estimation. In Image Processing (ICIP), 2016 IEEE Inter- national Conference on, pages 1629-1633. IEEE, 2016.","title":"Aria Ahmadi and Ioannis Patras. Unsupervised convolutional neural networks for motion estimation","authors":"Patras Ahmadi","year":"2016","venue":"Image Processing (ICIP), 2016 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Butler et al., 2012] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, Eu- ropean Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pages 611-625. Springer-Verlag, October 2012.","title":"A naturalistic open source movie for optical flow evaluation","authors":"D J Butler, J Wulff, G B Stanley, M J Black","year":"2012","venue":"European Conf. on Computer Vision (ECCV), Part IV, LNCS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Darrell and Pentland, 1991] Trevor Darrell and Alexander Pentland. Robust estimation of a multi-layered motion representation. In Visual Motion, 1991., Proceedings of the IEEE Workshop on, pages 173-178. IEEE, 1991.","title":"Robust estimation of a multi-layered motion representation","authors":"Trevor Darrell, Alexander Pentland","year":"1991","venue":"Proceedings of the IEEE Workshop on","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2015] A. Dosovitskiy, P. Fischery, E. Ilg, P. Häusser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cre- mers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In 2015 IEEE International Con- ference on Computer Vision (ICCV), pages 2758-2766, Dec 2015.","title":"Flownet: Learning optical flow with convolutional networks","authors":"A Dosovitskiy, P Fischery, E Ilg, P Häusser, C Hazirbas, V Golkov, P V D Smagt, D Cremers, T Brox","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Flynn et al., 2016] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to pre- dict new views from the world's imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5515-5524, 2016.","title":"Deepstereo: Learning to predict new views from the world's imagery","authors":"John Flynn, Ivan Neulander, James Philbin, Noah Snavely","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Geiger et al., 2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3354-3361. IEEE, 2012.","title":"Are we ready for autonomous driving? the kitti vision benchmark suite","authors":"Andreas Geiger, Philip Lenz, Raquel Urtasun","year":"2012","venue":"Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2013] Ian J. Goodfellow, David Warde- Farley, Mehdi Mirza, Aaron Courville, and Yoshua Ben- gio. Maxout networks. In Proceedings of the 30th Inter- national Conference on International Conference on Ma- chine Learning -Volume 28, ICML'13, pages III-1319- III-1327. JMLR.org, 2013.","title":"Maxout networks","authors":"Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio","year":"2013","venue":"Proceedings of the 30th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Horn and Schunck, 1981] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3):185-203, 1981.","title":"Determining optical flow","authors":"K P Berthold, Brian G Horn, Schunck","year":"1981","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hur and Roth, 2016] Junhwa Hur and Stefan Roth. Joint Optical Flow and Temporally Consistent Semantic Seg- mentation, pages 163-177. Springer International Publish- ing, Cham, 2016.","title":"Junhwa Hur and Stefan Roth","authors":"Roth Hur","year":"2016","venue":"Joint Optical Flow and Temporally Consistent Semantic Segmentation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In The IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), July 2017.","title":"Flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jepson and Black, 1993] A. Jepson and M. J. Black. Mix- ture models for optical flow computation. In Proceed- ings of IEEE Conference on Computer Vision and Pattern Recognition, pages 760-761, Jun 1993.","title":"Mixture models for optical flow computation","authors":"A Jepson, M J Black","year":"1993","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431- 3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ranjan and Black, 2017] Anurag Ranjan and Michael J. Black. Optical flow estimation using a spatial pyramid net- work. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.","title":"Optical flow estimation using a spatial pyramid network","authors":"Anurag Ranjan, Michael J Black","year":"2017","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ren et al., 2017] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, and Hongyuan Zha. Unsupervised deep learning for optical flow estimation. In Artificial Intelligence (AAAI-17), Proceedings of the Thirty-First AAAI Conference on, pages 1495-1501, 2017.","title":"Unsupervised deep learning for optical flow estimation","authors":"Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang, Hongyuan Zha","year":"2017","venue":"Artificial Intelligence (AAAI-17), Proceedings of the Thirty-First AAAI Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Revaud et al., 2015] Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Epicflow: Edge- preserving interpolation of correspondences for optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1164-1172, 2015.","title":"Epicflow: Edgepreserving interpolation of correspondences for optical flow","authors":"Revaud","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rohlfing et al., 2003] Torsten Rohlfing, Calvin R Maurer, David A Bluemke, and Michael A Jacobs. Volume- preserving nonrigid registration of mr breast images us- ing free-form deformation with an incompressibility con- straint. IEEE transactions on medical imaging, 22(6):730- 741, 2003.","title":"Volumepreserving nonrigid registration of mr breast images using free-form deformation with an incompressibility constraint","authors":"Torsten Rohlfing, Calvin R Maurer, David A Bluemke, Michael A Jacobs","year":"2003","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sevilla-Lara et al., 2016] Laura Sevilla-Lara, Deqing Sun, Varun Jampani, and Michael J. Black. Optical flow with semantic segmentation and localized layers. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Optical flow with semantic segmentation and localized layers","authors":"Laura Sevilla-Lara, Deqing Sun, Varun Jampani, Michael J Black","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2010] Deqing Sun, Erik B Sudderth, and Michael J Black. Layered image motion with explicit oc- clusions, temporal consistency, and depth ordering. In Ad- vances in Neural Information Processing Systems, pages 2226-2234, 2010.","title":"Layered image motion with explicit occlusions, temporal consistency, and depth ordering","authors":"Deqing Sun, Erik B Sudderth, Michael J Black","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Weinzaepfel et al., 2013] Philippe Weinzaepfel, Jerome Re- vaud, Zaid Harchaoui, and Cordelia Schmid. Deepflow: Large displacement optical flow with deep matching. In Proceedings of the IEEE International Conference on Computer Vision, pages 1385-1392, 2013.","title":"Deepflow: Large displacement optical flow with deep matching","authors":"Weinzaepfel","year":"2013","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang and Li, 2015] Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1019-1027, 2015.","title":"Jiaolong Yang and Hongdong Li. Dense, accurate optical flow estimation with piecewise parametric model","authors":"Yang, Li","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yu et al., 2016] Jason J. Yu, Adam W. Harley, and Kon- stantinos G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and mo- tion smoothness. CoRR, abs/1608.05842, 2016.","title":"Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness","authors":"Jason J Yu, Adam W Harley, Konstantinos G Derpanis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2016] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In European conference on computer vision, pages 286-301. Springer, 2016.","title":"View synthesis by appearance flow","authors":"Zhou","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}