{"type":"ScholarlyArticle","title":"A 3D Convolutional Approach to Spectral Object Segmentation in Space and Time","authors":[{"name":"Elena Burceanu","slug":"elena-burceanu","given":"Elena","family":"Burceanu","affiliation":"University of Bucharest , Romania"},{"name":"Marius Leordeanu","slug":"marius-leordeanu","given":"Marius","family":"Leordeanu","affiliation":"Institute of Mathematics of the Romanian Academy; University Politehnica of Bucharest , Romania"}],"abstract":"We formulate object segmentation in video as a spectral graph clustering problem in space and time, in which nodes are pixels and their relations form local neighbourhoods. We claim that the strongest cluster in this pixel-level graph represents the salient object segmentation. We compute the main cluster using a novel and fast 3D filtering technique that finds the spectral clustering solution, namely the principal eigenvector of the graph's adjacency matrix, without building the matrix explicitly - which would be intractable. Our method is based on the power iteration which we prove is equivalent to performing a specific set of 3D convolutions in the space-time feature volume. This allows us to avoid creating the matrix and have a fast parallel implementation on GPU. We show that our method is much faster than classical power iteration applied directly on the adjacency matrix. Different from other works, ours is dedicated to preserving object consistency in space and time at the level of pixels. \r\nIn experiments, we obtain consistent improvement over the top state of the art methods on DAVIS-2016 dataset. We also achieve top results on the well-known SegTrackv2 dataset.","doi":"10.24963/ijcai.2020/69","url":"https://ijcai.org/proceedings/2020/69","pdf":"https://ijcai.org/proceedings/2020/0069.pdf","markdown":"https://ijcai.org/proceedings/2020/69.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/69","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":69,"pages":{"first":495,"last":501},"keywords":["Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning","Data Mining: Theoretical Foundations","Computer Vision: Motion and Tracking","Data Mining: Clustering, Unsupervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25061/file_25061.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Caelles et al., 2017] S. Caelles, K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. One-shot video object segmentation. CVPR, 2017.","title":"One-shot video object segmentation","authors":"S Caelles, K Maninis, J Pont-Tuset, L Leal-Taixé, D Cremers, L Van Gool","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2018] J. Cheng, Y.-H. Tsai, W.-C. Hung, S. Wang, and M.-H. Yang. Fast and accurate online video object segmentation via tracking parts. CVPR, 2018.","title":"Fast and accurate online video object segmentation via tracking parts","authors":"J Cheng, Y.-H Tsai, W.-C Hung, S Wang, M.-H Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung, 2005] Chung. Laplacians and the cheeger inequality for directed graphs. Annals of Combinatorics, 2005.","title":"Laplacians and the cheeger inequality for directed graphs","authors":"Chung","year":"2005","venue":"Annals of Combinatorics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Faktor and Irani, 2014] A. Faktor and M. Irani. Video seg- mentation by non-local consensus voting. BMVC, 2014.","title":"Video segmentation by non-local consensus voting","authors":"Irani Faktor, A Faktor, M Irani","year":"2014","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Felzenszwalb and Huttenlocher, 2004] P. Felzenszwalb and D. Huttenlocher. Efficient graph-based image segmenta- tion. IJCV, 2004.","title":"Efficient graph-based image segmentation","authors":"P Felzenszwalb, D Huttenlocher","year":"2004","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frobenius, 1907] G. Frobenius. \"About a Fundamental The- orem of Group Theory. II. Session Reports of the Royal Prussian Academy of Sciences, 1907.","title":"About a Fundamental Theorem of Group Theory","authors":"G Frobenius","year":"1907","venue":"Session Reports of the Royal Prussian Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hickson et al., 2014] S. Hickson, S. Birchfield, I. Essa, and H. Christensen. Efficient hierarchical graph-based seg- mentation of rgbd videos. CVPR, 2014.","title":"Efficient hierarchical graph-based segmentation of rgbd videos","authors":"S Hickson, S Birchfield, I Essa, H Christensen","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jain et al., 2017] S. Jain, B. Xiong, and K. Grauman. Fu- sionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos. arXiv, 2017.","title":"Fusionseg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos","authors":"S Jain, B Xiong, K Grauman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khoreva et al., 2017] A. Khoreva, F. Perazzi, R. Benenson, B. Schiele, and A. Hornung. Learning video object seg- mentation from static images. CVPR, 2017.","title":"Learning video object segmentation from static images","authors":"A Khoreva, F Perazzi, R Benenson, B Schiele, A Hornung","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koh and Kim, 2017] Y. Koh and C. Kim. Primary object segmentation in videos based on region augmentation and reduction. CVPR, 2017.","title":"Primary object segmentation in videos based on region augmentation and reduction","authors":"Y Koh, C Kim","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krähenbühl and Koltun, 2011] P. Krähenbühl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. arXiv, 2011.","title":"Efficient inference in fully connected crfs with gaussian edge potentials","authors":"P Krähenbühl, V Koltun","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar and Hebert, 2003] S. Kumar and M. Hebert. Dis- criminative random fields: A discriminative framework for contextual interaction in classification. 2003.","title":"Discriminative random fields: A discriminative framework for contextual interaction in classification","authors":"S Kumar, M Hebert","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lanczos, 1950] C. Lanczos. An iteration method for the so- lution of the eigenvalue i problem . of linear differential and integral operators. 1950.","title":"An iteration method for the solution of the eigenvalue i problem . of linear differential and integral operators","authors":"C Lanczos","year":"1950","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Leordeanu and Hebert, 2005] M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. ICCV, 2005.","title":"A spectral technique for correspondence problems using pairwise constraints","authors":"M Leordeanu, M Hebert","year":"2005","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leordeanu et al., 2009] M. Leordeanu, M. Hebert, and R. Sukthankar. An integer projected fixed point method for graph matching and MAP inference. NIPS, 2009.","title":"An integer projected fixed point method for graph matching and MAP inference","authors":"M Leordeanu, M Hebert, R Sukthankar","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2013] F. Li, T. Kim, A. Humayun, D. Tsai, and J. Rehg. Video segmentation by tracking many figure- ground segments. ICCV, 2013.","title":"Video segmentation by tracking many figureground segments","authors":"F Li, T Kim, A Humayun, D Tsai, J Rehg","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] S. Li, B. Seybold, A. Vorobyov, X. Lei, and C. Kuo. Unsupervised video object segmentation with motion-based bilateral networks. ECCV, 2018.","title":"Unsupervised video object segmentation with motion-based bilateral networks","authors":"S Li, B Seybold, A Vorobyov, X Lei, C Kuo","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2019] X. Lu, W. Wang, C. Ma, J. Shen, L. Shao, and F. Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. CVPR, 2019.","title":"See more, know more: Unsupervised video object segmentation with co-attention siamese networks","authors":"Lu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luiten et al., 2018] J. Luiten, P. Voigtlaender, and B. Leibe. Premvos: Proposal-generation, refinement and merging for video object segmentation. ACCV, 2018.","title":"Premvos: Proposal-generation, refinement and merging for video object segmentation","authors":"J Luiten, P Voigtlaender, B Leibe","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[M. Siam, 2019] S. Lu L. Petrich M. Gamal M Elhoseiny M Jagersand M. Siam, C. Jiang. Video object segmen- tation using teacher-student adaptation in a human robot interaction (hri) setting. ICRA, 2019.","title":"Video object segmentation using teacher-student adaptation in a human robot interaction (hri) setting","authors":"S, Lu L Petrich, M Gamal, M Elhoseiny, M Jagersand, M Siam, C Jiang","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maninis et al., 2018] K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. Video object segmentation without temporal information. PAMI, 2018.","title":"Video object segmentation without temporal information","authors":"K Maninis, S Caelles, Y Chen, J Pont-Tuset, L Leal-Taixé, D Cremers, L Van Gool","year":"2018","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Meila and Shi, 2001] M. Meila and J. Shi. A random walks view of spectral segmentation. AISTATS, 2001.","title":"A random walks view of spectral segmentation","authors":"M Meila, J Shi","year":"2001","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ng et al., 2001] A. Ng, M. Jordan, and Y. Weiss. On spec- tral clustering: Analysis and an algorithm. NIPS, 2001.","title":"On spectral clustering: Analysis and an algorithm","authors":"A Ng, M Jordan, Y Weiss","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Paszke et al., 2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Autodiff in pytorch. 2017. [Perazzi et al., 2016]","title":"Autodiff in pytorch","authors":"A Paszke, S Gross, S Chintala, G Chanan, E Yang, Z Devito, Z Lin, A Desmaison, L Antiga, A Lerer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Hornung. A benchmark dataset and evaluation methodology for video object segmentation. CVPR, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"F Perazzi, J Pont-Tuset, B Mcwilliams, L Van Gool, M Gross, A Hornung","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pourian et al., 2015] N. Pourian, S. Karthikeyan, and B. Manjunath. Weakly supervised graph based seman- tic segmentation by learning communities of image-parts. ICCV, 2015.","title":"Weakly supervised graph based semantic segmentation by learning communities of image-parts","authors":"N Pourian, S Karthikeyan, B Manjunath","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Reda et al., 2017] F. Reda, R. Pottorff, J. Barker, and B. Catanzaro. flownet2-pytorch: Pytorch implementation of flownet 2.0: Evolution of optical flow estimation with deep networks, 2017.","title":"flownet2-pytorch: Pytorch implementation of flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"F Reda, R Pottorff, J Barker, B Catanzaro","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shi and Malik, 2000] J. Shi and J. Malik. Normalized cuts and image segmentation. Technical report, 2000.","title":"Normalized cuts and image segmentation","authors":"J Shi, J Malik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Song et al., 2018] H. Song, W. Wang, S. Zhao, J. Shen, and K. Lam. Pyramid dilated deeper convlstm for video salient object detection. ECCV, 2018.","title":"Pyramid dilated deeper convlstm for video salient object detection","authors":"H Song, W Wang, S Zhao, J Shen, K Lam","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tokmakov et al., 2017] P. Tokmakov, K. Alahari, and C. Schmid. Learning video object segmentation with vi- sual memory. ICCV, 2017.","title":"Learning video object segmentation with visual memory","authors":"P Tokmakov, K Alahari, C Schmid","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Voigtlaender and Leibe, 2017] P. Voigtlaender and B. Leibe. Online adaptation of convolutional neural networks for video object segmentation. BMVC, 2017.","title":"Online adaptation of convolutional neural networks for video object segmentation","authors":"P Voigtlaender, B Leibe","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2017] L. Wang, H. Lu, Y. Wang, D. Feng, M. sand Wang, B. Yin, and X. Ruan. Learning to detect salient objects with image-level supervision. CVPR, 2017.","title":"Learning to detect salient objects with image-level supervision","authors":"L Wang, H Lu, Y Wang, D Feng, M Wang, B Yin, X Ruan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2018] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. Katsaggelos. Efficient video object segmentation via network modulation. CVPR, 2018.","title":"Efficient video object segmentation via network modulation","authors":"L Yang, Y Wang, X Xiong, J Yang, A Katsaggelos","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2015] C. Yu, H. Le, G. Zelinsky, and D. Samaras. Efficient video segmentation using parametric graph parti- tioning. ICCV, 2015.","title":"Efficient video segmentation using parametric graph partitioning","authors":"C Yu, H Le, G Zelinsky, D Samaras","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}