{"type":"ScholarlyArticle","title":"Cross-Granularity Graph Inference for Semantic Video Object Segmentation","authors":[{"name":"Huiling Wang","slug":"huiling-wang","given":"Huiling","family":"Wang","affiliation":"Department of Signal Processing , Tampere University of Technology , Finland"},{"name":"Tinghuai Wang","slug":"tinghuai-wang","given":"Tinghuai","family":"Wang","affiliation":"Nokia Technologies , Finland"},{"name":"Ke Chen","slug":"ke-chen","given":"Ke","family":"Chen","affiliation":"Department of Signal Processing , Tampere University of Technology , Finland"},{"name":"Joni-Kristian Kämäräinen","slug":"joni-kristian-kamarainen","given":"Joni-Kristian","family":"Kämäräinen","affiliation":"Department of Signal Processing , Tampere University of Technology , Finland"}],"abstract":"We address semantic video object segmentation via a novel cross-granularity hierarchical graphical model to integrate tracklet and object proposal reasoning with superpixel labeling. Tracklet characterizes varying spatial-temporal relations of video object which, however, quite often suffers from sporadic local outliers. In order to acquire high-quality tracklets, we propose a transductive inference model which is capable of calibrating short-range noisy object tracklets with respect to long-range dependencies and high-level context cues. In the center of this work lies a new paradigm of semantic video object segmentation beyond modeling appearance and motion of objects locally, where the semantic label is inferred by jointly exploiting multi-scale contextual information and spatial-temporal relations of video object. We evaluate our method on two popular semantic video object segmentation benchmarks and demonstrate that it advances the state-of-the-art by achieving superior accuracy performance than other leading methods.","doi":"10.24963/ijcai.2017/634","url":"https://ijcai.org/proceedings/2017/634","pdf":"https://ijcai.org/proceedings/2017/0634.pdf","markdown":"https://ijcai.org/proceedings/2017/634.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/634","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Robotics and Vision","track":"Main track","number":634,"pages":{"first":4544,"last":4550},"keywords":["Robotics and Vision: Robotics and Vision","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Ayvaci and Soatto, 2012] Alper Ayvaci and Stefano Soatto. Detachable object detection: Segmentation and depth or- dering from short-baseline video. IEEE Trans. Pattern Anal. Mach. Intell., 34(10):1942-1951, 2012.","title":"Detachable object detection: Segmentation and depth ordering from short-baseline video","authors":"Alper Ayvaci, Stefano Soatto","year":"2012","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banica et al., 2013] Dan Banica, Alexandru Agape, Adrian Ion, and Cristian Sminchisescu. Video object segmenta- tion by salient segment chain composition. In ICCV Work- shops, pages 283-290, 2013.","title":"Video object segmentation by salient segment chain composition","authors":"Dan Banica, Alexandru Agape, Adrian Ion, Cristian Sminchisescu","year":"2013","venue":"ICCV Workshops","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boykov et al., 2001] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell., 23(11):1222-1239, 2001.","title":"Fast approximate energy minimization via graph cuts","authors":"Yuri Boykov, Olga Veksler, Ramin Zabih","year":"2001","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi and Savarese, 2012] Wongun Choi and Silvio Savarese. A unified framework for multi-target tracking and collective activity recognition. In ECCV, pages 215-230. Springer, 2012.","title":"A unified framework for multi-target tracking and collective activity recognition","authors":"Savarese Choi, Wongun Choi, Silvio Savarese","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Danelljan et al., 2015] Martin Danelljan, Gustav Hager, Fa- had Shahbaz Khan, and Michael Felsberg. Learning spa- tially regularized correlation filters for visual tracking. In ICCV, pages 4310-4318, 2015.","title":"Learning spatially regularized correlation filters for visual tracking","authors":"Martin Danelljan, Gustav Hager, Fahad Shahbaz Khan, Michael Felsberg","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Drayer and Brox, 2016] Benjamin Drayer and Thomas Brox. Object detection, tracking, and motion segmenta- tion for object-level video segmentation. arXiv preprint arXiv:1608.03066, 2016.","title":"Object detection, tracking, and motion segmentation for object-level video segmentation","authors":"Brox Drayer, Benjamin Drayer, Thomas Brox","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Endres and Hoiem, 2010] Ian Endres and Derek Hoiem. Category independent object proposals. In ECCV, pages 575-588, 2010.","title":"Category independent object proposals","authors":"Hoiem Endres, Ian Endres, Derek Hoiem","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Giordano et al., 2015] Daniela Giordano, Francesca Murabito, Simone Palazzo, and Concetto Spampinato. Superpixel-based video object segmentation using per- ceptual organization and location prior. In CVPR, pages 4814-4822, 2015.","title":"Superpixel-based video object segmentation using perceptual organization and location prior","authors":"Daniela Giordano, Francesca Murabito, Simone Palazzo, Concetto Spampinato","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In ICCV, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hartmann et al., 2012] Glenn Hartmann, Matthias Grund- mann, Judy Hoffman, David Tsai, Vivek Kwatra, Omid Madani, Sudheendra Vijayanarasimhan, Irfan A. Essa, James M. Rehg, and Rahul Sukthankar. Weakly supervised learning of object segmentations from web-scale video. In ECCV Workshop, pages 198-208, 2012.","title":"Weakly supervised learning of object segmentations from web-scale video","authors":"Glenn Hartmann, Matthias Grundmann, Judy Hoffman, David Tsai, Vivek Kwatra, Omid Madani, Sudheendra Vijayanarasimhan, Irfan A Essa, James M Rehg, Rahul Sukthankar","year":"2012","venue":"ECCV Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Keuper et al., 2015] Margret Keuper, Bjoern Andres, and Thomas Brox. Motion trajectory segmentation via mini- mum cost multicuts. In ICCV, pages 3271-3279, 2015.","title":"Motion trajectory segmentation via minimum cost multicuts","authors":"Margret Keuper, Bjoern Andres, Thomas Brox","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2011] Yong Jae Lee, Jaechul Kim, and Kristen Grauman. Key-segments for video object segmentation. In ICCV, pages 1995-2002, 2011.","title":"Key-segments for video object segmentation","authors":"Jae Yong, Jaechul Lee, Kristen Kim, Grauman","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2014] Xiao Liu, Dacheng Tao, Mingli Song, Ying Ruan, Chun Chen, and Jiajun Bu. Weakly supervised mul- ticlass video segmentation. In CVPR, pages 57-64, 2014.","title":"Weakly supervised multiclass video segmentation","authors":"Xiao Liu, Dacheng Tao, Mingli Song, Ying Ruan, Chun Chen, Jiajun Bu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2015] Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang. Hierarchical convolutional fea- tures for visual tracking. In ICCV, pages 3074-3082, 2015.","title":"Hierarchical convolutional features for visual tracking","authors":"Chao Ma, Jia-Bin Huang, Xiaokang Yang, Ming-Hsuan Yang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ochs et al., 2014] Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE Trans. Pattern Anal. Mach. Intell., 36(6):1187-1200, 2014.","title":"Segmentation of moving objects by long term video analysis","authors":"Peter Ochs, Jitendra Malik, Thomas Brox","year":"2014","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papazoglou and Ferrari, 2013] Anestis Papazoglou and Vit- torio Ferrari. Fast object segmentation in unconstrained video. In ICCV, pages 1777-1784, 2013.","title":"Fast object segmentation in unconstrained video","authors":"Anestis Papazoglou, Vittorio Ferrari","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Prest et al., 2012] Alessandro Prest, Christian Leistner, Javier Civera, Cordelia Schmid, and Vittorio Ferrari. Learning object class detectors from weakly annotated video. In CVPR, pages 3282-3289, 2012.","title":"Learning object class detectors from weakly annotated video","authors":"Alessandro Prest, Christian Leistner, Javier Civera, Cordelia Schmid, Vittorio Ferrari","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shankar Nagaraja et al., 2015] Naveen Shankar Nagaraja, Frank R Schmidt, and Thomas Brox. Video segmentation with just a few strokes. In ICCV, pages 3235-3243, 2015.","title":"Video segmentation with just a few strokes","authors":"Naveen Shankar Nagaraja, Frank R Schmidt, Thomas Brox","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sundberg et al., 2011] Patrik Sundberg, Thomas Brox, Michael Maire, Pablo Arbelaez, and Jitendra Malik. Oc- clusion boundary detection and figure/ground assignment from optical flow. In CVPR, pages 2233-2240, 2011.","title":"Occlusion boundary detection and figure/ground assignment from optical flow","authors":"Sundberg","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2013] Kevin D. Tang, Rahul Sukthankar, Jay Yagnik, and Fei-Fei Li. Discriminative segment annota- tion in weakly labeled video. In CVPR, pages 2483-2490, 2013.","title":"Discriminative segment annotation in weakly labeled video","authors":"Kevin D Tang, Rahul Sukthankar, Jay Yagnik, Fei-Fei Li","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Taylor et al., 2013] Brian Taylor, Alper Ayvaci, Avinash Ravichandran, and Stefano Soatto. Semantic video seg- mentation from occlusion relations within a convex op- timization framework. In EMMCVPR, pages 195-208. Springer, 2013.","title":"Semantic video segmentation from occlusion relations within a convex optimization framework","authors":"Brian Taylor, Alper Ayvaci, Avinash Ravichandran, Stefano Soatto","year":"2013","venue":"EMMCVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang and Wang, 2016] Huiling Wang and Tinghuai Wang. Primary object discovery and segmentation in videos via graph-based transductive inference. Comput. Vis. Image Underst., 143(2):159-172, 2016.","title":"Primary object discovery and segmentation in videos via graph-based transductive inference","authors":"Wang, Wang ; ] Huiling Wang, Tinghuai Wang","year":"2016","venue":"Comput. Vis. Image Underst","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2009] Chaohui Wang, Martin de La Gorce, and Nikos Paragios. Segmentation, ordering and multi-object tracking using graphical models. In ICCV, pages 747-754, 2009.","title":"Segmentation, ordering and multi-object tracking using graphical models","authors":"Wang","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2015] Wenguan Wang, Jianbing Shen, and Fatih Porikli. Saliency-aware geodesic video object seg- mentation. In CVPR, pages 3395-3402, 2015.","title":"Saliency-aware geodesic video object segmentation","authors":"Wang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2016] Huiling Wang, Tapani Raiko, Lasse Lensu, Tinghuai Wang, and Juha Karhunen. Semi- supervised domain adaptation for weakly labeled semantic video object segmentation. In ACCV, 2016.","title":"Semisupervised domain adaptation for weakly labeled semantic video object segmentation","authors":"Huiling Wang, Tapani Raiko, Lasse Lensu, Tinghuai Wang, Juha Karhunen","year":"2016","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiao and Lee, 2016] Fanyi Xiao and Yong Jae Lee. Track and segment: An iterative unsupervised approach for video object proposals. In CVPR, pages 551-566, 2016.","title":"Track and segment: An iterative unsupervised approach for video object proposals","authors":"Xiao, Lee ;, Fanyi Xiao, Yong Jae, Lee","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2015] Yu Zhang, Xiaowu Chen, Jia Li, Chen Wang, and Changqun Xia. Semantic object segmentation via detection in weakly labeled video. In CVPR, pages 3641-3649, 2015.","title":"Semantic object segmentation via detection in weakly labeled video","authors":"Yu Zhang, Xiaowu Chen, Jia Li, Chen Wang, Changqun Xia","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao and Fu, 2015] Handong Zhao and Yun Fu. Semantic single video segmentation with robust graph representa- tion. In IJCAI, pages 2219-2226, 2015.","title":"Semantic single video segmentation with robust graph representation","authors":"Handong Zhao, Yun Fu","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/314"},{"position":31,"raw":"[Zhou et al., 2004] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Sch. Learning with local and global consistency. In NIPS, pages 321-328, 2004.","title":"Learning with local and global consistency","authors":"Zhou","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}