{"type":"ScholarlyArticle","title":"E3SN: Efficient End-to-End Siamese Network for Video Object Segmentation","authors":[{"name":"Meng Lan","slug":"meng-lan","given":"Meng","family":"Lan","affiliation":"National Engineering Research Center for Multimedia Software ,"},{"name":"Yipeng Zhang","slug":"yipeng-zhang","given":"Yipeng","family":"Zhang","affiliation":"National Engineering Research Center for Multimedia Software ,"},{"name":"Qinning Xu","slug":"qinning-xu","given":"Qinning","family":"Xu","affiliation":"National Engineering Research Center for Multimedia Software ,"},{"name":"Lefei Zhang","slug":"lefei-zhang","given":"Lefei","family":"Zhang","affiliation":"Institute of Artificial Intelligence and School of Computer Science , Wuhan University , Wuhan , China"}],"abstract":"In the semi-supervised video object segmentation (VOS) field, SiamMask has achieved competitive accuracy and the fastest running speed. However, the two-stage training procedure requires additional manual intervention, and using only single-level features does not maximize the rich hierarchical feature information. This paper proposes an efficient end-to-end Siamese network for VOS. In particular, a supervised sampling strategy is designed to optimize the training procedure. Such an optimization facilitates the training of the entire model in an end-to-end manner. Moreover, a multilevel feature aggregation module is developed to enhance feature representability and improve segmentation accuracy. Experimental results on DAVIS2016 and DAVIS2017 datasets show that the proposed approach outperforms the SiamMask in accuracy with similar FPS. Moreover, this approach also achieves good accuracy-speed trade-off compared with that of  other state-of-the-art VOS algorithms.","doi":"10.24963/ijcai.2020/98","url":"https://ijcai.org/proceedings/2020/98","pdf":"https://ijcai.org/proceedings/2020/0098.pdf","markdown":"https://ijcai.org/proceedings/2020/98.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/98","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":98,"pages":{"first":701,"last":707},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Motion and Tracking"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24884/file_24884.m3u8","length_sec":210,"provider":"hls"}],"references":[{"position":1,"raw":"[Bertinetto et al., 2016] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fully-convolutional siamese networks for object tracking. In ECCV Workshops, volume 9914, pages 850-865, 2016.","title":"Fully-convolutional siamese networks for object tracking","authors":"Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, Philip H S Torr","year":"2016","venue":"ECCV Workshops","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caelles et al., 2017] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In CVPR, pages 5320-5329, 2017.","title":"One-shot video object segmentation","authors":"Kevis-Kokitsi Sergi Caelles, Jordi Maninis, Laura Pont-Tuset, Daniel Leal-Taixé, Luc Cremers, Van Gool","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018a] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834-848, 2018.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Chen","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018b] Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In CVPR, pages 1189-1198, 2018.","title":"Blazingly fast video object segmentation with pixel-wise metric learning","authors":"Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2018] Jingchun Cheng, Yi-Hsuan Tsai, Wei- Chih Hung, Shengjin Wang, and Ming-Hsuan Yang. Fast and accurate online video object segmentation via tracking parts. In CVPR, pages 7415-7424, 2018.","title":"Fast and accurate online video object segmentation via tracking parts","authors":"Cheng","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2014] Junwei Han, Dingwen Zhang, Gong Cheng, Lei Guo, and Jinchang Ren. Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning. IEEE Transac- tions on Geoscience and Remote Sensing, 53(6):3325- 3337, 2014.","title":"Object detection in optical remote sensing images based on weakly supervised learning and high-level feature learning","authors":"Han","year":"2014","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Yuan-Ting Hu, Jia-Bin Huang, and Alexan- der G. Schwing. Videomatch: Matching based video ob- ject segmentation. In ECCV, pages 56-73, 2018.","title":"Videomatch: Matching based video object segmentation","authors":"Yuan-Ting Hu, Jia-Bin Huang, Alexander G Schwing","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jampani et al., 2017] Varun Jampani, Raghudeep Gadde, and Peter V. Gehler. Video propagation networks. In CVPR, pages 3154-3164, 2017.","title":"Video propagation networks","authors":"Varun Jampani, Raghudeep Gadde, Peter V Gehler","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In CVPR, pages 8971- 8980, 2018.","title":"High performance visual tracking with siamese region proposal network","authors":"Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, Xiaolin Hu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In CVPR, pages 4282-4291, 2019.","title":"Siamrpn++: Evolution of siamese visual tracking with very deep networks","authors":"Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, and Serge J. Belongie et al. Microsoft COCO: common objects in context. In ECCV, pages 740-755, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luiten et al., 2018] Jonathon Luiten, Paul Voigtlaender, and Bastian Leibe. Premvos: Proposal-generation, refinement and merging for video object segmentation. In ACCV, pages 565-580, 2018.","title":"Premvos: Proposal-generation, refinement and merging for video object segmentation","authors":"Jonathon Luiten, Paul Voigtlaender, Bastian Leibe","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maninis et al., 2019] K.-. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. Video object segmentation without temporal information. TPAMI, 41(6):1515-1530, 2019.","title":"Video object segmentation without temporal information","authors":"K - Maninis, S Caelles, Y Chen, J Pont-Tuset, L Leal-Taixé, D Cremers, L Van Gool","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oh et al., 2018] Seoung Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, and Seon Joo Kim. Fast video object segmen- tation by reference-guided mask propagation. In CVPR, pages 7376-7385, 2018.","title":"Fast video object segmentation by reference-guided mask propagation","authors":"Oh","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus H. Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"Federico Perazzi, Jordi Pont-Tuset, Brian Mcwilliams, Luc Van Gool, Markus H Gross, Alexander Sorkine-Hornung","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Perazzi et al., 2017] Federico Perazzi, Anna Khoreva, Ro- drigo Benenson, Bernt Schiele, and Alexander Sorkine- Hornung. Learning video object segmentation from static images. In CVPR, pages 3491-3500, 2017.","title":"Learning video object segmentation from static images","authors":"Perazzi","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pont-Tuset et al., 2017] Jordi Pont-Tuset, Federico Perazzi, and Sergi Caelles et al. The 2017 DAVIS challenge on video object segmentation. abs/1704.00675, 2017.","title":"The 2017 DAVIS challenge on video object segmentation","authors":"Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, and Hao Su et al. Imagenet large scale visual recognition chal- lenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2020] Liangchen Song, Cheng Wang, Lefei Zhang, Bo Du, Qian Zhang, Chang Huang, and Xinggang Wang. Unsupervised domain adaptive re-identification: Theory and practice. Pattern Recognition, page 107173, 2020.","title":"Unsupervised domain adaptive re-identification: Theory and practice","authors":"Liangchen Song, Cheng Wang, Lefei Zhang, Bo Du, Qian Zhang, Chang Huang, Xinggang Wang","year":"2020","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Voigtlaender et al., 2019] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang- Chieh Chen. FEELVOS: fast end-to-end embedding learn- ing for video object segmentation. In CVPR, pages 9481- 9490, 2019.","title":"FEELVOS: fast end-to-end embedding learning for video object segmentation","authors":"Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, Liang-Chieh Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Wenguan Wang, Jianbing Shen, and Fatih Porikli. Selective video object cutout. TIP, 26(12):5645-5655, 2017.","title":"Selective video object cutout","authors":"Wang","year":"2017","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip H. S. Torr. Fast online ob- ject tracking and segmentation: A unifying approach. In CVPR, pages 1328-1338, 2019.","title":"Fast online object tracking and segmentation: A unifying approach","authors":"Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, Philip H S Torr","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018] Ning Xu, Linjie Yang, and Yuchen Fan et al. Youtube-vos: Sequence-to-sequence video object segmentation. In ECCV, pages 603-619, 2018.","title":"Youtube-vos: Sequence-to-sequence video object segmentation","authors":"Ning Xu, Linjie Yang, Yuchen Fan","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2018] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K. Katsaggelos. Ef- ficient video object segmentation via network modulation. In CVPR, pages 6499-6507, 2018.","title":"Efficient video object segmentation via network modulation","authors":"Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016] Liangpei Zhang, Lefei Zhang, and Bo Du. Deep learning for remote sensing data: A tech- nical tutorial on the state of the art. IEEE Geoscience and Remote Sensing Magazine, 4(2):22-40, 2016.","title":"Deep learning for remote sensing data: A technical tutorial on the state of the art","authors":"Zhang","year":"2016","venue":"IEEE Geoscience and Remote Sensing Magazine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}