{"type":"ScholarlyArticle","title":"Image-level to Pixel-wise Labeling: From Theory to Practice","authors":[{"name":"Tiezhu Sun","slug":"tiezhu-sun","given":"Tiezhu","family":"Sun","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Wei Zhang","slug":"wei-zhang","given":"Wei","family":"Zhang","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Zhijie Wang","slug":"zhijie-wang","given":"Zhijie","family":"Wang","affiliation":"School of Control Science and Engineering , Shandong University"},{"name":"Lin Ma","slug":"lin-ma","given":"Lin","family":"Ma","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Zequn Jie","slug":"zequn-jie","given":"Zequn","family":"Jie","affiliation":"Tencent AI Lab , Shenzhen , China"}],"abstract":"Conventional convolutional neural networks (CNNs) have achieved great success in image semantic segmentation. Existing methods mainly focus on learning pixel-wise labels from an image directly. In this paper, we advocate tackling the pixel-wise segmentation problem by considering the image-level classification labels. Theoretically, we analyze and discuss the effects of image-level labels on pixel-wise segmentation from the perspective of information theory. In practice, an end-to-end segmentation model is built by fusing the image-level and pixel-wise labeling networks. A generative network is included to reconstruct the input image and further boost the segmentation model training with an auxiliary loss. Extensive experimental results on benchmark dataset demonstrate the effectiveness of the proposed method, where good image-level labels can significantly improve the pixel-wise segmentation accuracy.","doi":"10.24963/ijcai.2018/129","url":"https://ijcai.org/proceedings/2018/129","pdf":"https://ijcai.org/proceedings/2018/0129.pdf","markdown":"https://ijcai.org/proceedings/2018/129.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/129","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":129,"pages":{"first":928,"last":934},"keywords":["Computer Vision: Perception","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2016] Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale- aware semantic image segmentation. In CVPR, 2016.","title":"Attention to scale: Scaleaware semantic image segmentation","authors":"Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, Alan L Yuille","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fulkerson et al., 2009] Brian Fulkerson, Andrea Vedaldi, and Stefano Soatto. Class segmentation and object local- ization with superpixel neighborhoods. In ICCV, 2009.","title":"Class segmentation and object localization with superpixel neighborhoods","authors":"Brian Fulkerson, Andrea Vedaldi, Stefano Soatto","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hariharan et al., 2011] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.","title":"Semantic contours from inverse detectors","authors":"Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, Jitendra Malik","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jie et al., 2016] Zequn Jie, Xiaodan Liang, Jiashi Feng, Xi- aojie Jin, Wen Feng Lu, and Shuicheng Yan. Tree- structured reinforcement learning for sequential object lo- calization. In NIPS, 2016.","title":"Treestructured reinforcement learning for sequential object localization","authors":"Jie","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jin et al., 2017] Xiaojie Jin, Yunpeng Chen, Zequn Jie, Ji- ashi Feng, and Shuicheng Yan. Multi-path feedback recur- rent neural networks for scene parsing. In AAAI, 2017.","title":"Multi-path feedback recurrent neural networks for scene parsing","authors":"Xiaojie Jin, Yunpeng Chen, Zequn Jie, Jiashi Feng, Shuicheng Yan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick W. Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In AISTATS, 2015.","title":"Deeplysupervised nets","authors":"Chen-Yu Lee, Saining Xie, Patrick W Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Xin Li, Zequn Jie, Wei Wang, Changsong Liu, Jimei Yang, Xiaohui Shen, Zhe Lin, Qiang Chen, Shuicheng Yan, and Jiashi Feng. Foveanet: Perspective- aware urban scene parsing. In ICCV, 2017.","title":"Foveanet: Perspectiveaware urban scene parsing","authors":"Xin Li, Zequn Jie, Wei Wang, Changsong Liu, Jimei Yang, Xiaohui Shen, Zhe Lin, Qiang Chen, Shuicheng Yan, Jiashi Feng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2016] Guosheng Lin, Chunhua Shen, Anton van den Hengel, and Ian Reid. Efficient piecewise training of deep structured models for semantic segmentation. In CVPR, 2016.","title":"Efficient piecewise training of deep structured models for semantic segmentation","authors":"Guosheng Lin, Chunhua Shen, Anton Van Den, Ian Hengel, Reid","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2015] Wei Liu, Andrew Rabinovich, and Alexan- der C Berg. Parsenet: Looking wider to see better. arXiv:1506.04579, 2015.","title":"Parsenet: Looking wider to see better","authors":"Wei Liu, Andrew Rabinovich, Alexander C Berg","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2017] Ping Luo, Guangrun Wang, Liang Lin, and Xiaogang Wang. Deep dual learning for semantic image segmentation. In CVPR, 2017.","title":"Deep dual learning for semantic image segmentation","authors":"Luo","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2016] Lin Ma, Zhengdong Lu, and Hang Li. Learning to answer questions from image using convolu- tional neural network. In AAAI, 2016.","title":"Learning to answer questions from image using convolutional neural network","authors":"Lin Ma, Zhengdong Lu, Hang Li","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan et al., 2017] Tianxiang Pan, Bin Wang, Guiguang Ding, and Jun-Hai Yong. Fully convolutional neural networks with full-scale-features for semantic segmentation. In AAAI, 2017.","title":"Fully convolutional neural networks with full-scale-features for semantic segmentation","authors":"Tianxiang Pan, Bin Wang, Guiguang Ding, Jun-Hai Yong","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papandreou et al., 2015] George Papandreou, Liang-Chieh Chen, Kevin Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a dcnn for semantic image segmentation. arXiv:1502.02734, 2015.","title":"Weakly-and semi-supervised learning of a dcnn for semantic image segmentation","authors":"George Papandreou, Liang-Chieh Chen, Kevin Murphy, Alan L Yuille","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rahman and Wang, 2016] Md Atiqur Rahman and Yang Wang. Optimizing intersection-over-union in deep neural networks for image segmentation. In ISVC, 2016.","title":"Optimizing intersection-over-union in deep neural networks for image segmentation","authors":"Atiqur Md, Yang Rahman, Wang","year":"2016","venue":"ISVC","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shotton et al., 2009] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for image un- derstanding: Multi-class object recognition and segmenta- tion by jointly modeling texture, layout, and context. IJCV, 81(1):2-23, 2009.","title":"Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context","authors":"Jamie Shotton, John Winn, Carsten Rother, Antonio Criminisi","year":"2009","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2017] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cot- trell. Understanding convolution for semantic segmenta- tion. arXiv:1702.08502, 2017.","title":"Understanding convolution for semantic segmentation","authors":"Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2014] Yunchao Wei, Wei Xia, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. Cnn: Single-label to multi-label. arXiv:1406.5726, 2014.","title":"Cnn: Single-label to multi-label","authors":"Yunchao Wei, Wei Xia, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, Shuicheng Yan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yi et al., 2017] Zili Yi, Hao Zhang, Ping Tan Gong, et al. Dualgan: Unsupervised dual learning for image-to-image translation. arXiv:1704.02510, 2017.","title":"Dualgan: Unsupervised dual learning for image-to-image translation","authors":"Zili Yi, Hao Zhang, Ping Tan Gong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Wei Zhang, Xiaodong Yu, and Xuanyu He. Learning bidirectional temporal cues for video-based person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 2017.","title":"Learning bidirectional temporal cues for video-based person re-identification","authors":"Wei Zhang, Xiaodong Yu, Xuanyu He","year":"2017","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Wei Zhang, Qi Chen, Weidong Zhang, and Xuanyu He. Long-range terrain perception using con- volutional neural networks. Neurocomputing, 275:781- 787, 2018.","title":"Long-range terrain perception using convolutional neural networks","authors":"Wei Zhang, Qi Chen, Weidong Zhang, Xuanyu He","year":"2018","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2016] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. arXiv:1612.01105, 2016.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}