{"type":"ScholarlyArticle","title":"Importance-Aware Semantic Segmentation for Autonomous Driving System","authors":[{"name":"Bi-ke Chen","slug":"bi-ke-chen","given":"Bi-ke","family":"Chen","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology Nanjing , 210094 , China"},{"name":"Chen Gong","slug":"chen-gong","given":"Chen","family":"Gong","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology Nanjing , 210094 , China"},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology Nanjing , 210094 , China"}],"abstract":"Semantic Segmentation (SS) partitions an image into several coherent semantically meaningful parts, and classifies each part into one of the pre-determined classes. In this paper, we argue that existing SS methods cannot be reliably applied to autonomous driving system as they ignore the different importance levels of distinct classes for safe-driving. For example, pedestrians in the scene are much more important than sky when driving a car, so their segmentations should be as accurate as possible. To incorporate the importance information possessed by various object classes, this paper designs an \"Importance-Aware Loss\" (IAL) that specifically emphasizes the critical objects for autonomous driving. IAL operates under a hierarchical structure, and the classes with different importance are located in different levels so that they are assigned distinct weights. Furthermore, we derive the  forward and backward propagation rules for IAL and apply them to deep neural networks for realizing SS in intelligent driving system. The experiments on CamVid and Cityscapes datasets reveal that by employing the proposed loss function, the existing deep learning models including FCN, SegNet and ENet are able to consistently obtain the improved segmentation results on the pre-defined important classes for safe-driving.","doi":"10.24963/ijcai.2017/208","url":"https://ijcai.org/proceedings/2017/208","pdf":"https://ijcai.org/proceedings/2017/0208.pdf","markdown":"https://ijcai.org/proceedings/2017/208.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/208","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":208,"pages":{"first":1504,"last":1510},"keywords":["Robotics and Vision: Robotics and Vision","Robotics and Vision: Vision and Perception","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Brostow et al., 2008] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and Recognition Using Structure from Motion Point Clouds. In ECCV, pages 44-57, 2008.","title":"Segmentation and Recognition Using Structure from Motion Point Clouds","authors":"J Gabriel, Jamie Brostow, Julien Shotton, Roberto Fauqueur, Cipolla","year":"2008","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brostow et al., 2009] Gabriel J. Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, pages 88-97, 2009.","title":"Semantic object classes in video: A high-definition ground truth database","authors":"J Gabriel, Julien Brostow, Roberto Fauqueur, Cipolla","year":"2009","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Liang-Chieh Chen, George Papandreou, I- asonas Kokkinos, Kevin Murphy, and Alan L Yuille. Seman- tic image segmentation with deep convolutional nets and fully connected crfs. Computer Science, pages 357-361, 2016.","title":"Semantic image segmentation with deep convolutional nets and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2016","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Sebas- tian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The C- ityscapes Dataset for Semantic Urban Scene Understanding. In CVPR, pages 3213-3223, 2016.","title":"The Cityscapes Dataset for Semantic Urban Scene Understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[de Boer et al., 2005] Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A Tutorial on the Cross-Entropy Method. Annals OR, pages 19-67, 2005.","title":"A Tutorial on the Cross-Entropy Method","authors":"De Boer","year":"2005","venue":"Annals OR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eigen and Fergus, 2015] David Eigen and Rob Fergus. Pre- dicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture. In CVPR, pages 2650-2658, 2015.","title":"Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture","authors":"David Eigen, Rob Fergus","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Everingham et al., 2015] Mark Everingham, S. M. Ali Esla- mi, Luc J. Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The Pascal Visual Object Classes Challenge: A Retrospective. In IJCV, pages 98-136, 2015.","title":"The Pascal Visual Object Classes Challenge: A Retrospective","authors":"S M Mark Everingham, Luc J Ali Eslami, Van Gool, K I Christopher, John M Williams, Andrew Winn, Zisserman","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farabet et al., 2012] Clément Farabet, Camille Couprie, Lau- rent Najman, and Yann LeCun. Scene Parsing with Multi- scale Feature Learning, Purity Trees, and Optimal Covers. In ICML, pages 575-582, 2012.","title":"Scene Parsing with Multiscale Feature Learning, Purity Trees, and Optimal Covers","authors":"Clément Farabet, Camille Couprie, Laurent Najman, Yann Lecun","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gatta et al., 2014] Carlo Gatta, Adriana Romero, and Joost van de Weijer. Unrolling Loopy Top-Down Semantic Feed- back in Convolutional Deep Networks. In CVPR Workshops, pages 504-511, 2014.","title":"Unrolling Loopy Top-Down Semantic Feedback in Convolutional Deep Networks","authors":"Carlo Gatta, Adriana Romero, Joost Van De Weijer","year":"2014","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gong et al., 2015] Chen Gong, Dacheng Tao, Wei Liu, Stephen J. Maybank, Meng Fang, Keren Fu, and Jie Yang. Saliency propagation from simple to difficult. In CVPR, pages 2531-2539, 2015.","title":"Saliency propagation from simple to difficult","authors":"Chen Gong, Dacheng Tao, Wei Liu, Stephen J Maybank, Meng Fang, Keren Fu, Jie Yang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grangier et al., 2011] David Grangier, Léon Bottou, and Ro- nan Collobert. Deep Convolutional Networks for Scene Pars- ing. In ICML Workshops, 2011.","title":"Deep Convolutional Networks for Scene Parsing","authors":"David Grangier, Léon Bottou, Ronan Collobert","year":"2011","venue":"ICML Workshops","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hong et al., 2015] Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled Deep Neural Network for Semi- supervised Semantic Segmentation. In NIPS, pages 1495- 1503, 2015.","title":"Decoupled Deep Neural Network for Semisupervised Semantic Segmentation","authors":"Hong","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kontschieder et al., 2011] Peter Kontschieder, Samuel Rota Bulò, Horst Bischof, and Marcello Pelillo. Structured class- labels in random forests for semantic image labelling. In ICCV, pages 2190-2197, 2011.","title":"Structured classlabels in random forests for semantic image labelling","authors":"Kontschieder","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ladicky et al., 2010] Lubor Ladicky, Paul Sturgess, Karteek Alahari, Christopher Russell, and Philip H. S. Torr. What, Where and How Many? Combining Object Detectors and CRFs. In ECCV, pages 424-437, 2010.","title":"What, Where and How Many? Combining Object Detectors and CRFs","authors":"Lubor Ladicky, Paul Sturgess, Karteek Alahari, Christopher Russell, Philip H S Torr","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Long","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Noh et al., 2015] Hyeonwoo Noh, Seunghoon Hong, and Bo- hyung Han. Learning Deconvolution Network for Semantic Segmentation. In ICCV, pages 1520-1528, 2015.","title":"Learning Deconvolution Network for Semantic Segmentation","authors":"Noh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2016] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. ENet: A deep neu- ral network architecture for real-time semantic segmentation. CoRR, abs/1606.02147, 2016.","title":"ENet: A deep neural network architecture for real-time semantic segmentation","authors":"Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pohlen et al., 2016] Tobias Pohlen, Alexander Hermans, Markus Mathias, and Bastian Leibe. Full-resolution residual networks for semantic segmentation in street scenes. CoRR, abs/1611.08323, 2016.","title":"Full-resolution residual networks for semantic segmentation in street scenes","authors":"Pohlen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2012] Xiaofeng Ren, Liefeng Bo, and Dieter Fox. RGB-D scene labeling: Features and algorithms. In CVPR, pages 2759-2766, 2012.","title":"RGB-D scene labeling: Features and algorithms","authors":"Xiaofeng Ren, Liefeng Bo, Dieter Fox","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2016] Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian L. Price, Eli Shechtman, and Ian Sachs. Automatic portrait segmentation for image styliza- tion. Computer Graphics Forum, pages 93-102, 2016.","title":"Automatic portrait segmentation for image stylization","authors":"Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian L Price, Eli Shechtman, Ian Sachs","year":"2016","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shotton et al., 2008] Jamie Shotton, Matthew Johnson, and Roberto Cipolla. Semantic texton forests for image catego- rization and segmentation. In CVPR, pages 1-8, 2008.","title":"Semantic texton forests for image categorization and segmentation","authors":"Jamie Shotton, Matthew Johnson, Roberto Cipolla","year":"2008","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silberman et al., 2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor Segmentation and Support Inference from RGBD Images. In ECCV, pages 746- 760, 2012.","title":"Indoor Segmentation and Support Inference from RGBD Images","authors":"Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sturgess et al., 2009] Paul Sturgess, Karteek Alahari, Lubor Ladicky, and Philip H. S. Torr. Combining Appearance and Structure from Motion Features for Road Scene Understand- ing. In BMVC, pages 1-11, 2009.","title":"Combining Appearance and Structure from Motion Features for Road Scene Understanding","authors":"Paul Sturgess, Karteek Alahari, Lubor Ladicky, Philip H S Torr","year":"2009","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2013] Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep Neural Networks for object detec- tion. In NIPS, pages 2553-2561, 2013.","title":"Deep Neural Networks for object detection","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Treml et al., 2016] Michael Treml, José Arjona-Medina, Michael Unterthiner, et al. Speeding up Semantic Segmenta- tion for Autonomous Driving. In NIPS Workshop, 2016.","title":"Speeding up Semantic Segmentation for Autonomous Driving","authors":"Michael Treml, José Arjona-Medina, Michael Unterthiner","year":"2016","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vijay Badrinarayanan and Cipolla, 2017] Alex Kendall Vi- jay Badrinarayanan and Roberto Cipolla. SegNet: A deep convolutional encoder-decoder architecture for image seg- mentation. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, pages 1-1, 2017.","title":"SegNet: A deep convolutional encoder-decoder architecture for image segmentation","authors":"Alex Kendall, Vijay Badrinarayanan, Roberto Cipolla","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2016] Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, and Ming-Hsuan Yang. Object Contour Detec- tion with a Fully Convolutional Encoder-Decoder Network. In CVPR, pages 193-202, 2016.","title":"Object Contour Detection with a Fully Convolutional Encoder-Decoder Network","authors":"Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2010] Chenxi Zhang, Liang Wang, and Ruigang Yang. Semantic Segmentation of Urban Scenes Using Dense Depth Maps. In ECCV, pages 708-721, 2010.","title":"Semantic Segmentation of Urban Scenes Using Dense Depth Maps","authors":"Chenxi Zhang, Liang Wang, Ruigang Yang","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2015] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr. Conditional Random Fields as Recurrent Neural Networks. In ICCV, pages 1529-1537, 2015.","title":"Conditional Random Fields as Recurrent Neural Networks","authors":"Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, Philip H S Torr","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}