{"type":"ScholarlyArticle","title":"SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor","authors":[{"name":"Jiachen Xu","slug":"jiachen-xu","given":"Jiachen","family":"Xu","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Jingyu Gong","slug":"jingyu-gong","given":"Jingyu","family":"Gong","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Jie Zhou","slug":"jie-zhou","given":"Jie","family":"Zhou","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Xin Tan","slug":"xin-tan","given":"Xin","family":"Tan","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China; Department of Computer Science , City University of Hong Kong , HKSAR , China"},{"name":"Yuan Xie","slug":"yuan-xie","given":"Yuan","family":"Xie","affiliation":"School of Computer Science and Technology , East China Normal University , Shanghai , China"},{"name":"Lizhuang Ma","slug":"lizhuang-ma","given":"Lizhuang","family":"Ma","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China; School of Computer Science and Technology , East China Normal University , Shanghai , China"}],"abstract":"Besides local features, global information plays an essential role in semantic segmentation, while recent works usually fail to explicitly extract the meaningful global information and make full use of it. In this paper, we propose a SceneEncoder module to impose a scene-aware guidance to enhance the effect of global information. The module predicts a scene descriptor, which learns to represent the categories of objects existing in the scene and directly guides the point-level semantic segmentation through filtering out categories not belonging to this scene. Additionally, to alleviate segmentation noise in local region, we design a region similarity loss to propagate distinguishing features to their own neighboring points with the same label, leading to the enhancement of the distinguishing ability of point-wise features. We integrate our methods into several prevailing networks and conduct extensive experiments on benchmark datasets ScanNet and ShapeNet. Results show that our methods greatly improve the performance of baselines and achieve state-of-the-art performance.","doi":"10.24963/ijcai.2020/84","url":"https://ijcai.org/proceedings/2020/84","pdf":"https://ijcai.org/proceedings/2020/0084.pdf","markdown":"https://ijcai.org/proceedings/2020/84.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/84","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":84,"pages":{"first":601,"last":607},"keywords":["Computer Vision: Perception","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: 2D and 3D Computer Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24192/file_24192.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Atzmon et al., 2018] Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by ex- tension operators. ACM Transactions on Graphics(TOG), 37(4):71, 2018.","title":"Point convolutional neural networks by extension operators","authors":"Matan Atzmon, Haggai Maron, Yaron Lipman","year":"2018","venue":"ACM Transactions on Graphics(TOG)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.","title":"Shapenet: An information-rich 3d model repository","authors":"Thomas Angel X Chang, Leonidas Funkhouser, Pat Guibas, Qixing Hanrahan, Zimo Huang, Silvio Li, Manolis Savarese, Shuran Savva, Hao Song, Su","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai and Nießner, 2018] Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In ECCV, pages 452-468, 2018.","title":"3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation","authors":"Angela Dai, Matthias Nießner","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828-5839, 2017.","title":"Scannet: Richly-annotated 3d reconstructions of indoor scenes","authors":"Angela Dai, X Angel, Manolis Chang, Maciej Savva, Thomas Halber, Matthias Funkhouser, Nießner","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Brabandere et al., 2017] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmen- tation with a discriminative loss function. arXiv preprint arXiv:1708.02551, 2017.","title":"Semantic instance segmentation with a discriminative loss function","authors":"De Brabandere","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Engelmann et al., 2018] Francis Engelmann, Theodora Kontogianni, Jonas Schult, and Bastian Leibe. Know what your neighbors do: 3d semantic segmentation of point clouds. In ECCV, pages 0-0, 2018.","title":"Know what your neighbors do: 3d semantic segmentation of point clouds","authors":"Francis Engelmann, Theodora Kontogianni, Jonas Schult, Bastian Leibe","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Graham et al., 2018] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmenta- tion with submanifold sparse convolutional networks. In CVPR, pages 9224-9232, 2018.","title":"3d semantic segmentation with submanifold sparse convolutional networks","authors":"Benjamin Graham, Martin Engelcke, Laurens Van Der Maaten","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gupta et al., 2013] Saurabh Gupta, Pablo Arbelaez, and Ji- tendra Malik. Perceptual organization and recognition of indoor scenes from rgb-d images. In CVPR, pages 564- 571, 2013.","title":"Perceptual organization and recognition of indoor scenes from rgb-d images","authors":"Saurabh Gupta, Pablo Arbelaez, Jitendra Malik","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2019] Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, and Leonidas J Guibas. Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes. In CVPR, pages 4440-4449, 2019.","title":"Texturenet: Consistent local parametrizations for learning from high-resolution signals on meshes","authors":"Huang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2019] Li Jiang, Hengshuang Zhao, Shu Liu, Xi- aoyong Shen, Chi-Wing Fu, and Jiaya Jia. Hierarchical point-edge interaction network for point cloud semantic segmentation. In ICCV, pages 10433-10441, 2019.","title":"Hierarchical point-edge interaction network for point cloud semantic segmentation","authors":"Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lei et al., 2019] Huan Lei, Naveed Akhtar, and Ajmal Mian. Octree guided cnn with spherical kernels for 3d point clouds. In CVPR, pages 9631-9640, 2019.","title":"Octree guided cnn with spherical kernels for 3d point clouds","authors":"Huan Lei, Naveed Akhtar, Ajmal Mian","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018a] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In CVPR, pages 9397-9406, 2018.","title":"So-net: Self-organizing network for point cloud analysis","authors":"Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018b] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convo- lution on x-transformed points. In NIPS, pages 820-830, 2018.","title":"Pointcnn: Convolution on x-transformed points","authors":"Li","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qi et al., 2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, pages 5099-5108, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Su et al., 2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view con- volutional neural networks for 3d shape recognition. In ICCV, pages 945-953, 2015.","title":"Multi-view convolutional neural networks for 3d shape recognition","authors":"Su","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tchapmi et al., 2017] Lyne Tchapmi, Christopher Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. In 3DV, pages 537-547. IEEE, 2017.","title":"Segcloud: Semantic segmentation of 3d point clouds","authors":"Lyne Tchapmi, Christopher Choy, Iro Armeni, Junyoung Gwak, Silvio Savarese","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2019] Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convo- lution for point cloud semantic segmentation. In CVPR, pages 10296-10305, 2019.","title":"Graph attention convolution for point cloud semantic segmentation","authors":"Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, Jie Shan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2019] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, pages 9621-9630, 2019.","title":"Pointconv: Deep convolutional networks on 3d point clouds","authors":"Wenxuan Wu, Zhongang Qi, Li Fuxin","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. In ECCV, pages 87-102, 2018.","title":"Spidercnn: Deep learning on point sets with parameterized convolutional filters","authors":"Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, Yu Qiao","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2019] Zhiyuan Zhang, Binh-Son Hua, and Sai- Kit Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics. In ICCV, pages 1607-1616, 2019.","title":"Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics","authors":"Zhiyuan Zhang, Binh-Son Hua, Sai-Kit Yeung","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhao et al., 2019] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighbor- hood features for point cloud processing. In CVPR, pages 5565-5573, 2019.","title":"Pointweb: Enhancing local neighborhood features for point cloud processing","authors":"Zhao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}