{"type":"ScholarlyArticle","title":"SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation","authors":[{"name":"Xuewei Li","slug":"xuewei-li","given":"Xuewei","family":"Li","affiliation":"College of Computer Science and Technology , Zhejiang University; ARC Lab , Tencent PCG"},{"name":"Tao Wu","slug":"tao-wu","given":"Tao","family":"Wu","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Zhongang Qi","slug":"zhongang-qi","given":"Zhongang","family":"Qi","affiliation":"ARC Lab , Tencent PCG"},{"name":"Gaoang Wang","slug":"gaoang-wang","given":"Gaoang","family":"Wang","affiliation":"Zhejiang University-University of Illinois at Urbana-Champaign Institute , Zhejiang University"},{"name":"Ying Shan","slug":"ying-shan","given":"Ying","family":"Shan","affiliation":"ARC Lab , Tencent PCG"},{"name":"Xi Li","slug":"xi-li","given":"Xi","family":"Li","affiliation":"College of Computer Science and Technology , Zhejiang University; Zhejiang -Singapore Innovation and AI Joint Research Lab , Hangzhou"}],"abstract":"As an important and challenging problem in computer vision, PAnoramic Semantic Segmentation (PASS) gives complete scene perception based on an ultra-wide angle of view. Usually, prevalent PASS methods with 2D panoramic image input focus on solving image distortions but lack consideration of the 3D properties of original 360 degree data. Therefore, their performance will drop a lot when inputting panoramic images with the 3D disturbance. To be more robust to 3D disturbance, we propose our Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation (SGAT4PASS), considering 3D spherical geometry knowledge. Specifically, a spherical geometry-aware framework is proposed for PASS. It includes three modules, i.e., spherical geometry-aware image projection, spherical deformable patch embedding, and a panorama-aware loss, which takes input images with 3D disturbance into account, adds a spherical geometry-aware constraint on the existing deformable patch embedding, and indicates the pixel density of original 360 degree data, respectively. Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS significantly improves performance and robustness, with approximately a 2% increase in mIoU, and when small 3D disturbances occur in the data, the stability of our performance is improved by an order of magnitude. Our code and supplementary material are available at https://github.com/TencentARC/SGAT4PASS.","doi":"10.24963/ijcai.2023/125","url":"https://ijcai.org/proceedings/2023/125","pdf":"https://ijcai.org/proceedings/2023/0125.pdf","markdown":"https://ijcai.org/proceedings/2023/125.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/125","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":125,"pages":{"first":1125,"last":1133},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Scene analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Ai et al., 2022] Hao Ai, Zidong Cao, Jinjing Zhu, Haotian Bai, Yucheng Chen, and Ling Wang. Deep learning for omnidirectional vision: A survey and new perspectives. arXiv preprint arXiv:2205.10468, 2022.","title":"Deep learning for omnidirectional vision: A survey and new perspectives","authors":"Ai Hao, Zidong Cao, Jinjing Zhu, Haotian Bai, Yucheng Chen, Ling Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Armeni et al., 2017] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017.","title":"Joint 2d-3d-semantic data for indoor scene understanding","authors":"Iro Armeni, Sasha Sax, Silvio Amir R Zamir, Savarese","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berenguel-Baeta et al., 2022] Bruno Berenguel-Baeta, Je- sus Bermudez-Cameo, and Jose J Guerrero. Fred- snet: Joint monocular depth and semantic segmenta- tion with fast fourier convolutions. arXiv preprint arXiv:2210.01595, 2022.","title":"Bruno Berenguel-Baeta","authors":"Berenguel-Baeta","year":"2022","venue":"Fredsnet: Joint monocular depth and semantic segmentation with fast fourier convolutions","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Zhiyang Chen, Yousong Zhu, Chaoyang Zhao, Guosheng Hu, Wei Zeng, Jinqiao Wang, and Ming Tang. Dpt: Deformable patch-based transformer for visual recognition. In Proc. ACM MM, 2021.","title":"Dpt: Deformable patch-based transformer for visual recognition","authors":"Zhiyang Chen, Yousong Zhu, Chaoyang Zhao, Guosheng Hu, Wei Zeng, Jinqiao Wang, Ming Tang","year":"2021","venue":"Proc. ACM MM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proc. ICCV, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[de La Garanderie et al., 2018] Greire Payen de La Garan- derie, Amir Atapour Abarghouei, and Toby P Breckon. Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic im- agery. In Proc. ECCV, pages 789-807, 2018.","title":"Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery","authors":"La De, Garanderie","year":"2018","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2017] Liuyuan Deng, Ming Yang, Yeqiang Qian, Chunxiang Wang, and Bing Wang. Cnn based se- mantic segmentation for urban traffic scenes using fisheye camera. In Proc. IEEE Intell. Vehicles Symp., pages 231- 236. IEEE, 2017.","title":"Cnn based semantic segmentation for urban traffic scenes using fisheye camera","authors":"Liuyuan Deng, Ming Yang, Yeqiang Qian, Chunxiang Wang, Bing Wang","year":"2017","venue":"Proc. IEEE Intell. Vehicles Symp","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Eder et al., 2020] Marc Eder, Mykhailo Shvets, John Lim, and Jan-Michael Frahm. Tangent images for mitigating spherical distortion. In Proc. CVPR, 2020.","title":"Tangent images for mitigating spherical distortion","authors":"Marc Eder, Mykhailo Shvets, John Lim, Jan-Michael Frahm","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Esteves et al., 2020] Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin-weighted spherical cnns. Proc. NeurIPS, 33:8614-8625, 2020.","title":"Spin-weighted spherical cnns","authors":"Carlos Esteves, Ameesh Makadia, Kostas Daniilidis","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2022] Shaohua Gao, Kailun Yang, Hao Shi, Kai- wei Wang, and Jian Bai. Review on panoramic imaging and its applications in scene understanding. arXiv preprint arXiv:2205.05570, 2022.","title":"Review on panoramic imaging and its applications in scene understanding","authors":"Shaohua Gao, Kailun Yang, Hao Shi, Kaiwei Wang, Jian Bai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2022] Xing Hu, Yi An, Cheng Shao, and Hu- osheng Hu. Distortion convolution module for semantic segmentation of panoramic images based on the image- forming principle. IEEE Trans. Instrum. Meas., 71:1-12, 2022.","title":"Distortion convolution module for semantic segmentation of panoramic images based on the imageforming principle","authors":"Xing Hu, Yi An, Cheng Shao, Huosheng Hu","year":"2022","venue":"IEEE Trans. Instrum. Meas","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2019] Chiyu Jiang, Jingwei Huang, Karthik Kashinath, Philip Marcus, Matthias Niessner, et al. Spherical cnns on unstructured grids. arXiv preprint arXiv:1901.02039, 2019.","title":"Spherical cnns on unstructured grids","authors":"Chiyu Jiang, Jingwei Huang, Karthik Kashinath, Philip Marcus, Matthias Niessner","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2018] Yeonkun Lee, Jaeseok Jeong, Jongseob Yun, Wonjune Cho, and Kuk-Jin Yoon. Spherephd: Ap- plying cnns on a spherical polyhedron representation of 360 degree images. arXiv preprint arXiv:1811.08196, 2018.","title":"Spherephd: Applying cnns on a spherical polyhedron representation of 360 degree images","authors":"Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2021] Chaoxiang Ma, Jiaming Zhang, Kailun Yang, Alina Roitberg, and Rainer Stiefelhagen. Densep- ass: Dense panoramic semantic segmentation via unsuper- vised domain adaptation with attention-augmented context exchange. In Proc. ITSC, pages 2766-2772. IEEE, 2021.","title":"Densepass: Dense panoramic semantic segmentation via unsupervised domain adaptation with attention-augmented context exchange","authors":"Chaoxiang Ma, Jiaming Zhang, Kailun Yang, Alina Roitberg, Rainer Stiefelhagen","year":"2021","venue":"Proc. ITSC","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pintore et al., 2021] Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schneider, and Enrico Gobbetti. Slicenet: deep dense depth estimation from a single indoor panorama using a slice-based representation. In Proc. CVPR, pages 11536-11545, 2021.","title":"Slicenet: deep dense depth estimation from a single indoor panorama using a slice-based representation","authors":"Giovanni Pintore, Marco Agus, Eva Almansa, Jens Schneider, Enrico Gobbetti","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Poulin-Girard and Thibault, 2012] Anne-Sophie Poulin- Girard and Simon Thibault. Optical testing of panoramic lenses. Opt. Eng., 51(5):053603, 2012.","title":"Optical testing of panoramic lenses","authors":"Poulin-Girard, ; -Girard Thibault, Simon Thibault","year":"2012","venue":"Opt. Eng","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rao et al., 2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsifi- cation. Proc. NeurIPS, 34:13937-13949, 2021.","title":"Dynamicvit: Efficient vision transformers with dynamic token sparsification","authors":"Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, Cho-Jui Hsieh","year":"2021","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2022] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360 • depth estimation. In Proc. ECCV, pages 195-211. Springer, 2022.","title":"Panoformer: Panorama transformer for indoor 360 • depth estimation","authors":"Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, Yao Zhao","year":"2022","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2021] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. In Proc. CVPR, pages 2573- 2582, 2021.","title":"Hohonet: 360 indoor holistic understanding with latent horizontal features","authors":"Cheng Sun, Min Sun, Hwann-Tzong Chen","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tateno et al., 2018] Keisuke Tateno, Nassir Navab, and Federico Tombari. Distortion-aware convolutional filters for dense prediction in panoramic images. In Proc. ECCV, pages 707-722, 2018.","title":"Distortion-aware convolutional filters for dense prediction in panoramic images","authors":"Keisuke Tateno, Nassir Navab, Federico Tombari","year":"2018","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021a] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In Proc. ICCV, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wang","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021b] Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Proc. NeurIPS, 34:11960-11973, 2021.","title":"Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition","authors":"Wang","year":"2021","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia et al., 2022] Zhuofan Xia, Xuran Pan, Shiji Song, Li Er- ran Li, and Gao Huang. Vision transformer with de- formable attention. In Proc. CVPR, pages 4794-4803, 2022.","title":"Vision transformer with deformable attention","authors":"Xia","year":"2022","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018] Mai Xu, Yuhang Song, Jianyi Wang, MingLang Qiao, Liangyu Huo, and Zulin Wang. Predict- ing head movement in panoramic video: A deep reinforce- ment learning approach. IEEE Trans. Pattern Anal. Mach. Intell., 41(11):2693-2708, 2018.","title":"Predicting head movement in panoramic video: A deep reinforcement learning approach","authors":"Mai Xu, Yuhang Song, Jianyi Wang, Minglang Qiao, Liangyu Huo, Zulin Wang","year":"2018","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2021] Yanyu Xu, Ziheng Zhang, and Shenghua Gao. Spherical dnns and their applications in 360 images and videos. IEEE Trans. Pattern Anal. Mach. Intell., 2021.","title":"Spherical dnns and their applications in 360 images and videos","authors":"Yanyu Xu, Ziheng Zhang, Shenghua Gao","year":"2021","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2022] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proc. AAAI, volume 36, pages 2964-2972, 2022.","title":"Evo-vit: Slow-fast token evolution for dynamic vision transformer","authors":"Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun","year":"2022","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2019] Kailun Yang, Xinxin Hu, Luis M Bergasa, Eduardo Romera, and Kaiwei Wang. Pass: Panoramic annular semantic segmentation. IEEE Trans. Intell. Trans. Syst., 21(10):4171-4185, 2019.","title":"Pass: Panoramic annular semantic segmentation","authors":"Yang","year":"2019","venue":"IEEE Trans. Intell. Trans. Syst","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2020] Kailun Yang, Xinxin Hu, Hao Chen, Kaite Xiang, Kaiwei Wang, and Rainer Stiefelhagen. Ds- pass: Detail-sensitive panoramic annular semantic seg- mentation through swaftnet for surrounding sensing. In Proc. IEEE Intell. Vehicles Symp., pages 457-464. IEEE, 2020.","title":"Dspass: Detail-sensitive panoramic annular semantic segmentation through swaftnet for surrounding sensing","authors":"Kailun Yang, Xinxin Hu, Hao Chen, Kaite Xiang, Kaiwei Wang, Rainer Stiefelhagen","year":"2020","venue":"Proc. IEEE Intell. Vehicles Symp","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2021a] Kailun Yang, Xinxin Hu, and Rainer Stiefelhagen. Is context-aware cnn ready for the surround- ings? panoramic semantic segmentation in the wild. IEEE Trans. Image Process., 30:1866-1881, 2021.","title":"Is context-aware cnn ready for the surroundings? panoramic semantic segmentation in the wild","authors":"Yang","year":"2021","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2021b] Kailun Yang, Jiaming Zhang, Simon Reiß, Xinxin Hu, and Rainer Stiefelhagen. Capturing omni-range context for omnidirectional segmentation. In Proc. CVPR, pages 1376-1386, 2021.","title":"Capturing omni-range context for omnidirectional segmentation","authors":"Yang","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye et al., 2021] Zhiyuan Ye, Hai-Bo Wang, Jun Xiong, and Kaige Wang. Ghost panorama using a convex mirror. Opt. Lett., 46(21):5389-5392, 2021.","title":"Ghost panorama using a convex mirror","authors":"Zhiyuan Ye, Hai-Bo Wang, Jun Xiong, Kaige Wang","year":"2021","venue":"Opt. Lett","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yin et al., 2022] Hongxu Yin, Arash Vahdat, Jose M Al- varez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A- vit: Adaptive tokens for efficient vision transformer. In Proc. CVPR, pages 10809-10818, 2022.","title":"Avit: Adaptive tokens for efficient vision transformer","authors":"Yin","year":"2022","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yue et al., 2021] Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip HS Torr, Wayne Zhang, and Dahua Lin. Vision transformer with progressive sampling. In Proc. ICCV, pages 387-396, 2021.","title":"Vision transformer with progressive sampling","authors":"Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Wayne Philip Hs Torr, Dahua Zhang, Lin","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2022a] Jiaming Zhang, Kailun Yang, Chaoxi- ang Ma, Simon Reiß, Kunyu Peng, and Rainer Stiefelha- gen. Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation. In Proc. CVPR, pages 16917-16927, 2022.","title":"Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation","authors":"Zhang","year":"2022","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2022b] Jiaming Zhang, Kailun Yang, Hao Shi, Simon Reiß, Kunyu Peng, Chaoxiang Ma, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Behind every do- main there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation. arXiv preprint arXiv:2207.11860, 2022.","title":"Behind every domain there is a shift: Adapting distortion-aware vision transformers for panoramic semantic segmentation","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zheng et al., 2023] Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Zhijie Shen, and Yao Zhao. Complementary bi-directional feature compression for indoor 360deg se- mantic segmentation with self-distillation. In Proc. WACV, pages 4501-4510, 2023.","title":"Complementary bi-directional feature compression for indoor 360deg semantic segmentation with self-distillation","authors":"Zheng","year":"2023","venue":"Proc. WACV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhuang et al., 2022] Chuanqing Zhuang, Zhengda Lu, Yiqun Wang, Jun Xiao, and Ying Wang. Acdnet: Adaptively combined dilated convolution for monocular panorama depth estimation. In Proc. AAAI, volume 36, pages 3653-3661, 2022.","title":"Acdnet: Adaptively combined dilated convolution for monocular panorama depth estimation","authors":"Zhuang","year":"2022","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}