{"type":"ScholarlyArticle","title":"OTOcc: Optimal Transport for Occupancy Prediction","authors":[{"name":"Pengteng Li","slug":"pengteng-li","given":"Pengteng","family":"Li","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"},{"name":"Ying He","slug":"ying-he","given":"Ying","family":"He","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China"},{"name":"F. Richard Yu","slug":"f-richard-yu","given":"F. Richard","family":"Yu","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)"},{"name":"Pinhao Song","slug":"pinhao-song","given":"Pinhao","family":"Song","affiliation":"KU Leuven"},{"name":"Xingchen Zhou","slug":"xingchen-zhou","given":"Xingchen","family":"Zhou","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ); Deeproute Inc."},{"name":"Guang Zhou","slug":"guang-zhou","given":"Guang","family":"Zhou","affiliation":"Deeproute Inc."}],"abstract":"The autonomous driving community is highly interested in 3D occupancy prediction due to its outstanding geometric perception and object recognition capabilities. However, previous methods are limited to existing semantic conversion mechanisms for solving sparse ground truths problem, causing excessive computational demands and sub-optimal voxels representation. To tackle the above limitations, we propose OTOcc, a novel 3D occupancy prediction framework that models semantic conversion from 2D pixels to 3D voxels as Optimal Transport (OT) problem, offering accurate semantic mapping to adapt to sparse scenarios without attention or depth estimation. Specifically, the unit transportation cost between each demander (voxel) and supplier (pixel) pair is defined as the weighted occupancy prediction loss. Then, we utilize the Sinkhorn-Knopp Iteration to find the best mapping matrices with minimal transportation costs. To reduce the computational cost, we propose a block reading technique with multi-perspective feature representation, which also brings fine-grained scene understanding. Extensive experiments show that OTOcc not only has the competitive prediction performance but also has about more than 4.58% reduction in computational overhead compared to state-of-the-art methods.","doi":"10.24963/ijcai.2024/112","url":"https://ijcai.org/proceedings/2024/112","pdf":"https://ijcai.org/proceedings/2024/0112.pdf","markdown":"https://ijcai.org/proceedings/2024/112.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/112","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":112,"pages":{"first":1010,"last":1019},"keywords":["Computer Vision: CV: Machine learning for vision","Computer Vision: CV: 3D computer vision","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Berman et al., 2018] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection- over-union measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4413-4421, 2018.","title":"The lovász-softmax loss: A tractable surrogate for the optimization of the intersectionover-union measure in neural networks","authors":"Berman","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driv- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11621- 11631, 2020.","title":"nuscenes: A multimodal dataset for autonomous driving","authors":"Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao and others, 2022] Cao et al. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3991-4001, 2022.","title":"Monoscene: Monocular 3d semantic scene completion","authors":"Cao, Cao","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cortinhal and others, 2020] Cortinhal et al. Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In Advances in Visual Computing: 15th Interna- tional Symposium, ISVC 2020, San Diego, CA, USA, Oc- tober 5-7, 2020, Proceedings, Part II 15, pages 207-222. Springer, 2020.","title":"Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds","authors":"Cortinhal","year":"2020","venue":"Advances in Visual Computing: 15th International Symposium, ISVC 2020","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. Advances in Neu- ral Information Processing Systems, 26, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ge et al., 2021] Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 303-312, 2021.","title":"Ota: Optimal transport assignment for object detection","authors":"Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, Jian Sun","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2021] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.","title":"Bevdet: High-performance multi-camera 3d object detection in bird-eye-view","authors":"Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, Dalong Du","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2023] Yuanhui Huang, Wenzhao Zheng, Yun- peng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9223-9232, 2023.","title":"Tri-perspective view for vision-based 3d semantic occupancy prediction","authors":"Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, Jiwen Lu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2023] Yupeng Jia, Jie He, Runze Chen, Fang Zhao, and Haiyong Luo. Occupancydetr: Making seman- tic scene completion as straightforward as object detec- tion. arXiv preprint arXiv:2309.08504, 2023.","title":"Occupancydetr: Making semantic scene completion as straightforward as object detection","authors":"Yupeng Jia, Jie He, Runze Chen, Fang Zhao, Haiyong Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022a] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Scan++: Enhanced semantic conditioned adaptation for domain adaptive object detection. IEEE Transactions on Multimedia, 2022.","title":"Scan++: Enhanced semantic conditioned adaptation for domain adaptive object detection","authors":"Li","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022b] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transform- ers. In European Conference on Computer Vision, pages 1-18. Springer, 2022.","title":"Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers","authors":"Li","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2023a] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel trans- former for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9087-9098, 2023.","title":"Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion","authors":"Li","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2023b] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhanc- ing depth estimation in multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1486-1494, 2023.","title":"Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo","authors":"Li","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023c] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zem- ing Li. Bevdepth: Acquisition of reliable depth for multi- view 3d object detection. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 1477-1485, 2023.","title":"Bevdepth: Acquisition of reliable depth for multiview 3d object detection","authors":"Li","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2023d] Zhiqi Li, Zhiding Yu, David Austin, Ming- sheng Fang, Shiyi Lan, Jan Kautz, and Jose M Al- varez. Fb-occ: 3d occupancy prediction based on forward-backward view transformation. arXiv preprint arXiv:2307.01492, 2023.","title":"Fb-occ: 3d occupancy prediction based on forward-backward view transformation","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2023] Yuhang Lu, Xinge Zhu, Tai Wang, and Yuexin Ma. Octreeocc: Efficient and multi-granularity oc- cupancy prediction using octree queries. arXiv preprint arXiv:2312.03774, 2023.","title":"Octreeocc: Efficient and multi-granularity occupancy prediction using octree queries","authors":"Yuhang Lu, Xinge Zhu, Tai Wang, Yuexin Ma","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Miao et al., 2023a] Ruihang Miao, Weizhou Liu, Mingrui Chen, Zheng Gong, Weixin Xu, Chen Hu, and Shuchang Zhou. Occdepth: A depth-aware method for 3d seman- tic scene completion. arXiv preprint arXiv:2302.13540, 2023.","title":"Occdepth: A depth-aware method for 3d semantic scene completion","authors":"Miao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miao et al., 2023b] Ruihang Miao, Weizhou Liu, Mingrui Chen, Zheng Gong, Weixin Xu, Chen Hu, and Shuchang Zhou. Occdepth: A depth-aware method for 3d seman- tic scene completion. arXiv preprint arXiv:2302.13540, 2023.","title":"Occdepth: A depth-aware method for 3d semantic scene completion","authors":"Miao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Milioto et al., 2019] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet++: Fast and ac- curate lidar semantic segmentation. In 2019 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 4213-4220. IEEE, 2019.","title":"Rangenet++: Fast and accurate lidar semantic segmentation","authors":"Andres Milioto, Ignacio Vizzo, Jens Behley, Cyrill Stachniss","year":"2019","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ouyang et al., 2024] Wenzhe Ouyang, Xiaolin Song, Bailan Feng, and Zenglin Xu. Octocc: High-resolution 3d occupancy prediction with octree. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4369-4377, 2024.","title":"Octocc: High-resolution 3d occupancy prediction with octree","authors":"Ouyang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pan et al., 2023] Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Li Liu, and Shanghang Zhang. Renderocc: Vision-centric 3d occupancy pre- diction with 2d rendering supervision. arXiv preprint arXiv:2309.09502, 2023.","title":"Renderocc: Vision-centric 3d occupancy prediction with 2d rendering supervision","authors":"Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Li Liu, Shanghang Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Qi et al., 2023] Zhangyang Qi, Jiaqi Wang, Xiaoyang Wu, and Hengshuang Zhao. Ocbev: Object-centric bev trans- former for multi-view 3d object detection. arXiv preprint arXiv:2306.01738, 2023.","title":"Ocbev: Object-centric bev transformer for multi-view 3d object detection","authors":"Qi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tan et al., 2023] Zhiyu Tan, Zichao Dong, Cheng Zhang, Weikun Zhang, Hang Ji, and Hao Li. Ovo: Open- vocabulary occupancy. arXiv preprint arXiv:2305.16133, 2023.","title":"Ovo: Openvocabulary occupancy","authors":"Zhiyu Tan, Zichao Dong, Cheng Zhang, Weikun Zhang, Hang Ji, Hao Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian et al., 2023] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driv- ing. arXiv preprint arXiv:2304.14365, 2023.","title":"Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving","authors":"Xiaoyu Tian, Tao Jiang, Longfei Yun, Yue Wang, Yilun Wang, Hang Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2023] Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, and Zhaoxiang Zhang. Panoocc: Unified oc- cupancy representation for camera-based 3d panoptic seg- mentation. arXiv preprint arXiv:2306.10013, 2023.","title":"Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation","authors":"Yuqi Wang, Yuntao Chen, Xingyu Liao, Lue Fan, Zhaoxiang Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2021] Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 16024- 16033, 2021.","title":"Rpvnet: A deep and efficient range-point-voxel fusion network for lidar point cloud segmentation","authors":"Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, Shiliang Pu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2023] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 17830-17839, 2023.","title":"Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision","authors":"Chenyu Yang, Yuntao Chen, Chenxin Hao Tian, Xizhou Tao, Zhaoxiang Zhu, Gao Zhang, Hongyang Huang, Yu Li, Lewei Qiao, Lu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and others, 2023] Zhang et al. Occformer: Dual-path transformer for vision-based 3d semantic occupancy pre- diction. arXiv preprint arXiv:2304.05316, 2023.","title":"Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2020] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Has- san Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601-9610, 2020.","title":"Polarnet: An improved grid representation for online lidar point clouds semantic segmentation","authors":"Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, Hassan Foroosh","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2022] Hongyu Zhou, Zheng Ge, Zeming Li, and Xiangyu Zhang. Matrixvt: Efficient multi-camera to bev transformation for 3d perception. arXiv preprint arXiv:2211.10593, 2022.","title":"Matrixvt: Efficient multi-camera to bev transformation for 3d perception","authors":"Hongyu Zhou, Zheng Ge, Zeming Li, Xiangyu Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2021] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-9948, 2021.","title":"Cylindrical and asymmetrical 3d convolution networks for lidar segmentation","authors":"Zhu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}