{"type":"ScholarlyArticle","title":"AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection","authors":[{"name":"Zehui Chen","slug":"zehui-chen","given":"Zehui","family":"Chen","affiliation":"University of Science and Technology of China"},{"name":"Zhenyu Li","slug":"zhenyu-li","given":"Zhenyu","family":"Li","affiliation":"Harbin Institute of Technology"},{"name":"Shiquan Zhang","slug":"shiquan-zhang","given":"Shiquan","family":"Zhang","affiliation":"SenseTime Research"},{"name":"Liangji Fang","slug":"liangji-fang","given":"Liangji","family":"Fang","affiliation":"SenseTime Research"},{"name":"Qinhong Jiang","slug":"qinhong-jiang","given":"Qinhong","family":"Jiang","affiliation":"SenseTime Research"},{"name":"Feng Zhao","slug":"feng-zhao","given":"Feng","family":"Zhao","affiliation":"University of Science and Technology of China"},{"name":"Bolei Zhou","slug":"bolei-zhou","given":"Bolei","family":"Zhou","affiliation":"The Chinese University of Hong Kong"},{"name":"Hang Zhao","slug":"hang-zhao","given":"Hang","family":"Zhao","affiliation":"Tsinghua University"}],"abstract":"Object detection through either RGB images or the LiDAR point clouds has been extensively explored in autonomous driving. However, it remains challenging to make these two data sources complementary and beneficial to each other.  In this paper, we propose AutoAlign, an automatic feature fusion strategy for 3D object detection. Instead of establishing deterministic correspondence with camera projection matrix, we model the mapping relationship between the image and point clouds with a learnable alignment map. This map enables our model to automate the alignment of non-homogenous features in a dynamic and data-driven manner. Specifically, a cross-attention feature alignment module is devised to adaptively aggregate pixel-level image features for each voxel. To enhance the semantic consistency during feature alignment, we also design a self-supervised cross-modal feature interaction module, through which the model can learn feature aggregation with instance-level feature guidance. Extensive experimental results show that our approach can lead to 2.3 mAP and 7.0 mAP improvements on the KITTI and nuScenes datasets respectively. Notably, our best model reaches 70.9 NDS on the nuScenes testing leaderboard, achieving competitive performance among various state-of-the-arts.","doi":"10.24963/ijcai.2022/116","url":"https://ijcai.org/proceedings/2022/116","pdf":"https://ijcai.org/proceedings/2022/0116.pdf","markdown":"https://ijcai.org/proceedings/2022/116.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/116","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":116,"pages":{"first":827,"last":833},"keywords":["Computer Vision: Recognition (object detection, categorization)","Machine Learning: Multi-modal learning","Computer Vision: 3D Computer Vision"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986591","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984734","length_sec":305,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Explor- ing simple siamese representation learning. In CVPR, pages 15750-15758, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3D object detection for autonomous driving. In CVPR, pages 2147-2156, 2016.","title":"Monocular 3D object detection for autonomous driving","authors":"Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, Raquel Urtasun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3D object detection net- work for autonomous driving. In CVPR, pages 1907-1915, 2017.","title":"Multi-view 3D object detection network for autonomous driving","authors":"Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Tian Xia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Contributors, 2020] MMDet3D Contributors. MMDetec- tion3D: OpenMMLab next-generation platform for gen- eral 3D object detection. https://github.com/open-mmlab/ mmdetection3d, 2020. Accessed: 2020-07-23.","title":"MMDet3D Contributors. MMDetection3D: OpenMMLab next-generation platform for general 3D object detection","authors":null,"year":"2020","venue":null,"doi":null,"url":"https://github.com/open-mmlab/mmdetection3d","internal_url":null},{"position":5,"raw":"[Fan et al., 2021] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. RangeDet: In defense of range view for LiDAR-based 3D object detection. arXiv preprint arXiv:2103.10039, 2021.","title":"RangeDet: In defense of range view for LiDAR-based 3D object detection","authors":"Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, Zhaoxiang Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2961-2969, 2017.","title":"Mask R-CNN","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2020] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. EPNet: Enhancing point features with im- age semantics for 3D object detection. In ECCV, pages 35-52, 2020.","title":"EPNet: Enhancing point features with image semantics for 3D object detection","authors":"Huang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ku et al., 2018] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3D pro- posal generation and object detection from view aggrega- tion. In IROS, pages 1-8, 2018.","title":"Joint 3D proposal generation and object detection from view aggregation","authors":"Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, Steven L Waslander","year":"2018","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lang et al., 2019] Alex H Lang, Sourabh Vora, Holger Cae- sar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- Pillars: Fast encoders for object detection from point clouds. In CVPR, pages 12697-12705, 2019.","title":"Point-Pillars: Fast encoders for object detection from point clouds","authors":"Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, Oscar Beijbom","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Peiliang Li, Xiaozhi Chen, and Shaojie Shen. Stereo R-CNN based 3D object detection for au- tonomous driving. In CVPR, pages 7644-7652, 2019.","title":"Stereo R-CNN based 3D object detection for autonomous driving","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2018] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi- sensor 3D object detection. In ECCV, pages 641-656, 2018.","title":"Deep continuous fusion for multisensor 3D object detection","authors":"Ming Liang, Bin Yang, Shenlong Wang, Raquel Urtasun","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mousavian et al., 2017] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3D bounding box estimation using deep learning and geometry. In CVPR, pages 7074-7082, 2017.","title":"3D bounding box estimation using deep learning and geometry","authors":"Arsalan Mousavian, Dragomir Anguelov, John Flynn, Jana Kosecka","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pang et al., 2020] Su Pang, Daniel Morris, and Hayder Radha. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection. In IROS, pages 1-10, 2020.","title":"CLOCs: Camera-LiDAR object candidates fusion for 3D object detection","authors":"Su Pang, Daniel Morris, Hayder Radha","year":"2020","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2018] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum PointNets for 3D object detection from RGB-D data. In CVPR, pages 918- 927, 2018.","title":"Frustum PointNets for 3D object detection from RGB-D data","authors":"Wei Charles R Qi, Chenxia Liu, Hao Wu, Leonidas J Su, Guibas","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. NeurIPS, 28:91-99, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi et al., 2019] Shaoshuai Shi, Xiaogang Wang, and Hong- sheng Li. PointRCNN: 3D object proposal generation and detection from point cloud. In CVPR, pages 770-779, 2019.","title":"PointRCNN: 3D object proposal generation and detection from point cloud","authors":"Shaoshuai Shi, Xiaogang Wang, Hongsheng Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2020] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020.","title":"From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network","authors":"Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, Hongsheng Li","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sindagi et al., 2019] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. MVX-Net: Multimodal VoxelNet for 3D ob- ject detection. In ICRA, pages 7276-7282, 2019.","title":"MVX-Net: Multimodal VoxelNet for 3D object detection","authors":"A Vishwanath, Yin Sindagi, Oncel Zhou, Tuzel","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vora et al., 2020] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. PointPainting: Sequential fu- sion for 3D object detection. In CVPR, pages 4604-4612, 2020.","title":"PointPainting: Sequential fusion for 3D object detection","authors":"Sourabh Vora, Alex H Lang, Bassam Helou, Oscar Beijbom","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Solomon, 2021] Yue Wang and Justin M Solomon. Object DGCNN: 3D object detection using dynamic graphs. NeurIPS, 34, 2021.","title":"Object DGCNN: 3D object detection using dynamic graphs","authors":"Yue Wang, Justin M Solomon","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan et al., 2018] Yan Yan, Yuxing Mao, and Bo Li. SEC- OND: Sparsely embedded convolutional detection. Sen- sors, pages 3337-2247, 2018.","title":"SEC-OND: Sparsely embedded convolutional detection","authors":"Yan Yan, Yuxing Mao, Bo Li","year":"2018","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2021a] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3D object detection and track- ing. In CVPR, pages 11784-11793, 2021.","title":"Center-based 3D object detection and tracking","authors":"Yin","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin et al., 2021b] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3D detection. NeurIPS, 34, 2021.","title":"Multimodal virtual point 3D detection","authors":"Yin","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yoo et al., 2020] Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection. In ECCV, pages 720-736, 2020.","title":"3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection","authors":"Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, Jun Won Choi","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[You et al., 2019] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving. arXiv preprint arXiv:1906.06310, 2019.","title":"Pseudo-LiDAR++: Accurate depth for 3D object detection in autonomous driving","authors":"Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou and Tuzel, 2018] Yin Zhou and Oncel Tuzel. Voxel- Net: End-to-end learning for point cloud based 3D object detection. In CVPR, pages 4490-4499, 2018.","title":"Voxel-Net: End-to-end learning for point cloud based 3D object detection","authors":"Tuzel ; ] Yin Zhou, Oncel Zhou, Tuzel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}