{"type":"ScholarlyArticle","title":"Do You Remember the Future? Weak-to-Strong Generalization in 3D Object Detection","authors":[{"name":"Alexander Gambashidze","slug":"alexander-gambashidze","given":"Alexander","family":"Gambashidze","affiliation":"Artificial Intelligence Research Institute; HSE University"},{"name":"Aleksandr Dadukin","slug":"aleksandr-dadukin","given":"Aleksandr","family":"Dadukin","affiliation":"HSE University"},{"name":"Maxim Golyadkin","slug":"maxim-golyadkin","given":"Maxim","family":"Golyadkin","affiliation":"Artificial Intelligence Research Institute; HSE University"},{"name":"Maria Razzhivina","slug":"maria-razzhivina","given":"Maria","family":"Razzhivina","affiliation":"HSE University"},{"name":"Ilya Makarov","slug":"ilya-makarov","given":"Ilya","family":"Makarov","affiliation":"Artificial Intelligence Research Institute; ISP RAS Research Center for Trusted Artificial Intelligence"}],"abstract":"This paper demonstrates a novel method for LiDAR-based 3D object detection, addressing major field challenges: sparsity and occlusion. Our approach leverages temporal point cloud sequences to generate frames that provide comprehensive views of objects from multiple angles. To address the challenge of generating these frames in real-time, we employ Knowledge Distillation within a Teacher-Student framework, allowing the Student model to emulate the Teacher’s advanced perception. We pioneered the application of weak-to-strong generalization in computer vision by training our Teacher model on enriched, object-complete data. In this demo, we showcase the exceptional quality of labels produced by the X-Ray Teacher on object-complete frames, showing our method distilling its knowledge to enhance object 3D detection models.","doi":"10.24963/ijcai.2024/1001","url":"https://ijcai.org/proceedings/2024/1001","pdf":"https://ijcai.org/proceedings/2024/1001.pdf","markdown":"https://ijcai.org/proceedings/2024/1001.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1001","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1001,"pages":{"first":8653,"last":8656},"keywords":["Computer Vision: CV: 3D computer vision","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.","title":"nuscenes: A multimodal dataset for autonomous driving","authors":"Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2023] Hyeon Cho, Junyong Choi, Geonwoo Baek, and Wonjun Hwang. itkd: Interchange transfer- based knowledge distillation for 3d object detection. In CVPR, 2023.","title":"itkd: Interchange transferbased knowledge distillation for 3d object detection","authors":"Hyeon Cho, Junyong Choi, Geonwoo Baek, Wonjun Hwang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gambashidze et al., 2024] Alexander Gambashidze, Alek- sandr Dadukin, Maksim Golyadkin, Maria Razzhivina, and Ilya Makarov. Weak-to-strong 3d object detection with x-ray distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1- 15, 2024.","title":"Weak-to-strong 3d object detection with x-ray distillation","authors":"Alexander Gambashidze, Aleksandr Dadukin, Maksim Golyadkin, Maria Razzhivina, Ilya Makarov","year":"2024","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Geoffrey Hinton, 2015] Jeff Dean Geoffrey Hinton, Oriol Vinyals. Distilling the knowledge in a neural network. In arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Jeff Dean, Geoffrey Hinton, Oriol Vinyals","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Indyk and Makarov, 2023] Ilia Indyk and Ilya Makarov. Monovan: Visual attention for self-supervised monocular depth estimation. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 1211- 1220. IEEE, 2023.","title":"Monovan: Visual attention for self-supervised monocular depth estimation","authors":"Makarov ; ] Ilia Indyk, Ilya Indyk, Makarov","year":"2023","venue":"IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karpov and Makarov, 2022] Aleksei Karpov and Ilya Makarov. Exploring efficiency of vision transformers for self-supervised monocular depth estimation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 711-719, 2022.","title":"Exploring efficiency of vision transformers for self-supervised monocular depth estimation","authors":"Makarov Karpov, Aleksei Karpov, Ilya Makarov","year":"2022","venue":"IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Luginov and Makarov, 2023] Albert Luginov and Ilya Makarov. Swiftdepth: An efficient hybrid cnn-transformer model for self-supervised monocular depth estima- tion on mobile devices. In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 642-647. IEEE, 2023.","title":"Albert Luginov and Ilya Makarov. Swiftdepth: An efficient hybrid cnn-transformer model for self-supervised monocular depth estimation on mobile devices","authors":"Makarov Luginov","year":"2023","venue":"IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Qi et al., 2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Qi et al., 2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sun et al., 2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vi- jay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Alek- sei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for au- tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2020.","title":"Scalability in perception for autonomous driving: Waymo open dataset","authors":"Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, Dragomir Anguelov","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Wang et al., 2023] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with ro- tated sets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13520-13529, June 2023.","title":"Dsvt: Dynamic sparse voxel transformer with rotated sets","authors":"Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, Liwei Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Xu et al., 2022] Qiangeng Xu, Yiqi Zhong, and Ulrich Neu- mann. Behind the curtain: Learning occluded shapes for 3d object detection. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 36 (3), pages 2893- 2901, 2022.","title":"Behind the curtain: Learning occluded shapes for 3d object detection","authors":"Xu","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Yang et al., 2018] Bin Yang, Wenjie Luo, and Raquel Ur- tasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Com- puter Vision and Pattern Recognition, pages 7652-7660, 2018.","title":"Pixor: Real-time 3d object detection from point clouds","authors":"Bin Yang, Wenjie Luo, Raquel Urtasun","year":"2018","venue":"Proceedings of the IEEE conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Yang et al., 2022] Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Towards efficient 3d object detection with knowledge distillation. In NeurIPS, 2022.","title":"Towards efficient 3d object detection with knowledge distillation","authors":"Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, Xiaojuan Qi","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Yin et al., 2021] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and track- ing. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 11784-11793, 2021.","title":"Center-based 3d object detection and tracking","authors":"Xingyi Tianwei Yin, Philipp Zhou, Krahenbuhl","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zhou et al., 2022] Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Cen- terformer: Center-based transformer for 3d object detection. In European Conference on Computer Vision, pages 496-513. Springer, 2022.","title":"Centerformer: Center-based transformer for 3d object detection","authors":"Zhou","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}