{"type":"ScholarlyArticle","title":"CF-Deformable DETR: An End-to-End Alignment-Free Model for Weakly Aligned Visible-Infrared Object Detection","authors":[{"name":"Haolong Fu","slug":"haolong-fu","given":"Haolong","family":"Fu","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"},{"name":"Jin Yuan","slug":"jin-yuan","given":"Jin","family":"Yuan","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"},{"name":"Guojin Zhong","slug":"guojin-zhong","given":"Guojin","family":"Zhong","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"},{"name":"Xuan He","slug":"xuan-he","given":"Xuan","family":"He","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"},{"name":"Jiacheng Lin","slug":"jiacheng-lin","given":"Jiacheng","family":"Lin","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"},{"name":"Zhiyong Li","slug":"zhiyong-li","given":"Zhiyong","family":"Li","affiliation":"School of Information Science and Engineering , Hunan University , Changsha , China"}],"abstract":"Weakly aligned visible-infrared object detection poses significant challenges due to the imprecise alignment between visible and infrared images. Most existing methods explore the alignment strategies between visible and infrared images, yielding unbearable computation costs. This paper first proposes an end-to-end alignment-free architecture Cross-modal Fusion Deformable DEtection TRansformer (``CF-Deformable DETR'') for weakly aligned visible-infrared object detection. Abandoning the traditional image alignment, CF-Deformable DETR introduces a simple yet effective cross-modal deformable attention mechanism to directly implement automatic cross-modal point mapping, generating well-aligned bimodal features with high efficiency. Moreover, we design a Point-level Feature Consistency Loss to guide the cross-modal point mapping, ensuring the consistency of paired features to support the following fusion.  Extensive experiments are conducted on three benchmark datasets. The experimental results demonstrate that CF-Deformable DETR achieves close accuracy on weakly aligned and strictly aligned data as well as maintains stable performance to a certain extent against various offset degrees of weakly aligned data. Code is available at https://github.com/116508/CF-Deformable-DETR.","doi":"10.24963/ijcai.2024/84","url":"https://ijcai.org/proceedings/2024/84","pdf":"https://ijcai.org/proceedings/2024/0084.pdf","markdown":"https://ijcai.org/proceedings/2024/84.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/84","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":84,"pages":{"first":758,"last":766},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Multimodal learning"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2022] Yanpeng Cao, Xing Luo, Jiangxin Yang, Yanlong Cao, and Michael Ying Yang. Locality guided cross-modal feature aggregation and pixel-level fusion for multispectral pedestrian detection. Information Fusion, 88:1-11, 2022.","title":"Locality guided cross-modal feature aggregation and pixel-level fusion for multispectral pedestrian detection","authors":"Cao","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter- national Conference on Machine Learning, pages 1597- 1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022a] Xu Chen, Lei Liu, and Xin Tan. Robust pedestrian detection based on multi-spectral image fusion and convolutional neural networks. Electronics, 11(1):1, 2022.","title":"Robust pedestrian detection based on multi-spectral image fusion and convolutional neural networks","authors":"Chen","year":"2022","venue":"Electronics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022b] Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, and Shu Kong. Mul- timodal object detection via probabilistic ensembling. In European Conference on Computer Vision, pages 139- 158. Springer, 2022.","title":"Multimodal object detection via probabilistic ensembling","authors":"Chen","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2023] Nuo Chen, Jin Xie, Jing Nie, Jiale Cao, Zhuang Shao, and Yanwei Pang. Attentive alignment net- work for multispectral pedestrian detection. In Proceed- ings of the ACM International Conference on Multimedia, pages 3787-3795, 2023.","title":"Attentive alignment network for multispectral pedestrian detection","authors":"Nuo Chen, Jin Xie, Jing Nie, Jiale Cao, Zhuang Shao, Yanwei Pang","year":"2023","venue":"Proceedings of the ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2023] Fuchen Chu, Jiale Cao, Zhanjie Song, Zhuang Shao, Yanwei Pang, and Xuelong Li. Toward gen- eralizable multispectral pedestrian detection. IEEE Trans- actions on Intelligent Transportation Systems, 2023.","title":"Toward generalizable multispectral pedestrian detection","authors":"Chu","year":"2023","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2023] Chenhang Cui, Jinyu Xie, and Yechenhao Yang. Bright channel prior attention for multispectral pedestrian detection. arXiv preprint arXiv:2305.12845, 2023.","title":"Bright channel prior attention for multispectral pedestrian detection","authors":"Cui","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dasgupta et al., 2022] Kinjal Dasgupta, Arindam Das, Sudip Das, Ujjwal Bhattacharya, and Senthil Yogamani. Spatio-contextual deep network-based multimodal pedes- trian detection for autonomous driving. IEEE Transac- tions on Intelligent Transportation Systems, 23(9):15940- 15950, 2022.","title":"Spatio-contextual deep network-based multimodal pedestrian detection for autonomous driving","authors":"Kinjal Dasgupta, Arindam Das, Sudip Das, Ujjwal Bhattacharya, Senthil Yogamani","year":"2022","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng and Ma, 2022] Yuxin Deng and Jiayi Ma. Redfeat: Recoupling detection and description for multimodal fea- ture learning. IEEE Transactions on Image Processing, 32:591-602, 2022.","title":"Redfeat: Recoupling detection and description for multimodal feature learning","authors":"Ma ; ] Yuxin Deng, Jiayi Deng, Ma","year":"2022","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2021] Lu Ding, Yong Wang, Robert Laganiere, Dan Huang, Xinbin Luo, and Huanlong Zhang. A robust and fast multispectral pedestrian detection deep network. Knowledge-Based Systems, 227:106990, 2021.","title":"A robust and fast multispectral pedestrian detection deep network","authors":"Lu Ding, Yong Wang, Robert Laganiere, Dan Huang, Xinbin Luo, Huanlong Zhang","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2022] Xiaoxu Feng, Xiwen Yao, Gong Cheng, and Junwei Han. Weakly supervised rotation-invariant aerial object detection network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14146-14155, 2022.","title":"Weakly supervised rotation-invariant aerial object detection network","authors":"Xiaoxu Feng, Xiwen Yao, Gong Cheng, Junwei Han","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[FLIR, 2018] Teledyne FLIR. Flir thermal dataset for algorithm training. https://www.flir.in/oem/adas/ adas-dataset-form, 2018. Accessed: 2019-08-16.","title":"Teledyne FLIR. Flir thermal dataset for algorithm training","authors":"Flir","year":"2018","venue":null,"doi":null,"url":"https://www.flir.in/oem/adas/adas-dataset-form,2018","internal_url":null},{"position":13,"raw":"[Fu et al., 2023] Haolong Fu, Shixun Wang, Puhong Duan, Changyan Xiao, Renwei Dian, Shutao Li, and Zhiyong Li. Lraf-net: Long-range attention fusion network for visible- infrared object detection. IEEE Transactions on Neural Networks and Learning Systems, 2023.","title":"Lraf-net: Long-range attention fusion network for visibleinfrared object detection","authors":"Fu","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hwang et al., 2015] Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. Multispectral pedestrian detection: Benchmark dataset and baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1037-1045, 2015.","title":"Multispectral pedestrian detection: Benchmark dataset and baseline","authors":"Hwang","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jain et al., 2023] Deepak Kumar Jain, Xudong Zhao, Germán González-Almagro, Chenquan Gan, and Ketan Kotecha. Multimodal pedestrian detection using meta- heuristics with deep convolutional neural network in crowded scenes. Information Fusion, 95:401-414, 2023.","title":"Multimodal pedestrian detection using metaheuristics with deep convolutional neural network in crowded scenes","authors":"Deepak Kumar Jain, Xudong Zhao, Germán González-Almagro, Chenquan Gan, Ketan Kotecha","year":"2023","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jia et al., 2021] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Llvip: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3496-3504, 2021.","title":"Llvip: A visible-infrared paired dataset for low-light vision","authors":"Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, Wenli Zhou","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khrulkov et al., 2020] Valentin Khrulkov, Leyla Mir- vakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6418-6428, 2020.","title":"Hyperbolic image embeddings","authors":"Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, Victor Lempitsky","year":"2020","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim et al., 2021] Jiwon Kim, Hyeongjun Kim, Taejoo Kim, Namil Kim, and Yukyung Choi. Mlpd: Multi-label pedes- trian detector in multispectral domain. IEEE Robotics and Automation Letters, 6(4):7846-7853, 2021.","title":"Mlpd: Multi-label pedestrian detector in multispectral domain","authors":"Jiwon Kim, Hyeongjun Kim, Taejoo Kim, Namil Kim, Yukyung Choi","year":"2021","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kim et al., 2022] Jung Uk Kim, Sungjune Park, and Yong Man Ro. Towards versatile pedestrian detector with multisensory-matching and multispectral recalling mem- ory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1157-1165, 2022.","title":"Towards versatile pedestrian detector with multisensory-matching and multispectral recalling memory","authors":"Kim","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2022a] Qing Li, Changqing Zhang, Qinghua Hu, Huazhu Fu, and Pengfei Zhu. Confidence-aware fusion using dempster-shafer theory for multispectral pedestrian detection. IEEE Transactions on Multimedia, 2022.","title":"Confidence-aware fusion using dempster-shafer theory for multispectral pedestrian detection","authors":"Li","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2022b] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfu- sion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17182- 17191, 2022.","title":"Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection","authors":"Li","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2016] Jingjing Liu, Shaoting Zhang, Shu Wang, and Dimitris N Metaxas. Multispectral deep neu- ral networks for pedestrian detection. arXiv preprint arXiv:1611.02644, 2016.","title":"Multispectral deep neural networks for pedestrian detection","authors":"Jingjing Liu, Shaoting Zhang, Shu Wang, Dimitris N Metaxas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021] Tianshan Liu, Kin-Man Lam, Rui Zhao, and Guoping Qiu. Deep cross-modal representation learn- ing and distillation for illumination-invariant pedestrian detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(1):315-329, 2021.","title":"Deep cross-modal representation learning and distillation for illumination-invariant pedestrian detection","authors":"Liu","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qingyun and Zhaokui, 2022] Fang Qingyun and Wang Zhaokui. Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery. Pattern Recognition, page 108786, 2022.","title":"Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery","authors":"Zhaokui Qingyun, Fang Qingyun, Wang Zhaokui","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qingyun et al., 2021] Fang Qingyun, Han Dapeng, and Wang Zhaokui. Cross-modality fusion transformer for multispectral object detection. arXiv preprint arXiv:2111.00273, 2021.","title":"Cross-modality fusion transformer for multispectral object detection","authors":"Fang Qingyun, Han Dapeng, Wang Zhaokui","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022] Qingwang Wang, Yongke Chi, Tao Shen, Jian Song, Zifeng Zhang, and Yan Zhu. Improving rgb- infrared pedestrian detection by reducing cross-modality redundancy. In IEEE International Conference on Image Processing, pages 526-530, 2022.","title":"Improving rgbinfrared pedestrian detection by reducing cross-modality redundancy","authors":"Wang","year":"2022","venue":"IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xing et al., 2023] Yinghui Xing, Song Wang, Guoqiang Liang, Qingyi Li, Xiuwei Zhang, Shizhou Zhang, and Yanning Zhang. Multispectral pedestrian detection via reference box constrained cross attention and modality balanced optimization. arXiv preprint arXiv:2302.00290, 2023.","title":"Multispectral pedestrian detection via reference box constrained cross attention and modality balanced optimization","authors":"Yinghui Xing, Song Wang, Guoqiang Liang, Qingyi Li, Xiuwei Zhang, Shizhou Zhang, Yanning Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2022] Xiaoxiao Yang, Yeqiang Qian, Huijie Zhu, Chunxiang Wang, and Ming Yang. Baanet: Learn- ing bi-directional adaptive attention gates for multispectral pedestrian detection. In 2022 International Conference on Robotics and Automation, pages 2920-2926. IEEE, 2022.","title":"Baanet: Learning bi-directional adaptive attention gates for multispectral pedestrian detection","authors":"Xiaoxiao Yang, Yeqiang Qian, Huijie Zhu, Chunxiang Wang, Ming Yang","year":"2022","venue":"2022 International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yaras et al., 2022] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with nor- malized features: A geometric analysis over the rieman- nian manifold. Advances in Neural Information Process- ing Systems, 35:11547-11560, 2022.","title":"Neural collapse with normalized features: A geometric analysis over the riemannian manifold","authors":"Yaras","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2019] Lu Zhang, Xiangyu Zhu, Xiangyu Chen, Xu Yang, Zhen Lei, and Zhiyong Liu. Weakly aligned cross-modal learning for multispectral pedestrian detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5127-5137, 2019.","title":"Weakly aligned cross-modal learning for multispectral pedestrian detection","authors":"Lu Zhang, Xiangyu Zhu, Xiangyu Chen, Xu Yang, Zhen Lei, Zhiyong Liu","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2020] Heng Zhang, Elisa Fromont, Sébastien Lefèvre, and Bruno Avignon. Multispectral fusion for ob- ject detection with cyclic fuse-and-refine blocks. In IEEE International Conference on Image Processing, pages 276-280, 2020.","title":"Multispectral fusion for object detection with cyclic fuse-and-refine blocks","authors":"Heng Zhang, Elisa Fromont, Sébastien Lefèvre, Bruno Avignon","year":"2020","venue":"IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Lu Zhang, Zhiyong Liu, Xiangyu Zhu, Zhan Song, Xu Yang, Zhen Lei, and Hong Qiao. Weakly aligned feature fusion for multimodal object detection. IEEE Transactions on Neural Networks and Learning Sys- tems, 2021.","title":"Weakly aligned feature fusion for multimodal object detection","authors":"Lu Zhang, Zhiyong Liu, Xiangyu Zhu, Zhan Song, Xu Yang, Zhen Lei, Hong Qiao","year":"2021","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022] Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, and Xiang Ruan. Visible-thermal uav track- ing: A large-scale benchmark and new baseline. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8886-8895, 2022.","title":"Visible-thermal uav tracking: A large-scale benchmark and new baseline","authors":"Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, Xiang Ruan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2020] Kailai Zhou, Linsen Chen, and Xun Cao. Improving multispectral pedestrian detection by address- ing modality imbalance problems. In European Confer- ence on Computer Vision, pages 787-803. Springer, 2020.","title":"Improving multispectral pedestrian detection by addressing modality imbalance problems","authors":"Kailai Zhou, Linsen Chen, Xun Cao","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}