{"type":"ScholarlyArticle","title":"Unified Single-Stage Transformer Network for Efficient RGB-T Tracking","authors":[{"name":"Jianqiang Xia","slug":"jianqiang-xia","given":"Jianqiang","family":"Xia","affiliation":"Intelligent Game and Decision Lab , China; Tianjin Artificial Intelligence Innovation Center , China"},{"name":"Dianxi Shi","slug":"dianxi-shi","given":"Dianxi","family":"Shi","affiliation":"Intelligent Game and Decision Lab , China"},{"name":"Ke Song","slug":"ke-song","given":"Ke","family":"Song","affiliation":"School of Control Science and Engineering , Shandong University , China"},{"name":"Linna Song","slug":"linna-song","given":"Linna","family":"Song","affiliation":"College of Computer , National University of Defense Technology , China"},{"name":"Xiaolei Wang","slug":"xiaolei-wang","given":"Xiaolei","family":"Wang","affiliation":"Intelligent Game and Decision Lab , China"},{"name":"Songchang Jin","slug":"songchang-jin","given":"Songchang","family":"Jin","affiliation":"Intelligent Game and Decision Lab , China"},{"name":"Chenran Zhao","slug":"chenran-zhao","given":"Chenran","family":"Zhao","affiliation":"College of Computer , National University of Defense Technology , China"},{"name":"Yu Cheng","slug":"yu-cheng","given":"Yu","family":"Cheng","affiliation":"Department of Electrical and Computer Engineering , National University of Singapore , Singapore"},{"name":"Lei Jin","slug":"lei-jin","given":"Lei","family":"Jin","affiliation":"Electronic Engineering Institute , Beijing University of Posts and Telecommunications , China"},{"name":"Zheng Zhu","slug":"zheng-zhu","given":"Zheng","family":"Zhu","affiliation":"Department of Computer Science and Technology , Tsinghua University , China"},{"name":"Jianan Li","slug":"jianan-li","given":"Jianan","family":"Li","affiliation":"School of Optoelectronics , Beijing Institute of Technology , China"},{"name":"Gang Wang","slug":"gang-wang","given":"Gang","family":"Wang","affiliation":"Beijing Institute of Basic Medical Sciences , China"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"Department of Computer Science and Technology , Tsinghua University , China; Telecom , China"},{"name":"Jian Zhao","slug":"jian-zhao","given":"Jian","family":"Zhao","affiliation":"EVOL Lab , Institute of AI (TeleAI) , China; School of Artificial Intelligence, Optics and Electronics , Northwestern Polytechnical University , China"}],"abstract":"Most existing RGB-T tracking networks extract modality features in a separate manner, which lacks interaction and mutual guidance between modalities. This limits the network's ability to adapt to the diverse dual-modality appearances of targets and the dynamic relationships between the modalities. Additionally, the three-stage fusion tracking paradigm followed by these networks significantly restricts the tracking speed. To overcome these problems, we propose a unified single-stage Transformer RGB-T tracking network, namely USTrack, which unifies the above three stages into a single ViT (Vision Transformer) backbone through joint feature extraction, fusion and relation modeling. With this structure, the network can not only extract the fusion features of templates and search regions under the interaction of modalities, but also significantly improve tracking speed through the single-stage fusion tracking paradigm. Furthermore, we introduce a novel feature selection mechanism based on modality reliability to mitigate the influence of invalid modalities for final prediction. Extensive experiments on three mainstream RGB-T tracking benchmarks show that our method achieves the new state-of-the-art while achieving the fastest tracking speed of 84.2FPS. Code is available at https://github.com/xiajianqiang/USTrack.","doi":"10.24963/ijcai.2024/163","url":"https://ijcai.org/proceedings/2024/163","pdf":"https://ijcai.org/proceedings/2024/0163.pdf","markdown":"https://ijcai.org/proceedings/2024/163.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/163","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":163,"pages":{"first":1471,"last":1479},"keywords":["Computer Vision: CV: Video analysis and understanding","Computer Vision: CV: Motion and tracking","Computer Vision: CV: Multimodal learning"],"videos":[],"references":[{"position":1,"raw":"[Bertinetto et al., 2016] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H. S. Torr. Fully-convolutional siamese networks for object tracking. ArXiv, abs/1606.09549, 2016.","title":"Fully-convolutional siamese networks for object tracking","authors":"Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, Philip H S Torr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhat et al., 2019] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. ICCV, pages 6181-6190, 2019.","title":"Learning discriminative model prediction for tracking","authors":"Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2018] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. CVPR, pages 5369- 5378, 2018.","title":"Lasot: A high-quality benchmark for large-scale single object tracking","authors":"Liting Heng Fan, Fan Lin, Peng Yang, Ge Chu, Sijia Deng, Hexin Yu, Yong Bai, Chunyuan Xu, Haibin Liao, Ling","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2019] Yuan Gao, Chenglong Li, Yabin Zhu, Jin Tang, Tao He, and Futian Wang. Deep adaptive fusion net- work for high performance rgbt tracking. ICCVW, pages 91-99, 2019.","title":"Deep adaptive fusion network for high performance rgbt tracking","authors":"Yuan Gao, Chenglong Li, Yabin Zhu, Jin Tang, Tao He, Futian Wang","year":"2019","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2015] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pages 770-778, 2015.","title":"Deep residual learning for image recognition","authors":"X Kaiming He, Shaoqing Zhang, Jian Ren, Sun","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hou et al., 2022] Ruichao Hou, Tongwei Ren, and Gang- shan Wu. Mirnet: A robust rgbt tracking jointly with multi- modal interaction and refinement. ICME, pages 1-6, 2022.","title":"Mirnet: A robust rgbt tracking jointly with multimodal interaction and refinement","authors":"Hou","year":"2022","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2018] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. TPAMI, 43:1562-1577, 2018.","title":"Got-10k: A large high-diversity benchmark for generic object tracking in the wild","authors":"Huang","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hui et al., 2023] Tianrui Hui, Zizheng Xun, Fengguang Peng, Junshi Huang, Xiaoming Wei, Xiaolin Wei, Jiao Dai, Jizhong Han, and Si Liu. Bridging search region in- teraction with template for rgb-t tracking. In CVPR, pages 13630-13639, 2023.","title":"Bridging search region interaction with template for rgb-t tracking","authors":"Hui","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jung et al., 2018] Ilchae Jung, Jeany Son, Mooyeol Baek, and Bohyung Han. Real-time mdnet. In ECCV, 2018.","title":"Real-time mdnet","authors":"Jung","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kristan et al., 2019] Matej Kristan, Jiri Matas, et al. The seventh visual object tracking vot2019 challenge results. ICCVW, pages 2206-2241, 2019.","title":"The seventh visual object tracking vot2019 challenge results","authors":"Matej Kristan, Jiri Matas","year":"2019","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] Chenglong Li, Hui Cheng, Shiyi Hu, Xi- aobai Liu, Jin Tang, and Liang Lin. Learning collaborative sparse representation for grayscale-thermal tracking. TIP, 25:5743-5756, 2016.","title":"Learning collaborative sparse representation for grayscale-thermal tracking","authors":"Chenglong Li, Hui Cheng, Shiyi Hu, Xiaobai Liu, Jin Tang, Liang Lin","year":"2016","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, and Jin Tang. Rgb-t object tracking: Bench- mark and baseline. PR, 96:106977, 2019.","title":"Rgb-t object tracking: Benchmark and baseline","authors":"Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, Jin Tang","year":"2019","venue":"PR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Chenglong Li, Lei Liu, Andong Lu, Qing Ji, and Jin Tang. Challenge-aware rgbt tracking. In ECCV, pages 222-237. Springer, 2020.","title":"Challenge-aware rgbt tracking","authors":"Chenglong Li, Lei Liu, Andong Lu, Qing Ji, Jin Tang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2020a] Andong Lu, Chenglong Li, Y. Yan, Jin Tang, and Bin Luo. Rgbt tracking via multi-adapter net- work with hierarchical divergence loss. TIP, 30:5613- 5625, 2020.","title":"Rgbt tracking via multi-adapter network with hierarchical divergence loss","authors":"Lu","year":"2020","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2020b] Andong Lu, Chenglong Li, Y. Yan, Jin Tang, and Bin Luo. Rgbt tracking via multi-adapter net- work with hierarchical divergence loss. TIP, 30:5613- 5625, 2020.","title":"Rgbt tracking via multi-adapter network with hierarchical divergence loss","authors":"Lu","year":"2020","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2020c] Andong Lu, Cun Qian, Chenglong Li, Jin Tang, and Liang Wang. Duality-gated mutual condition network for rgbt tracking. TNNLS, PP, 2020.","title":"Duality-gated mutual condition network for rgbt tracking","authors":"Lu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mei et al., 2023] Jiatian Mei, Dongming Zhou, Jinde Cao, Rencan Nie, and Kangjian He. Differential reinforcement and global collaboration network for rgbt tracking. Sen- sors, 23:7301-7311, 2023.","title":"Differential reinforcement and global collaboration network for rgbt tracking","authors":"Mei","year":"2023","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Muller et al., 2018] Matthias Muller, Adel Bibi, Silvio Gi- ancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, pages 300-317, 2018.","title":"Trackingnet: A large-scale dataset and benchmark for object tracking in the wild","authors":"Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, Bernard Ghanem","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nam and Han, 2015] Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. CVPR, pages 4293-4302, 2015.","title":"Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking","authors":"Nam, Han","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tu et al., 2021] Zhengzheng Tu, Chun Lin, Wei Zhao, Chenglong Li, and Jin Tang. M5l: Multi-modal multi- margin metric learning for rgbt tracking. TIP, 31:85-98, 2021.","title":"M5l: Multi-modal multimargin metric learning for rgbt tracking","authors":"Tu","year":"2021","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Chaoqun Wang, Chunyan Xu, Zhen Cui, Lingli Zhou, Tong Zhang, Xiaoya Zhang, and Jian Yang. Cross-modal pattern-propagation for rgb-t tracking. CVPR, pages 7062-7071, 2020.","title":"Cross-modal pattern-propagation for rgb-t tracking","authors":"Chaoqun Wang, Chunyan Xu, Zhen Cui, Lingli Zhou, Tong Zhang, Xiaoya Zhang, Jian Yang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Xiao Wang, Xiu Shu, Shiliang Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Mfgnet: Dynamic modality-aware filter generation for rgb-t tracking. ArXiv, abs/2107.10433, 2021.","title":"Mfgnet: Dynamic modality-aware filter generation for rgb-t tracking","authors":"Xiao Wang, Xiu Shu, Shiliang Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, Feng Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiao et al., 2022] Yun Xiao, Mengmeng Yang, Chenglong Li, Lei Liu, and Jin Tang. Attribute-based progressive fu- sion network for rgbt tracking. In AAAI, 2022.","title":"Attribute-based progressive fusion network for rgbt tracking","authors":"Yun Xiao, Mengmeng Yang, Chenglong Li, Lei Liu, Jin Tang","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ye et al., 2022] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, pages 341-357. Springer, 2022.","title":"Joint feature learning and relation modeling for tracking: A one-stream framework","authors":"Ye","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Lichao Zhang, Martin Danelljan, Abel Gonzalez-Garcia, Joost van de Weijer, and Fahad Shahbaz Khan. Multi-modal fusion for end-to-end rgb-t tracking. ICCVW, pages 2252-2261, 2019.","title":"Multi-modal fusion for end-to-end rgb-t tracking","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020] Pengyu Zhang, Jie Zhao, Chunjuan Bo, Dong Wang, Huchuan Lu, and Xiaoyun Yang. Jointly modeling motion and appearance cues for robust rgb-t tracking. TIP, 30:3335-3347, 2020.","title":"Jointly modeling motion and appearance cues for robust rgb-t tracking","authors":"Pengyu Zhang, Jie Zhao, Chunjuan Bo, Dong Wang, Huchuan Lu, Xiaoyun Yang","year":"2020","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Pengyu Zhang, Dong Wang, Huchuan Lu, and Xiaoyun Yang. Learning adaptive attribute-driven representation for real-time rgb-t tracking. IJCV, 129:2714 -2729, 2021.","title":"Learning adaptive attribute-driven representation for real-time rgb-t tracking","authors":"Pengyu Zhang, Dong Wang, Huchuan Lu, Xiaoyun Yang","year":"2021","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2022a] Pengyu Zhang, Jie Zhao, D. Wang, Huchuan Lu, and Xiang Ruan. Visible-thermal uav track- ing: A large-scale benchmark and new baseline. CVPR, pages 8876-8885, 2022.","title":"Visible-thermal uav tracking: A large-scale benchmark and new baseline","authors":"Zhang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2022b] Tianlu Zhang, Xueru Liu, Qiang Zhang, and Jungong Han. Siamcda: Complementarity- and distractor-aware rgb-t tracking based on siamese net- work. TCSVT, 32:1403-1417, 2022.","title":"Siamcda: Complementarityand distractor-aware rgb-t tracking based on siamese network","authors":"Zhang","year":"2022","venue":"TCSVT","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2023] Tianlu Zhang, Hongyuan Guo, Qiang Jiao, Qiang Zhang, and Jungong Han. Efficient rgb-t track- ing via cross-modality distillation. In CVPR, pages 5404- 5413, 2023.","title":"Efficient rgb-t tracking via cross-modality distillation","authors":"Zhang","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2019] Yabin Zhu, Chenglong Li, Bin Luo, Jin Tang, and Xiao Wang. Dense feature aggregation and pruning for rgbt tracking. ACM MM, 2019.","title":"Dense feature aggregation and pruning for rgbt tracking","authors":"Yabin Zhu, Chenglong Li, Bin Luo, Jin Tang, Xiao Wang","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2022] Yabin Zhu, Chenglong Li, Jin Tang, Bin Luo, and Liang Wang. Rgbt tracking by trident fusion net- work. TCSVT, 32:579-592, 2022.","title":"Rgbt tracking by trident fusion network","authors":"Zhu","year":"2022","venue":"TCSVT","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2023] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal track- ing. In CVPR, pages 9516-9526, 2023.","title":"Visual prompt multi-modal tracking","authors":"Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, Huchuan Lu","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}