{"type":"ScholarlyArticle","title":"AQT: Adversarial Query Transformers for Domain Adaptive Object Detection","authors":[{"name":"Wei-Jie Huang","slug":"wei-jie-huang","given":"Wei-Jie","family":"Huang","affiliation":"National Yang Ming Chiao Tung University"},{"name":"Yu-Lin Lu","slug":"yu-lin-lu","given":"Yu-Lin","family":"Lu","affiliation":"National Yang Ming Chiao Tung University"},{"name":"Shih-Yao Lin","slug":"shih-yao-lin","given":"Shih-Yao","family":"Lin","affiliation":"Sony Corporation of America"},{"name":"Yusheng Xie","slug":"yusheng-xie","given":"Yusheng","family":"Xie","affiliation":""},{"name":"Yen-Yu Lin","slug":"yen-yu-lin","given":"Yen-Yu","family":"Lin","affiliation":"National Yang Ming Chiao Tung University; Academia Sinica"}],"abstract":"Adversarial feature alignment is widely used in domain adaptive object detection. Despite the effectiveness on CNN-based detectors, its applicability to transformer-based detectors is less studied. In this paper, we present AQT (adversarial query transformers) to integrate adversarial feature alignment into detection transformers. The generator is a detection transformer which yields a sequence of feature tokens, and the discriminator consists of a novel adversarial token and a stack of cross-attention layers. The cross-attention layers take the adversarial token as the query and the feature tokens from the generator as the key-value pairs. Through adversarial learning, the adversarial token in the discriminator attends to the domain-specific feature tokens, while the generator produces domain-invariant features, especially on the attended tokens, hence realizing adversarial feature alignment on transformers. Thorough experiments over several domain adaptive object detection benchmarks demonstrate that our approach performs favorably against the state-of-the-art methods. Source code is available at https://github.com/weii41392/AQT.","doi":"10.24963/ijcai.2022/136","url":"https://ijcai.org/proceedings/2022/136","pdf":"https://ijcai.org/proceedings/2022/0136.pdf","markdown":"https://ijcai.org/proceedings/2022/136.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/136","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":136,"pages":{"first":972,"last":979},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986974","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985117","length_sec":524,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cai et al., 2019] Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao. Exploring object relation in mean teacher for cross-domain detection. In CVPR, 2019.","title":"Exploring object relation in mean teacher for cross-domain detection","authors":"Cai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In CVPR, 2018.","title":"Domain adaptive faster r-cnn for object detection in the wild","authors":"Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, Luc Van Gool","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Chaoqi Chen, Zebiao Zheng, Xinghao Ding, Yue Huang, and Qi Dou. Harmonizing transferability and dis- criminability for adapting object detectors. In CVPR, 2020.","title":"Harmonizing transferability and discriminability for adapting object detectors","authors":"Chaoqi Chen, Zebiao Zheng, Xinghao Ding, Yue Huang, Qi Dou","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2021] Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan. Unbiased mean teacher for cross-domain object detection. In CVPR, 2021.","title":"Unbiased mean teacher for cross-domain object detection","authors":"Jinhong Deng, Wen Li, Yuhua Chen, Lixin Duan","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempit- sky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Bengio. Un- derstanding the difficulty of training deep feedforward neural net- works. In AISTATS, 2010.","title":"Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot, Xavier Glorot, Yoshua Bengio","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hoffman et al., 2018] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, 2018.","title":"CyCADA: Cycle-consistent adversarial domain adaptation","authors":"Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, Trevor Darrell","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hsu et al., 2020] Cheng-Chun Hsu, Yi-Hsuan Tsai, Yen-Yu Lin, and Ming-Hsuan Yang. Every pixel matters: Center-aware fea- ture alignment for domain adaptive object detector. In ECCV, 2020.","title":"Every pixel matters: Center-aware feature alignment for domain adaptive object detector","authors":"Cheng-Chun Hsu, Yi-Hsuan Tsai, Yen-Yu Lin, Ming-Hsuan Yang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson-Roberson et al., 2017] Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In ICRA, 2017.","title":"Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?","authors":"Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Nittur Sharath, Karl Sridhar, Ram Rosaen, Vasudevan","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2019] Seunghyeon Kim, Jaehoon Choi, Taekyung Kim, and Changick Kim. Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection. In ICCV, 2019.","title":"Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection","authors":"Kim","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in con- text. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaim- ing He, and Piotr Dollar. Focal loss for dense object detection. In ICCV, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexan- der C. Berg. Ssd: Single shot multibox detector. In ECCV, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Munir et al., 2021] Muhammad Akhtar Munir, Muhammad Haris Khan, M. Sarfraz, and Mohsen Ali. Ssal: Synergizing between self-training and adversarial learning for domain adaptive object detection. In NeurIPS, 2021.","title":"Ssal: Synergizing between self-training and adversarial learning for domain adaptive object detection","authors":"Munir","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rezaeianaran et al., 2021] Farzaneh Rezaeianaran, Rakshith Shetty, Rahaf Aljundi, Daniel Olmeda Reino, Shanshan Zhang, and Bernt Schiele. Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection. In ICCV, 2021.","title":"Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection","authors":"Rakshith Farzaneh Rezaeianaran, Rahaf Shetty, Daniel Aljundi, Shanshan Olmeda Reino, Bernt Zhang, Schiele","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saito et al., 2019] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution alignment for adaptive object detection. In CVPR, 2019.","title":"Strong-weak distribution alignment for adaptive object detection","authors":"Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, Kate Saenko","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sakaridis et al., 2018] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data. IJCV, 2018.","title":"Semantic foggy scene understanding with synthetic data","authors":"Christos Sakaridis, Dengxin Dai, Luc Van Gool","year":"2018","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian et al., 2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, 2019.","title":"Fcos: Fully convolutional one-stage object detection","authors":"Chunhua Zhi Tian, Hao Shen, Tong Chen, He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tian et al., 2021] Kun Tian, Chenghao Zhang, Ying Wang, Shim- ing Xiang, and Chunhong Pan. Knowledge mining and transfer- ring for domain adaptive object detection. In ICCV, 2021.","title":"Knowledge mining and transferring for domain adaptive object detection","authors":"Chenghao Kun Tian, Ying Zhang, Shiming Wang, Chunhong Xiang, Pan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[VS et al., 2021] Vibashan VS, Vikram Gupta, Poojan Oza, Vish- wanath A. Sindagi, and Vishal M. Patel. Mega-cda: Memory guided attention for category-aware unsupervised domain adap- tive object detection. In CVPR, 2021.","title":"Mega-cda: Memory guided attention for category-aware unsupervised domain adaptive object detection","authors":"V S Vibashan, Vikram Gupta, Poojan Oza, A Vishwanath, Vishal M Sindagi, Patel","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021a] Wen Wang, Yang Cao, Jing Zhang, Fengxi- ang He, Zheng-Jun Zha, Yonggang Wen, and Dacheng Tao. Ex- ploring sequence feature alignment for domain adaptive detection transformers. In ACM MM, 2021.","title":"Exploring sequence feature alignment for domain adaptive detection transformers","authors":"Wang","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021b] Yu Wang, Rui Zhang, Shuo Zhang, Miao Li, Yangyang Xia, Xishan Zhang, and Shaoli Liu. Domain-specific suppression for adaptive object detection. In CVPR, 2021.","title":"Domain-specific suppression for adaptive object detection","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2021] Aming Wu, Rui Liu, Yahong Han, Linchao Zhu, and Yi Yang. Vector-decomposed disentanglement for domain- invariant object detection. In ICCV, 2021.","title":"Vector-decomposed disentanglement for domaininvariant object detection","authors":"Aming Wu, Rui Liu, Yahong Han, Linchao Zhu, Yi Yang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2020a] Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, and Xiu-Shen Wei. Exploring categorical regularization for domain adaptive object detection. In CVPR, 2020.","title":"Exploring categorical regularization for domain adaptive object detection","authors":"Xu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2020b] Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang. Cross-domain detection via graph-induced prototype alignment. In CVPR, 2020.","title":"Cross-domain detection via graph-induced prototype alignment","authors":"Xu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yu et al., 2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.","title":"Bdd100k: A diverse driving dataset for heterogeneous multitask learning","authors":"Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, Trevor Darrell","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Yixin Zhang, Zilei Wang, and Yushi Mao. Rpn prototype alignment for domain adaptive object detector. In CVPR, 2021.","title":"Rpn prototype alignment for domain adaptive object detector","authors":"Yixin Zhang, Zilei Wang, Yushi Mao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xi- aogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}