{"type":"ScholarlyArticle","title":"MLP-DINO: Category Modeling and Query Graphing with Deep MLP for Object Detection","authors":[{"name":"Guiping Cao","slug":"guiping-cao","given":"Guiping","family":"Cao","affiliation":"Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen , China; Peng Cheng Laboratory , Shenzhen , China"},{"name":"Wenjian Huang","slug":"wenjian-huang","given":"Wenjian","family":"Huang","affiliation":"Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen , China"},{"name":"Xiangyuan Lan","slug":"xiangyuan-lan","given":"Xiangyuan","family":"Lan","affiliation":"Peng Cheng Laboratory , Shenzhen , China"},{"name":"Jianguo Zhang","slug":"jianguo-zhang","given":"Jianguo","family":"Zhang","affiliation":"Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen , China; Peng Cheng Laboratory , Shenzhen , China"},{"name":"Dongmei Jiang","slug":"dongmei-jiang","given":"Dongmei","family":"Jiang","affiliation":"Peng Cheng Laboratory , Shenzhen , China"},{"name":"Yaowei Wang","slug":"yaowei-wang","given":"Yaowei","family":"Wang","affiliation":"Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"Popular transformer-based detectors detect objects in a one-to-one manner, where both the bounding box and category of each object are predicted only by the single query, leading to the box-sensitive category predictions. Additionally, the initialization of positional queries solely based on the predicted confidence scores or learnable embeddings neglects the significant spatial interrelation between different queries. This oversight leads to an imbalanced spatial distribution of queries (SDQ). In this paper, we propose a new MLP-DINO model to address these issues. Firstly, we present a new Query-Independent Category Supervision (QICS) approach for modeling categories information, decoupling the sensitive bounding box prediction process to improve the detection performance. Additionally, to further improve the category predictions, we introduce a deep MLP model into transformer-based detection framework to capture the long-range and short-range information simultaneously. Thirdly, to balance the SDQ, we design a novel Graph-based Query Selection (GQS) method that distributes each query point in a discrete manner by graphing the spatial information of queries to cover a broader range of potential objects, significantly enhancing the hit-rate of queries. Experimental results on COCO indicate that our MLP-DINO achieves 54.6% AP with only 44M parame\tters under 36-epoch setting, greatly outperforming the original DINO by +3.7% AP with fewer parameters and FLOPs. The source codes will be available at https://github.com/Med-Process/MLP-DINO.","doi":"10.24963/ijcai.2024/67","url":"https://ijcai.org/proceedings/2024/67","pdf":"https://ijcai.org/proceedings/2024/0067.pdf","markdown":"https://ijcai.org/proceedings/2024/67.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/67","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":67,"pages":{"first":605,"last":613},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Scene analysis and understanding","Machine Learning: ML: Multi-label learning"],"videos":[],"references":[{"position":1,"raw":"[Abdi, 2010] Hervé Abdi. Coefficient of variation. Encyclo- pedia of research design, 1(5), 2010.","title":"Coefficient of variation","authors":"Hervé Abdi","year":"2010","venue":"Encyclopedia of research design","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai and Vasconcelos, 2018] Zhaowei Cai and Nuno Vas- concelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 6154-6162, 2018.","title":"Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection","authors":"Vasconcelos Cai","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2023] Guiping Cao, Shengda Luo, Wenjian Huang, Xiangyuan Lan, Dongmei Jiang, Yaowei Wang, and Jianguo Zhang. Strip-mlp: Efficient token interaction for vision mlp. arXiv preprint arXiv:2307.11458, 2023.","title":"Strip-mlp: Efficient token interaction for vision mlp","authors":"Guiping Cao, Shengda Luo, Wenjian Huang, Xiangyuan Lan, Dongmei Jiang, Yaowei Wang, Jianguo Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2107.10224, 2021.","title":"Cyclemlp: A mlp-like architecture for dense prediction","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2019] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 6569-6578, 2019.","title":"Centernet: Keypoint triplets for object detection","authors":"Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, Qi Tian","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ge et al., 2021] Zheng Ge, Songtao Liu, Feng Wang, Zem- ing Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.","title":"Yolox: Exceeding yolo series in 2021","authors":"Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, Jian Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2022] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mlp via hierarchical rearrange- ment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 826-836, 2022.","title":"Hire-mlp: Vision mlp via hierarchical rearrangement","authors":"Guo","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2023] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19702-19712, 2023.","title":"Detrs with hybrid matching","authors":"Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, Han Hu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2022] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13619-13627, 2022.","title":"Dn-detr: Accelerate detr training by introducing query denoising","authors":"Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, Lei Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2013] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400, 2013.","title":"Network in network","authors":"Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2020] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense ob- ject detection. IEEE Transactions on Pattern Analysis & Machine Intelligence, 42(02):318-327, 2020.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar","year":"2020","venue":"IEEE Transactions on Pattern Analysis & Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 21-37. Springer, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021a] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204-9215, 2021.","title":"Pay attention to mlps","authors":"Liu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022a] Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, and Hai-Tao Zheng. Are we ready for a new paradigm shift? a survey on visual deep mlp. Patterns, 3(7), 2022.","title":"Are we ready for a new paradigm shift? a survey on visual deep mlp","authors":"Liu","year":"2022","venue":"Patterns","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2022b] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329, 2022.","title":"Dab-detr: Dynamic anchor boxes are better queries for detr","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.","title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","authors":"Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Meng et al., 2021] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660, 2021.","title":"Conditional detr for fast training convergence","authors":"Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ren et al., 2023] Tianhe Ren, Shilong Liu, Feng Li, Hao Zhang, Ailing Zeng, Jie Yang, Xingyu Liao, Ding Jia, Hongyang Li, He Cao, et al. detrex: Benchmarking detec- tion transformers. arXiv preprint arXiv:2306.07265, 2023.","title":"detrex: Benchmarking detection transformers","authors":"Ren","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2022a] Chuanxin Tang, Yucheng Zhao, Guangt- ing Wang, Chong Luo, Wenxuan Xie, and Wenjun Zeng. Sparse mlp for image recognition: Is self-attention really necessary? In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 36, pages 2344-2351, 2022.","title":"Sparse mlp for image recognition: Is self-attention really necessary","authors":"Tang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tang et al., 2022b] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An im- age patch is a wave: Phase-aware vision mlp. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10935-10944, 2022.","title":"An image patch is a wave: Phase-aware vision mlp","authors":"Tang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tolstikhin et al., 2021] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural infor- mation processing systems, 34:24261-24272, 2021.","title":"Mlp-mixer: An all-mlp architecture for vision","authors":"Neil Ilya O Tolstikhin, Alexander Houlsby, Lucas Kolesnikov, Xiaohua Beyer, Thomas Zhai, Jessica Unterthiner, Andreas Yung, Daniel Steiner, Jakob Keysers, Uszkoreit","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2023] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object de- tectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7464- 7475, 2023.","title":"Yolov7: Trainable bag-offreebies sets new state-of-the-art for real-time object detectors","authors":"Chien-Yao Wang, Alexey Bochkovskiy, Hong-Yuan Mark Liao","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yao et al., 2021] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object de- tector with dense prior. arXiv preprint arXiv:2104.01318, 2021.","title":"Efficient detr: improving end-to-end object detector with dense prior","authors":"Yao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2022] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising an- chor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022.","title":"Dino: Detr with improved denoising anchor boxes for end-to-end object detection","authors":"Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, Heung-Yeung Shum","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2023] Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less is more: Focus at- tention for efficient detr. arXiv preprint arXiv:2307.12612, 2023.","title":"Less is more: Focus attention for efficient detr","authors":"Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, Yunhe Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zong et al., 2023] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments train- ing. In Proceedings of the IEEE/CVF international con- ference on computer vision, pages 6748-6758, 2023.","title":"Detrs with collaborative hybrid assignments training","authors":"Zong","year":"2023","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}