{"type":"ScholarlyArticle","title":"Regression Residual Reasoning with Pseudo-labeled Contrastive Learning for Uncovering Multiple Complex Compositional Relations","authors":[{"name":"Chengtai Li","slug":"chengtai-li","given":"Chengtai","family":"Li","affiliation":"Digital Port Technologies Lab , School of Computer Science , University of Nottingham Ningbo China; Cixi Institute of Biomedical Engineering , Ningbo Institute of Industrial Technology , Chinese Academy of Sciences"},{"name":"Yuting He","slug":"yuting-he","given":"Yuting","family":"He","affiliation":"Digital Port Technologies Lab , School of Computer Science , University of Nottingham Ningbo China; Shenzhen Institute of Advanced Technology , Chinese Academy of Sciences"},{"name":"Jianfeng Ren","slug":"jianfeng-ren","given":"Jianfeng","family":"Ren","affiliation":"Digital Port Technologies Lab , School of Computer Science , University of Nottingham Ningbo China; Nottingham Ningbo China Beacons of Excellence Research and Innovation Institute ,"},{"name":"Ruibin Bai","slug":"ruibin-bai","given":"Ruibin","family":"Bai","affiliation":"Digital Port Technologies Lab , School of Computer Science , University of Nottingham Ningbo China; Nottingham Ningbo China Beacons of Excellence Research and Innovation Institute ,"},{"name":"Yitian Zhao","slug":"yitian-zhao","given":"Yitian","family":"Zhao","affiliation":"Cixi Institute of Biomedical Engineering , Ningbo Institute of Industrial Technology , Chinese Academy of Sciences"},{"name":"Heng Yu","slug":"heng-yu","given":"Heng","family":"Yu","affiliation":"Digital Port Technologies Lab , School of Computer Science , University of Nottingham Ningbo China; University of Nottingham Ningbo China"},{"name":"Xudong Jiang","slug":"xudong-jiang","given":"Xudong","family":"Jiang","affiliation":"School of Electrical & Electronic Engineering , Nanyang Technological University"}],"abstract":"Abstract Visual Reasoning (AVR) has been widely studied in literature. Our study reveals that AVR models tend to rely on appearance matching rather than a genuine understanding of underlying rules. We hence develop a challenging benchmark, Multiple Complex Compositional Reasoning (MC2R), composed of diverse compositional rules on attributes with intentionally increased variations. It aims to identify two outliers from five given images, in contrast to single-answer questions in previous AVR tasks. To solve MC2R tasks, a Regression Residual Reasoning with Pseudo-labeled Contrastive Learning (R3PCL) is proposed, which first transforms the original problem by selecting three images following the same rule, and iteratively regresses one normal image by using the other two, allowing the model to gradually comprehend the underlying rules. The proposed PCL leverages a set of min-max operations to generate more reliable pseudo labels, and exploits contrastive learning with data augmentation on pseudo-labeled images to boost the discrimination and generalization of features. Experimental results on two AVR datasets show that the proposed R3PCL significantly outperforms state-of-the-art models.","doi":"10.24963/ijcai.2024/384","url":"https://ijcai.org/proceedings/2024/384","pdf":"https://ijcai.org/proceedings/2024/0384.pdf","markdown":"https://ijcai.org/proceedings/2024/384.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/384","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":384,"pages":{"first":3466,"last":3474},"keywords":["Knowledge Representation and Reasoning: KRR: Learning and reasoning"],"videos":[],"references":[{"position":1,"raw":"[Barrett et al., 2018] David Barrett, Felix Hill, Adam San- toro, Ari Morcos, and Timothy Lillicrap. Measuring ab- stract reasoning in neural networks. In International Con- ference on Machine Learning (ICML), pages 511-520.","title":"Measuring abstract reasoning in neural networks","authors":"David Barrett, Felix Hill, Adam Santoro, Ari Morcos, Timothy Lillicrap","year":null,"venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benny et al., 2021] Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12557-12565, 2021.","title":"Scale-localized abstract reasoning","authors":"Yaniv Benny, Niv Pekar, Lior Wolf","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter- national Conference on Machine Learning (ICML), pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] X Chen, S Xie, and K He. An empir- ical study of training self-supervised vision transformer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9620-9629, 2021.","title":"An empirical study of training self-supervised vision transformer","authors":"S Chen, Xie, He","year":"2021","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chollet, 2019] Franc ¸ois Chollet. On the measure of intelli- gence. arXiv preprint arXiv:1911.01547, 2019.","title":"Franc ¸ois Chollet. On the measure of intelligence","authors":"Chollet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chuang et al., 2022] Ching-Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learn- ing against noisy views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16670-16681, 2022.","title":"Robust contrastive learning against noisy views","authors":"Ching-Yao Chuang, Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, Yale Song","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. Ieee, 2009.","title":"Imagenet: A largescale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ding et al., 2022] Ruxin Ding, Jianfeng Ren, Heng Yu, and Jiawei Li. Dynamic texture recognition using PDV hash- ing and dictionary learning on multi-scale volume local bi- nary pattern. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1840-1844. IEEE, 2022.","title":"Dynamic texture recognition using PDV hashing and dictionary learning on multi-scale volume local binary pattern","authors":"Ruxin Ding, Jianfeng Ren, Heng Yu, Jiawei Li","year":"2022","venue":"ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations (ICLR), 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 770- 778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2023] Wentao He, Jialu Zhang, Jianfeng Ren, Ruibin Bai, and Xudong Jiang. Hierarchical convit with attention-based relational reasoner for visual analogical reasoning. In Proceedings of the AAAI Conference on Arti- ficial Intelligence (AAAI), volume 37, pages 22-30, 2023.","title":"Hierarchical convit with attention-based relational reasoner for visual analogical reasoning","authors":"Wentao He, Jialu Zhang, Jianfeng Ren, Ruibin Bai, Xudong Jiang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hofstadter, 2001] Douglas R Hofstadter. Analogy as the core of cognition. The analogical mind: Perspectives from cognitive science, pages 499-538, 2001.","title":"Analogy as the core of cognition","authors":"Douglas R Hofstadter","year":"2001","venue":"The analogical mind: Perspectives from cognitive science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hu et al., 2021] Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 1567-1574, 2021.","title":"Stratified rule-aware network for abstract visual reasoning","authors":"Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, Shihao Bai","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2022] Zhanghao Jiang, Ke Xu, Heshan Du, Huan Jin, Zheng Lu, and Qian Zhang. Occlusion-invariant representation alignment for entity re-identification. In 2022 IEEE International Conference on Image Processing (ICIP), pages 3266-3270. IEEE, 2022.","title":"Occlusion-invariant representation alignment for entity re-identification","authors":"Zhanghao Jiang, Ke Xu, Heshan Du, Huan Jin, Zheng Lu, Qian Zhang","year":"2022","venue":"2022 IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zit- nick, and Ross Girshick. CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 1988-1997. IEEE, 2017.","title":"CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. Advances in neural information process- ing systems (NeurIPS), 33:18661-18673, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lake and Baroni, 2018] Brenden Lake and Marco Baroni. Generalization without systematicity: On the compo- sitional skills of sequence-to-sequence recurrent net- works. In International Conference on Machine Learning (ICML), pages 2873-2882. PMLR, 2018.","title":"Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks","authors":"Baroni Lake, Brenden Lake, Marco Baroni","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2022] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vi- sion transformers for classification and detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4804-4814, 2022.","title":"Mvitv2: Improved multiscale vision transformers for classification and detection","authors":"Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Małkiński and Mańdziuk, 2023] Mikołaj Małkiński and Jacek Mańdziuk. A review of emerging research direc- tions in abstract visual reasoning. Information Fusion, 91:713-736, 2023.","title":"Mikołaj Małkiński and Jacek Mańdziuk. A review of emerging research directions in abstract visual reasoning","authors":"Mańdziuk Małkiński","year":"2023","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":null,"venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Robinson et al., 2020] Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Con- ference on Learning Representations (ICLR), 2020.","title":"Contrastive learning with hard negative samples","authors":"Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie Jegelka","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Saxton et al., 2018] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathemati- cal reasoning abilities of neural models. In International Conference on Learning Representations (ICLR), 2018.","title":"Analysing mathematical reasoning abilities of neural models","authors":"David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli","year":"2018","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Son, 2022] Jeany Son. Contrastive learning for space-time correspondence via self-cycle consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 14679-14688, 2022.","title":"Contrastive learning for space-time correspondence via self-cycle consistency","authors":"Son ;, Jeany Son","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2023] Xingke Song, Jiahuan Jin, Chenglin Yao, Shihe Wang, Jianfeng Ren, and Ruibin Bai. Siamese- discriminant deep reinforcement learning for solving jig- saw puzzles with large eroded gaps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2303-2311, 2023.","title":"Siamesediscriminant deep reinforcement learning for solving jigsaw puzzles with large eroded gaps","authors":"Xingke Song, Jiahuan Jin, Chenglin Yao, Shihe Wang, Jianfeng Ren, Ruibin Bai","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thrush et al., 2022] Tristan Thrush, Ryan Jiang, Max Bar- tolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5238-5248, 2022.","title":"Winoground: Probing vision and language models for visio-linguistic compositionality","authors":"Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, Candace Ross","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2020] Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Dis- covering objects, attributes, relationships in analogical rea- soning. arXiv preprint arXiv:2007.04212, 2020.","title":"The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2023] Jingyi Xu, Tushar Vaidya, Yufei Wu, Saket Chandra, Zhangsheng Lai, and Kai Fong Ernest Chong. Abstract visual reasoning: An algebraic approach for solv- ing raven's progressive matrices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6715-6724, 2023.","title":"Abstract visual reasoning: An algebraic approach for solving raven's progressive matrices","authors":"Jingyi Xu, Tushar Vaidya, Yufei Wu, Saket Chandra, Zhangsheng Lai, Kai Fong, Ernest Chong","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2023] Lingxiao Yang, Hongzhi You, Zonglei Zhen, Dahui Wang, Xiaohong Wan, Xiaohua Xie, and Ru- Yuan Zhang. Neural prediction errors enable analogical visual reasoning in human standard intelligence tests. In International Conference on Machine Learning (ICML). PMLR, 2023.","title":"Neural prediction errors enable analogical visual reasoning in human standard intelligence tests","authors":"Yang","year":"2023","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zerroug et al., 2022] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in Neural Information Processing Systems (NeurIPS), 35:29776-29788, 2022.","title":"A benchmark for compositional visual reasoning","authors":"Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, Thomas Serre","year":"2022","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2019] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5317-5327, 2019.","title":"Raven: A dataset for relational and analogical visual reasoning","authors":"Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2023] Jialu Zhang, Jianfeng Ren, Qian Zhang, Jiang Liu, and Xudong Jiang. Spatial context-aware object-attentional network for multi-label image classifi- cation. IEEE Transactions on Image Processing, 2023.","title":"Spatial context-aware object-attentional network for multi-label image classification","authors":"Zhang","year":"2023","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2024] Jialu Zhang, Xiaoying Yang, Wentao He, Jianfeng Ren, Qian Zhang, Yitian Zhao, Ruibin Bai, Xiangjian He, and Jiang Liu. Scale optimization using evolutionary reinforcement learning for object detection on drone imagery. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pages 410-418, 2024.","title":"Scale optimization using evolutionary reinforcement learning for object detection on drone imagery","authors":"Zhang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}