{"type":"ScholarlyArticle","title":"Overcoming Language Priors with Self-supervised Learning for Visual Question Answering","authors":[{"name":"Xi Zhu","slug":"xi-zhu","given":"Xi","family":"Zhu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zhendong Mao","slug":"zhendong-mao","given":"Zhendong","family":"Mao","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Chunxiao Liu","slug":"chunxiao-liu","given":"Chunxiao","family":"Liu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Peng Zhang","slug":"peng-zhang","given":"Peng","family":"Zhang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China"},{"name":"Bin Wang","slug":"bin-wang","given":"Bin","family":"Wang","affiliation":"Xiaomi AI Lab , Xiaomi Inc. , Beijing , China"},{"name":"Yongdong Zhang","slug":"yongdong-zhang","given":"Yongdong","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"}],"abstract":"Most Visual Question Answering (VQA) models suffer from the language prior problem, which is caused by inherent data biases. Specifically, VQA models tend to answer questions (e.g., what color is the banana?) based on the high-frequency answers (e.g., yellow) ignoring image contents. Existing approaches tackle this problem by creating delicate models or introducing additional visual annotations to reduce question dependency and strengthen image dependency. However, they are still subject to the language prior problem since the data biases have not been fundamentally addressed. In this paper, we introduce a self-supervised learning framework to solve this problem. Concretely, we first automatically generate labeled data to balance the biased data, and then propose a self-supervised auxiliary task to utilize the balanced data to assist the VQA model to overcome language priors. Our method can compensate for the data biases by generating balanced data without introducing external annotations. Experimental results show that our method achieves state-of-the-art performance, improving the overall accuracy from 49.50% to 57.59% on the most commonly used benchmark VQA-CP v2. In other words, we can increase the performance of annotation-based methods by 16% without using external annotations. Our code is available on GitHub.","doi":"10.24963/ijcai.2020/151","url":"https://ijcai.org/proceedings/2020/151","pdf":"https://ijcai.org/proceedings/2020/0151.pdf","markdown":"https://ijcai.org/proceedings/2020/151.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/151","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":151,"pages":{"first":1083,"last":1089},"keywords":["Computer Vision: Language and Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24898/file_24898.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Agrawal et al., 2018] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971- 4980, 2018.","title":"Don't just assume; look and answer: Overcoming priors for visual question answering","authors":"Agrawal","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 39-48, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cadene et al., 2019] Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing uni- modal biases for visual question answering. In Advances in Neural Information Processing Systems, pages 839- 850, 2019.","title":"Rubi: Reducing unimodal biases for visual question answering","authors":"Cadene","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2016] Alexis Conneau, Holger Schwenk, Loıc Barrault, and Yann Lecun. Very deep convolutional networks for natural language processing. arXiv preprint arXiv:1606.01781, 2, 2016.","title":"Very deep convolutional networks for natural language processing","authors":"Alexis Conneau, Holger Schwenk, Loıc Barrault, Yann Lecun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learn- ing by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2019] Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yibing Liu, Yinglong Wang, and Mohan Kankanhalli. Quantifying and alleviating the language prior problem in visual question answering. In Proceedings of the 42nd In- ternational ACM SIGIR Conference on Research and De- velopment in Information Retrieval, pages 75-84, 2019.","title":"Quantifying and alleviating the language prior problem in visual question answering","authors":"Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yibing Liu, Yinglong Wang, Mohan Kankanhalli","year":"2019","venue":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jing et al., 2020] Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, and Qi Wu. Overcoming language priors in vqa via decomposed linguistic representations. 2020.","title":"Overcoming language priors in vqa via decomposed linguistic representations","authors":"Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, Qi Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2018] Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, pages 1564- 1574, 2018.","title":"Bilinear attention networks","authors":"Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] An-An Liu, Yu-Ting Su, Wei-Zhi Nie, and Mohan Kankanhalli. Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence, 39(1):102-114, 2016.","title":"Hierarchical clustering multi-task learning for joint human action grouping and recognition","authors":"Liu","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018] Anan Liu, Ning Xu, Hanwang Zhang, Weizhi Nie, Yuting Su, and Yongdong Zhang. Multi-level policy and reward reinforcement learning for image cap- tioning. In Proceedings of the International Joint Confer- ence on Artificial Intelligence, pages 821-827, 2018.","title":"Multi-level policy and reward reinforcement learning for image captioning","authors":"Anan Liu, Ning Xu, Hanwang Zhang, Weizhi Nie, Yuting Su, Yongdong Zhang","year":"2018","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/114"},{"position":14,"raw":"[Mao et al., 2019] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584, 2019.","title":"The neurosymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision","authors":"Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, Jiajun Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Parkhi et al., 2015] Omkar M Parkhi, Andrea Vedaldi, An- drew Zisserman, et al. Deep face recognition. In British Machine Vision Association, page 6, 2015.","title":"Deep face recognition","authors":"Andrea Omkar M Parkhi, Andrew Vedaldi, Zisserman","year":"2015","venue":"British Machine Vision Association","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ramakrishnan et al., 2018] Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. Overcoming lan- guage priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, pages 1541-1551, 2018.","title":"Overcoming language priors in visual question answering with adversarial regularization","authors":"Ramakrishnan","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Selvaraju et al., 2019] Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking a hint: Leverag- ing explanations to make vision and language models more grounded. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 2591-2600, 2019.","title":"Taking a hint: Leveraging explanations to make vision and language models more grounded","authors":"Stefan Ramprasaath R Selvaraju, Yilin Lee, Hongxia Shen, Shalini Jin, Larry Ghosh, Dhruv Heck, Devi Batra, Parikh","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu and Mooney, 2019] Jialin Wu and Raymond Mooney. Self-critical reasoning for robust visual question answer- ing. In Advances in Neural Information Processing Sys- tems, pages 8601-8611, 2019.","title":"Self-critical reasoning for robust visual question answering","authors":"Jialin Wu, Raymond Mooney","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 4622-4630, 2016.","title":"Ask me anything: Free-form visual question answering based on knowledge from external sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 21-29, 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yi et al., 2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Informa- tion Processing Systems, pages 1031-1042, 2018.","title":"Neuralsymbolic vqa: Disentangling reasoning from vision and language understanding","authors":"Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, Josh Tenenbaum","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}