{"type":"ScholarlyArticle","title":"Densely Connected Attention Flow for Visual Question Answering","authors":[{"name":"Fei Liu","slug":"fei-liu","given":"Fei","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Jing Liu","slug":"jing-liu","given":"Jing","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"},{"name":"Zhiwei Fang","slug":"zhiwei-fang","given":"Zhiwei","family":"Fang","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Richang Hong","slug":"richang-hong","given":"Richang","family":"Hong","affiliation":"School of Computer and Information , Hefei University of Technology"},{"name":"Hanqing Lu","slug":"hanqing-lu","given":"Hanqing","family":"Lu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"}],"abstract":"Learning effective interactions between multi-modal features is at the heart of visual question answering (VQA). A common defect of the existing VQA approaches is that they only consider a very limited amount of interactions, which may be not enough to model latent complex image-question relations that are necessary for accurately answering questions. Therefore, in this paper, we propose a novel DCAF (Densely Connected Attention Flow) framework for modeling dense interactions. It densely connects all pairwise layers of the network via Attention Connectors, capturing fine-grained interplay between image and question across all hierarchical levels. The proposed Attention Connector efficiently connects the multi-modal features at any two layers with symmetric co-attention, and produces interaction-aware attention features. Experimental results on three publicly available datasets show that the proposed method achieves state-of-the-art performance.","doi":"10.24963/ijcai.2019/122","url":"https://ijcai.org/proceedings/2019/122","pdf":"https://ijcai.org/proceedings/2019/0122.pdf","markdown":"https://ijcai.org/proceedings/2019/122.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/122","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":122,"pages":{"first":869,"last":875},"keywords":["Computer Vision: Computer Vision","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2018] Yalong Bai, Jianlong Fu, Tiejun Zhao, and Tao Mei. Deep attention neural tensor network for visual question answering. In ECCV, 2018.","title":"Deep attention neural tensor network for visual question answering","authors":"Yalong Bai, Jianlong Fu, Tiejun Zhao, Tao Mei","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ben-Younes et al., 2017] Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Interna- tional Conference on Computer Vision (ICCV), 2017.","title":"Mutan: Multimodal tucker fusion for visual question answering","authors":"Ben-Younes","year":"2017","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2015] Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. Abc- cnn: An attention based convolutional neural network for visual question answering. arXiv:1511.05960, 2015.","title":"Abccnn: An attention based convolutional neural network for visual question answering","authors":"Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, Ram Nevatia","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. In EMNLP, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2018] Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven CH Hoi, and Xiaogang Wang. Question-guided hybrid convolution for visual question answering. In European Conference on Computer Vision (ECCV), 2018.","title":"Question-guided hybrid convolution for visual question answering","authors":"Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven Ch Hoi, Xiaogang Wang","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image under- standing in visual question answering. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Making the v in vqa matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"International Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"International Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kafle and Kanan, 2017] Kushal Kafle and Christopher Kanan. An analysis of visual question answering algo- rithms. In International Conference on Computer Vision (ICCV), 2017.","title":"An analysis of visual question answering algorithms","authors":"Kushal Kafle, Christopher Kanan","year":"2017","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2016] Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for vi- sual qa. In NeurIPS, 2016.","title":"Multimodal residual learning for visual qa","authors":"Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2017] Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. In International Conference on Learning Representations (ICLR), 2017.","title":"Hadamard product for low-rank bilinear pooling","authors":"Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang","year":"2017","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2018] Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Interna- tional Conference on Neural Information Processing Sys- tems (NeurIPS), 2018.","title":"Bilinear attention networks","authors":"Kim","year":"2018","venue":"International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In International Conference on Neural Information Processing Systems (NeurIPS), 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"International Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen and Okatani, 2018] Duy-Kien Nguyen and Takayuki Okatani. Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering. In CVPR, 2018.","title":"Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering","authors":"Duy-Kien Nguyen, Takayuki Okatani","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Reddi et al., 2018] Sashank J Reddi, Satyen Kale, and San- jiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR), 2018.","title":"On the convergence of adam and beyond","authors":"J Sashank, Satyen Reddi, Sanjiv Kale, Kumar","year":"2018","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In In- ternational Conference on Neural Information Processing Systems (NeurIPS), 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"ternational Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2018] Yang Shi, Tommaso Furlanello, Sheng Zha, and Animashree Anandkumar. Question type guided attention in visual question answering. In European Con- ference on Computer Vision (ECCV), 2018.","title":"Question type guided attention in visual question answering","authors":"Yang Shi, Tommaso Furlanello, Sheng Zha, Animashree Anandkumar","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Su et al., 2018] Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, and Jianguo Li. Learning visual knowledge memory networks for visual question answer- ing. In CVPR, 2018.","title":"Learning visual knowledge memory networks for visual question answering","authors":"Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, Jianguo Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2018] Chenfei Wu, Jinlai Liu, Xiaojie Wang, and Xuan Dong. Chain of reasoning for visual question an- swering. In NeurIPS, 2018.","title":"Chain of reasoning for visual question answering","authors":"Chenfei Wu, Jinlai Liu, Xiaojie Wang, Xuan Dong","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu and Saenko, 2016] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial at- tention for visual question answering. In European Con- ference on Computer Vision (ECCV), 2016.","title":"Ask, attend and answer: Exploring question-guided spatial attention for visual question answering","authors":"Saenko Xu, Huijuan Xu, Kate Saenko","year":"2016","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In CVPR, 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In International Conference on Computer Vision, 2017.","title":"Multi-modal factorized bilinear pooling with co-attention learning for visual question answering","authors":"Zhou Yu, Jun Yu, Jianping Fan, Dacheng Tao","year":"2017","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2018] Zhou Yu, Jun Yu, Chenchao Xiang, Jian- ping Fan, and Dacheng Tao. Beyond bilinear: Generalized multi-modal factorized high-order pooling for visual ques- tion answering. IEEE Transactions on Neural Networks and Learning Systems, 2018.","title":"Beyond bilinear: Generalized multi-modal factorized high-order pooling for visual question answering","authors":"Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, Dacheng Tao","year":"2018","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Learning to count objects in natural im- ages for visual question answering. In ICLR, 2018.","title":"Learning to count objects in natural images for visual question answering","authors":"Yan Zhang, Jonathon Hare, Adam Prügel-Bennett","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}