{"type":"ScholarlyArticle","title":"From Pixels to Objects: Cubic Visual Attention for Visual Question Answering","authors":[{"name":"Jingkuan Song","slug":"jingkuan-song","given":"Jingkuan","family":"Song","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China,"},{"name":"Pengpeng Zeng","slug":"pengpeng-zeng","given":"Pengpeng","family":"Zeng","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China,"},{"name":"Lianli Gao","slug":"lianli-gao","given":"Lianli","family":"Gao","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China,"},{"name":"Heng Tao Shen","slug":"heng-tao-shen","given":"Heng Tao","family":"Shen","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu 611731 , China,"}],"abstract":"Recently, attention-based Visual Question Answering (VQA) has achieved great success by utilizing question to selectively target different visual areas that are related to the answer. Existing visual attention models are generally planar, i.e., different channels of the last conv-layer feature map of an image share the same weight. This conflicts with the attention mechanism because CNN features are naturally spatial and channel-wise. Also, visual attention models are usually conducted on pixel-level, which may cause region discontinuous problem. In this paper we propose a Cubic Visual Attention (CVA) model by successfully applying a novel channel and spatial attention on object regions to improve VQA task. Specifically, instead of attending to pixels, we first take advantage of the object proposal networks to generate a set of object candidates and extract their associated conv features. Then, we utilize the question to guide channel attention and spatial attention calculation based on the con-layer feature map. Finally, the attended visual features and the question are combined to infer the answer. We assess the performance of our proposed CVA on three public image QA datasets, including COCO-QA, VQA and Visual7W. Experimental results show that our proposed method significantly outperforms the state-of-the-arts.","doi":"10.24963/ijcai.2018/126","url":"https://ijcai.org/proceedings/2018/126","pdf":"https://ijcai.org/proceedings/2018/0126.pdf","markdown":"https://ijcai.org/proceedings/2018/126.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/126","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":126,"pages":{"first":906,"last":912},"keywords":["Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"Q:What is this person laying on? Our Prediction: bench Human Answer: bench Q:Is that a fuel efficient truck? Our Prediction: no Human Answer: no Q:Is the laptop turned on? Our Prediction: yes Human Answer: yes Q:What kind of cake is it? Our Prediction: chocolate Human Answer: Chocolate cake References [","title":"What is this person laying on? Our Prediction: bench Human Answer: bench Q:Is that a fuel efficient truck? Our Prediction: no Human Answer: no Q:Is the laptop turned on? Our Prediction: yes Human Answer: yes Q:What kind of cake is it? Our Prediction: chocolate Human Answer: Chocolate cake References","authors":"Q","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answer- ing. In ICCV, pages 2425-2433, 2015.","title":"VQA: visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca- cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In CVPR, 2017.","title":"Scacnn: Spatial and channel-wise attention in convolutional networks for image captioning","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. In EMNLP, pages 457-468, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2015] Haoyuan Gao, Junhua Mao, Jie Zhou, Zhi- heng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question. In NIPS, pages 2296-2304, 2015.","title":"Are you talking to a machine? dataset and methods for multilingual image question","authors":"Gao","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2017] Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. Video captioning with attention-based LSTM and semantic consistency. IEEE Trans. Multimedia, 19(9):2045-2055, 2017.","title":"Video captioning with attention-based LSTM and semantic consistency","authors":"Gao","year":"2017","venue":"IEEE Trans. Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image under- standing in visual question answering. In CVPR, pages 6325-6334, 2017.","title":"Making the V in VQA matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hyeonseob Nam and Kim, 2017] Jung-Woo Ha Hyeon- seob Nam and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In CVPR, 2017.","title":"Dual attention networks for multimodal reasoning and matching","authors":"Hyeonseob Nam, Kim ; Jung-Woo Ha, Hyeonseob Nam, Jeonghee Kim","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ilievski et al., 2016] Ilija Ilievski, Shuicheng Yan, and Ji- ashi Feng. A focused dynamic attention model for visual question answering. CoRR, abs/1604.01485, 2016.","title":"A focused dynamic attention model for visual question answering","authors":"Ilija Ilievski, Shuicheng Yan, Jiashi Feng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Jia, 2016] Ruiyu Li and Jiaya Jia. Visual question answering with question representation update (QRU). In NIPS, pages 4655-4663, 2016.","title":"Visual question answering with question representation update (QRU)","authors":"Jia ; Li, Ruiyu Li, Jiaya Jia","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NIPS, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2018] Pan Lu, Hongsheng Li, Wei Zhang, Jiany- ong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In AAAI, 2018.","title":"Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering","authors":"Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2016] Lin Ma, Zhengdong Lu, and Hang Li. Learning to answer questions from image using convolu- tional neural network. In AAAI, pages 3567-3573, 2016.","title":"Learning to answer questions from image using convolutional neural network","authors":"Lin Ma, Zhengdong Lu, Hang Li","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Malinowski et al., 2015] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural- based approach to answering questions about images. In ICCV, pages 1-9, 2015.","title":"Ask your neurons: A neuralbased approach to answering questions about images","authors":"Mateusz Malinowski, Marcus Rohrbach, Mario Fritz","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Noh et al., 2016] Hyeonwoo Noh, Paul Hongsuck Seo, and Bohyung Han. Image question answering using convolu- tional neural network with dynamic parameter prediction. In CVPR, pages 30-38, 2016.","title":"Image question answering using convolutional neural network with dynamic parameter prediction","authors":"Noh","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2015] Mengye Ren, Ryan Kiros, and Richard S. Zemel. Exploring models and data for image question an- swering. In NIPS, pages 2953-2961, 2015.","title":"Exploring models and data for image question answering","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(6):1137-1149, 2017.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shih et al., 2016] Kevin J. Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621, 2016.","title":"Where to look: Focus regions for visual question answering","authors":"Kevin J Shih, Saurabh Singh, Derek Hoiem","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2017] Jingkuan Song, Lianli Gao, Zhao Guo, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hier- archical LSTM with adjusted temporal attention for video captioning. In IJCAI, pages 2737-2743, 2017.","title":"Hierarchical LSTM with adjusted temporal attention for video captioning","authors":"Song","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/381"},{"position":22,"raw":"[Wang et al., 2017] Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. The vqa-machine: Learning how to use existing vision algorithms to answer new questions. In CVPR, pages 3909-3918, 2017.","title":"The vqa-machine: Learning how to use existing vision algorithms to answer new questions","authors":"Peng Wang, Qi Wu, Chunhua Shen, Anton Van Den, Hengel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony R. Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In CVPR, pages 4622- 4630, 2016.","title":"Ask me anything: Free-form visual question answering based on knowledge from external sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony R Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiong et al., 2016] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In ICML, pages 2397-2406, 2016.","title":"Dynamic memory networks for visual and textual question answering","authors":"Xiong","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and Saenko, 2016] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial at- tention for visual question answering. In ECCV, 2016.","title":"Ask, attend and answer: Exploring question-guided spatial attention for visual question answering","authors":"Saenko Xu, Huijuan Xu, Kate Saenko","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. Stacked attention networks for image question answering. In CVPR, 2016.","title":"Stacked attention networks for image question answering","authors":"Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alexander J Smola","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2017] Dongfei Yu, Jianlong Fu, Tao Mei, and Yong Rui. Multi-level attention networks for visual ques- tion answering. In CVPR, pages 4187-4195, 2017.","title":"Multi-level attention networks for visual question answering","authors":"Yu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2016] Yuke Zhu, Oliver Groth, Michael S. Bern- stein, and Li Fei-Fei. Visual7w: Grounded question an- swering in images. In CVPR, pages 4995-5004, 2016.","title":"Visual7w: Grounded question answering in images","authors":"Yuke Zhu, Oliver Groth, Michael S Bernstein, Li Fei-Fei","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}