{"type":"ScholarlyArticle","title":"Dual Visual Attention Network for Visual Dialog","authors":[{"name":"Dan Guo","slug":"dan-guo","given":"Dan","family":"Guo","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Hui Wang","slug":"hui-wang","given":"Hui","family":"Wang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"}],"abstract":"Visual dialog is a challenging task, which involves multi-round semantic transformations between vision and language. This paper aims to address cross-modal semantic correlation for visual dialog. Motivated by that Vg (global vision), Vl (local vision), Q (question) and H (history) have inseparable relevances, the paper proposes a novel Dual Visual Attention Network (DVAN) to realize (Vg, Vl, Q, H)--> A. DVAN is a three-stage query-adaptive attention model. In order to acquire accurate A (answer), it first explores the textual attention, which imposes the question on history to pick out related context H'. Then, based on Q and H', it implements respective visual attentions to discover related global image visual hints Vg' and local object-based visual hints Vl'. Next, a dual crossing visual attention is proposed. Vg' and Vl' are mutually embedded to learn the complementary of visual semantics. Finally, the attended textual and visual features are combined to infer the answer. Experimental results on the VisDial v0.9 and v1.0 datasets validate the effectiveness of the proposed approach.","doi":"10.24963/ijcai.2019/693","url":"https://ijcai.org/proceedings/2019/693","pdf":"https://ijcai.org/proceedings/2019/0693.pdf","markdown":"https://ijcai.org/proceedings/2019/693.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/693","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":693,"pages":{"first":4989,"last":4995},"keywords":["Natural Language Processing: Dialogue","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module network- s. In CVPR, pages 39-48, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Z- itnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Das et al., 2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, pages 326-335, 2017.","title":"Visual dialog","authors":"Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, M F Jose, Devi Moura, Dhruv Parikh, Batra","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[de Vries et al., 2017] Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In CVPR, pages 5503-5512, 2017.","title":"Guesswhat?! visual object discovery through multi-modal dialogue","authors":"De Vries","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Farazi and Khan, 2018] Moshiur R. Farazi and Salman Khan. Reciprocal attention fusion for visual question an- swering. In BMVC, 2018.","title":"Reciprocal attention fusion for visual question answering","authors":"R Moshiur, Salman Farazi, Khan","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2016] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, pages 4555-4564, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, ab- s/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kottur et al., 2018] Satwik Kottur, Jose M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Visual coref- erence resolution in visual dialog using neural module net- works. In ECCV, pages 153-169, 2018.","title":"Visual coreference resolution in visual dialog using neural module networks","authors":"Kottur","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliv- er Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. IJCV, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2017a] Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, and Dhruv Batra. Best of both worlds: Trans- ferring knowledge from discriminative learning to a gener- ative visual dialog model. In NIPS, pages 314-324. 2017.","title":"Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model","authors":"Lu","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2017b] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR, pages 375-383, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2018] Pan Lu, Hongsheng Li, Wei Zhang, Jiany- ong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In AAAI, pages 7218-7225, 2018.","title":"Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering","authors":"Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Niu et al., 2018] Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, and Ji-Rong Wen. Recur- sive visual attention in visual dialog. arXiv preprint arX- iv:1812.02664, 2018.","title":"Recursive visual attention in visual dialog","authors":"Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, Ji-Rong Wen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Patro and Namboodiri, 2018] Badri N. Patro and Vinay P. Namboodiri. Differential attention for visual question an- swering. In CVPR, pages 7680-7688, 2018.","title":"Differential attention for visual question answering","authors":"N Badri, Patro, P Vinay, Namboodiri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99. 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seo et al., 2017] Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal. Visual reference reso- lution using attention memory for visual dialog. In NIPS, pages 3719-3729. 2017.","title":"Visual reference resolution using attention memory for visual dialog","authors":"Paul Hongsuck, Seo, Andreas Lehrmann, Bohyung Han, Leonid Sigal","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Serban et al., 2017] Iulian V. Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In AAAI, pages 3295-3301, 2017.","title":"A hierarchical latent variable encoder-decoder model for generating dialogues","authors":"V Iulian, Alessandro Serban, Ryan Sordoni, Laurent Lowe, Joelle Charlin, Aaron Pineau, Yoshua Courville, Bengio","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2018] Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, and Anton van den Hengel. Are you talking to me? reasoned visual dialog generation through adversarial learning. In CVPR, pages 6106-6115, 2018.","title":"Are you talking to me? reasoned visual dialog generation through adversarial learning","authors":"Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, Anton Van Den, Hengel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdi- nov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Hanwang Zhang, Yulei Niu, and Shih- Fu Chang. Grounding referring expressions in images by variational context. In CVPR, pages 4158-4166, 2018.","title":"Grounding referring expressions in images by variational context","authors":"Hanwang Zhang, Yulei Niu, Shih-Fu Chang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}