{"type":"ScholarlyArticle","title":"DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue","authors":[{"name":"Xiaoze Jiang","slug":"xiaoze-jiang","given":"Xiaoze","family":"Jiang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; Intelligent Computing and Machine Learning Lab , School of ASEE , Beihang University , Beijing , China"},{"name":"Jing Yu","slug":"jing-yu","given":"Jing","family":"Yu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yajing Sun","slug":"yajing-sun","given":"Yajing","family":"Sun","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zengchang Qin","slug":"zengchang-qin","given":"Zengchang","family":"Qin","affiliation":"Intelligent Computing and Machine Learning Lab , School of ASEE , Beihang University , Beijing , China; AI Research , Codemao Inc."},{"name":"Zihao Zhu","slug":"zihao-zhu","given":"Zihao","family":"Zhu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yue Hu","slug":"yue-hu","given":"Yue","family":"Hu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Qi Wu","slug":"qi-wu","given":"Qi","family":"Wu","affiliation":"University of Adelaide , Australia"}],"abstract":"Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.","doi":"10.24963/ijcai.2020/96","url":"https://ijcai.org/proceedings/2020/96","pdf":"https://ijcai.org/proceedings/2020/0096.pdf","markdown":"https://ijcai.org/proceedings/2020/96.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/96","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":96,"pages":{"first":687,"last":693},"keywords":["Natural Language Processing: Dialogue","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24139/file_24139.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2020] Feilong Chen, Fandong Meng, Jiaming Xu, Peng Li, Bo Xu, and Jie Zhou. Dmrm: A dual-channel multi-hop reasoning model for visual dialog. In AAAI, 2020.","title":"Dmrm: A dual-channel multi-hop reasoning model for visual dialog","authors":"Feilong Chen, Fandong Meng, Jiaming Xu, Peng Li, Bo Xu, Jie Zhou","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Das et al., 2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, pages 1080-1089, 2017.","title":"Visual dialog","authors":"Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, M F José, Devi Moura, Dhruv Parikh, Batra","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gan et al., 2019] Zhe Gan, Yu Cheng, Ahmed Ei Kholy, Linjie Li, and Jianfeng Gao. Multi-step reasoning via recurrent dual attention for visual dialog. In ACL, page 6463-6474, 2019.","title":"Multi-step reasoning via recurrent dual attention for visual dialog","authors":"Zhe Gan, Yu Cheng, Ahmed Ei Kholy, Linjie Li, Jianfeng Gao","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2019] Dalu Guo, Chang Xu, and Dacheng Tao. Image-question-answer synergistic network for visual dia- log. In CVPR, pages 10434-10443, 2019.","title":"Image-question-answer synergistic network for visual dialog","authors":"Dalu Guo, Chang Xu, Dacheng Tao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hopfield, 1982] John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America, 79(8):2554-2558, 1982.","title":"Neural networks and physical systems with emergent collective computational abilities","authors":"John J Hopfield","year":"1982","venue":"Proceedings of the National Academy of Sciences of the United States of America","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2020] Xiaoze Jiang, Jing Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, and Qi Wu. Dualvd: An adaptive dual encoding model for deep visual understanding in visual dialogue. In AAAI, 2020.","title":"Dualvd: An adaptive dual encoding model for deep visual understanding in visual dialogue","authors":"Xiaoze Jiang, Jing Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, Qi Wu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kong and Wu, 2018] Dejiang Kong and Fei Wu. Visual dia- log with multi-turn attentional memory network. In PCM, pages 611-621, 2018.","title":"Visual dialog with multi-turn attentional memory network","authors":"Wu Kong, Dejiang Kong, Fei Wu","year":"2018","venue":"PCM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kottur et al., 2018] Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Visual coref- erence resolution in visual dialog using neural module net- works. In ECCV, pages 153-169, 2018.","title":"Visual coreference resolution in visual dialog using neural module networks","authors":"Kottur","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lei et al., 2018] Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequic- ity: Simplifying task-oriented dialogue systems with sin- gle sequence-to-sequence architectures. In ACL, pages 1437-1447, 2018.","title":"Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures","authors":"Lei","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2017] Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, and Dhruv Batra. Best of both worlds: Trans- ferring knowledge from discriminative learning to a gener- ative visual dialog model. In NIPS, pages 314-324, 2017.","title":"Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model","authors":"Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, Dhruv Batra","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Madotto et al., 2018] Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog sys- tems. In ACL, pages 1468-1478, 2018.","title":"Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems","authors":"Andrea Madotto, Chien-Sheng Wu, Pascale Fung","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Niu et al., 2019] Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, and Ji-Rong Wen. Recursive visual attention in visual dialog. In CVPR, pages 6679- 6688, 2019.","title":"Recursive visual attention in visual dialog","authors":"Niu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peng et al., 2018] Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. Deep Dyna-Q: Integrat- ing planning for task-completion dialogue policy learning. In ACL, pages 2182-2192, 2018.","title":"Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning","authors":"Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qi et al., 2020] Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang. Two causal principles for improving vi- sual dialog. In CVPR, 2020.","title":"Two causal principles for improving visual dialog","authors":"Jiaxin Qi, Yulei Niu, Jianqiang Huang, Hanwang Zhang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rashkin et al., 2019] Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: A new benchmark and dataset. In ACL, pages 5370-5381, 2019.","title":"Towards empathetic open-domain conversation models: A new benchmark and dataset","authors":"Eric Michael Hannah Rashkin, Margaret Smith, Y-Lan Li, Boureau","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schwartz et al., 2019] Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. Factor graph attention. In CVPR, pages 2039-2048, 2019.","title":"Factor graph attention","authors":"Idan Schwartz, Seunghak Yu, Tamir Hazan, Alexander G Schwing","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[See et al., 2017] Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. In ACL, page 1073-1083, 2017.","title":"Get to the point: Summarization with pointergenerator networks","authors":"Abigail See, Peter J Liu, Christopher D Manning","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shao et al., 2017] Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. Generating high-quality and informative conversation re- sponses with sequence-to-sequence models. In EMNLP, page 2210-2219, 2017.","title":"Generating high-quality and informative conversation responses with sequence-to-sequence models","authors":"Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, Ray Kurzweil","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2019] Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuanjing Huang. Generating responses with a specific emotion in dialog. In ACL, pages 3685-3695, 2019.","title":"Generating responses with a specific emotion in dialog","authors":"Song","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tian et al., 2019] Zhiliang Tian, Wei Bi, Xiaopeng Li, and Nevin L. Zhang. Learning to abstract for memory- augmented conversational response generation. In ACL, pages 3816-3825, 2019.","title":"Learning to abstract for memoryaugmented conversational response generation","authors":"Zhiliang Tian, Wei Bi, Xiaopeng Li, Nevin L Zhang","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2018] Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, and Anton van den Hengel. Are you talking to me? reasoned visual dialog generation through adversarial learning. In CVPR, pages 6106-6115, 2018.","title":"Are you talking to me? reasoned visual dialog generation through adversarial learning","authors":"Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, Anton Van Den, Hengel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Can Xu, Wei Wu, Chongyang Tao, Huang Hu, Matt Schuerman, and Ying Wang. Neural response generation with meta-words. In ACL, pages 5416-5426, 2019.","title":"Neural response generation with meta-words","authors":"Xu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2019] Tianhao Yang, Zheng-Jun Zha, and Han- wang Zhang. Making history matter: Gold-critic sequence training for visual dialog. In ICCV, pages 2561-2569, 2019.","title":"Making history matter: Gold-critic sequence training for visual dialog","authors":"Tianhao Yang, Zheng-Jun Zha, Hanwang Zhang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Heming Zhang, Shalini Ghosh, Larry Heck, Stephen Walsh, Junting Zhang, Jie Zhang, and C- C Jay Kuo. Generative visual dialogue system via adaptive reasoning and weighted likelihood estimation. In IJCAI, pages 1025-1031, 2019.","title":"Generative visual dialogue system via adaptive reasoning and weighted likelihood estimation","authors":"Heming Zhang, Shalini Ghosh, Larry Heck, Stephen Walsh, Junting Zhang, Jie Zhang, C-C Jay Kuo","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2019] Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. Reasoning visual dialogs with structural and partial observations. In CVPR, pages 6669- 6678, 2019.","title":"Reasoning visual dialogs with structural and partial observations","authors":"Zilong Zheng, Wenguan Wang, Siyuan Qi, Song-Chun Zhu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}