{"type":"ScholarlyArticle","title":"Generative Visual Dialogue System via Weighted Likelihood Estimation","authors":[{"name":"Heming Zhang","slug":"heming-zhang","given":"Heming","family":"Zhang","affiliation":"University of Southern California"},{"name":"Shalini Ghosh","slug":"shalini-ghosh","given":"Shalini","family":"Ghosh","affiliation":"Samsung Research America"},{"name":"Larry Heck","slug":"larry-heck","given":"Larry","family":"Heck","affiliation":"Samsung Research America"},{"name":"Stephen Walsh","slug":"stephen-walsh","given":"Stephen","family":"Walsh","affiliation":"Samsung Research America"},{"name":"Junting Zhang","slug":"junting-zhang","given":"Junting","family":"Zhang","affiliation":"University of Southern California"},{"name":"Jie Zhang","slug":"jie-zhang","given":"Jie","family":"Zhang","affiliation":"Arizona State University"},{"name":"C.-C. Jay Kuo","slug":"c-c-jay-kuo","given":"C.-C. Jay","family":"Kuo","affiliation":"University of Southern California"}],"abstract":"The key challenge of generative Visual Dialogue (VD) systems is to respond to human queries with informative answers in natural and contiguous conversation flow. Traditional Maximum Likelihood Estimation-based methods only learn from positive responses but ignore the negative responses, and consequently tend to yield safe or generic responses. To address this issue, we propose a novel training scheme in conjunction with weighted likelihood estimation method. Furthermore, an adaptive multi-modal reasoning module is designed, to accommodate various dialogue scenarios automatically and select relevant information accordingly. The experimental results on the VisDial benchmark demonstrate the superiority of our proposed algorithm over other state-of-the-art approaches, with an improvement of 5.81% on recall@10.","doi":"10.24963/ijcai.2019/144","url":"https://ijcai.org/proceedings/2019/144","pdf":"https://ijcai.org/proceedings/2019/0144.pdf","markdown":"https://ijcai.org/proceedings/2019/144.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/144","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":144,"pages":{"first":1025,"last":1031},"keywords":["Agent-based and Multi-agent Systems: Human-Agent Interaction","Computer Vision: Language and Vision","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on com- puter vision, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Das et al., 2017] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017.","title":"Visual dialog","authors":"Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, M F José, Devi Moura, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Vries et al., 2017] Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In CVPR, volume 1, page 3, 2017.","title":"Guesswhat?! visual object discovery through multi-modal dialogue","authors":"De Vries","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gan et al., 2019] Zhe Gan, Yu Cheng, Ahmed EI Kholy, Linjie Li, Jingjing Liu, and Jianfeng Gao. Multi-step rea- soning via recurrent dual attention for visual dialog. arXiv preprint arXiv:1902.00579, 2019.","title":"Multi-step reasoning via recurrent dual attention for visual dialog","authors":"Zhe Gan, Yu Cheng, Ahmed Ei Kholy, Linjie Li, Jingjing Liu, Jianfeng Gao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Hexiang Hu, Wei-Lun Chao, and Fei Sha. Learning answer embeddings for visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5428-5436, 2018.","title":"Learning answer embeddings for visual question answering","authors":"Hu","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kottur et al., 2018] Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Visual coref- erence resolution in visual dialog using neural module net- works. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-169, 2018.","title":"Visual coreference resolution in visual dialog using neural module networks","authors":"Kottur","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversar- ial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547, 2017.","title":"Adversarial learning for neural dialogue generation","authors":"Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. International conference on computer vision, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"International conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2017] Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, and Dhruv Batra. Best of both worlds: Trans- ferring knowledge from discriminative learning to a gen- erative visual dialog model. In Advances in Neural Infor- mation Processing Systems, pages 314-324, 2017.","title":"Best of both worlds: Transferring knowledge from discriminative learning to a generative visual dialog model","authors":"Jiasen Lu, Anitha Kannan, Jianwei Yang, Devi Parikh, Dhruv Batra","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Malinowski et al., 2015] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural- based approach to answering questions about images. In Proceedings of the IEEE international conference on computer vision, pages 1-9, 2015.","title":"Ask your neurons: A neuralbased approach to answering questions about images","authors":"Mateusz Malinowski, Marcus Rohrbach, Mario Fritz","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Massiceti et al., 2018] Daniela Massiceti, N Siddharth, Puneet K Dokania, and Philip HS Torr. Flipdial: A gener- ative model for two-way visual dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Flipdial: A generative model for two-way visual dialogue","authors":"Daniela Massiceti, Siddharth, Philip Hs Puneet K Dokania, Torr","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ning et al., 2015] Kefeng Ning, Min Liu, and Mingyu Dong. A new robust elm method based on a bayesian framework with heavy-tailed distribution and weighted likelihood function. Neurocomputing, 149:891-903, 2015.","title":"A new robust elm method based on a bayesian framework with heavy-tailed distribution and weighted likelihood function","authors":"Kefeng Ning, Min Liu, Mingyu Dong","year":"2015","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rowley, 1999] Henry A Rowley. Neural network-based face detection. Technical report, Carnegie-Mellon Univ Pittsburgh PA Dept of Computer Science, 1999.","title":"Neural network-based face detection","authors":"A Henry, Rowley","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shrivastava et al., 2016] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 761-769, 2016.","title":"Training region-based object detectors with online hard example mining","authors":"Abhinav Shrivastava, Abhinav Gupta, Ross Girshick","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Confer- ence on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Warm, 1989] Thomas A Warm. Weighted likelihood esti- mation of ability in item response theory. Psychometrika, 54(3):427-450, 1989.","title":"Weighted likelihood estimation of ability in item response theory","authors":"A Thomas, Warm","year":"1989","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wen et al., 2016] Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network- based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562, 2016.","title":"A networkbased end-to-end trainable task-oriented dialogue system","authors":"Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, Steve Young","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2018] Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, and Anton van den Hengel. Are you talking to me? reasoned visual dialog generation through adversar- ial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Are you talking to me? reasoned visual dialog generation through adversarial learning","authors":"Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, Anton Van Den, Hengel","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu and Saenko, 2016] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial at- tention for visual question answering. In European Con- ference on Computer Vision, pages 451-466. Springer, 2016.","title":"Ask, attend and answer: Exploring question-guided spatial attention for visual question answering","authors":"Saenko Xu, Huijuan Xu, Kate Saenko","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Conference on computer vision and pattern recognition, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"Proceedings of the IEEE Conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018a] Heming Zhang, Xiaolong Wang, Jing- wen Zhu, and C-C Jay Kuo. Accelerating proposal gener- ation network for fast face detection on mobile devices. In 2018 25th IEEE International Conference on Image Pro- cessing (ICIP), pages 326-330. IEEE, 2018.","title":"Accelerating proposal generation network for fast face detection on mobile devices","authors":"Zhang","year":"2018","venue":"25th IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018b] Jie Zhang, Xiaolong Wang, Dawei Li, and Yalin Wang. Dynamically hierarchy revolution: dir- net for compressing recurrent neural network on mobile devices. IJCAI, 2018.","title":"Dynamically hierarchy revolution: dirnet for compressing recurrent neural network on mobile devices","authors":"Zhang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/429"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}