{"type":"ScholarlyArticle","title":"End-to-end optimization of goal-driven and visually grounded dialogue systems","authors":[{"name":"Florian Strub","slug":"florian-strub","given":"Florian","family":"Strub","affiliation":"Univ. Lille , Inria , CNRS , Centrale Lille , UMR 9189 - CRIStAL"},{"name":"Harm de Vries","slug":"harm-de-vries","given":"Harm","family":"de Vries","affiliation":"University of Montreal"},{"name":"Jérémie Mary","slug":"jeremie-mary","given":"Jérémie","family":"Mary","affiliation":"Univ. Lille , Inria , CNRS , Centrale Lille , UMR 9189 - CRIStAL"},{"name":"Bilal Piot","slug":"bilal-piot","given":"Bilal","family":"Piot","affiliation":"DeepMind"},{"name":"Aaron Courville","slug":"aaron-courville","given":"Aaron","family":"Courville","affiliation":"University of Montreal"},{"name":"Olivier Pietquin","slug":"olivier-pietquin","given":"Olivier","family":"Pietquin","affiliation":"DeepMind"}],"abstract":"End-to-end design of dialogue systems has recently become a popular research topic thanks to powerful tools such as encoder-decoder architectures for sequence-to-sequence learning. Yet, most current approaches cast human-machine dialogue management as a supervised learning problem, aiming at predicting the next utterance of a participant given the full history of the dialogue. This vision may fail to correctly render the planning problem inherent to dialogue as well as its contextual and grounded nature. In this paper, we introduce a Deep Reinforcement Learning method to optimize visually grounded task-oriented dialogues, based on the policy gradient algorithm. This approach is tested on the question generation task from the dataset GuessWhat?! containing 120k dialogues and provides encouraging results at solving both the problem of generating natural dialogues and the task of discovering a specific object in a complex picture.","doi":"10.24963/ijcai.2017/385","url":"https://ijcai.org/proceedings/2017/385","pdf":"https://ijcai.org/proceedings/2017/0385.pdf","markdown":"https://ijcai.org/proceedings/2017/385.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/385","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":385,"pages":{"first":2765,"last":2771},"keywords":["Natural Language Processing: Dialogue","Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, Z. Lawrence, and D. Parikh. Vqa: Visual question answering. In Proc. of ICCV, 2015.","title":"Vqa: Visual question answering","authors":"S Antol, A Agrawal, J Lu, M Mitchell, D Batra, Z Lawrence, D Parikh","year":"2015","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2017] D. Bahdanau, P. Brakel, K. Kelvin, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio. An actor-critic algorithm for sequence prediction. Proc. of ICLR, 2017.","title":"An actor-critic algorithm for sequence prediction","authors":"D Bahdanau, P Brakel, K Kelvin, A Goyal, R Lowe, J Pineau, A Courville, Y Bengio","year":"2017","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barsalou, 2008] L. Barsalou. Grounded cognition. Annu. Rev. Psy- chol., 59:617-645, 2008.","title":"Grounded cognition","authors":"L Barsalou","year":"2008","venue":"Annu. Rev. Psychol","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. JMLR, 3(Feb):1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Y Bengio, R Ducharme, P Vincent, C Jauvin","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Clark and Schaefer, 1989] H. Clark and E. Schaefer. Contributing to discourse. Cognitive Science, 13(2):259-294, 1989.","title":"Contributing to discourse","authors":"H Clark, E Schaefer","year":"1989","venue":"Cognitive Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das et al., 2017a] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. Moura, D. Parikh, and D. Batra. Visual Dialog. In Proc. of CVPR, 2017.","title":"Visual Dialog","authors":"Das","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Das et al., 2017b] A. Das, S. Kottur, J. Moura, S. Lee, and D. Ba- tra. Learning Cooperative Visual Dialog Agents with Deep Rein- forcement Learning. arXiv preprint arXiv:1703.06585, 2017.","title":"Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning","authors":"Das","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[de Vries et al., 2016] H. de Vries, F. Strub, S. Chandar, O. Pietquin, H. Larochelle, and A. Courville. GuessWhat?! Visual object discovery through multi-modal dialogue. Proc. of CVPR, 2016.","title":"GuessWhat?! Visual object discovery through multi-modal dialogue","authors":"H Vries, F Strub, S Chandar, O Pietquin, H Larochelle, A Courville","year":"2016","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Eckert et al., 1997] W. Eckert, E. Levin, and R. Pieraccini. User modeling for spoken dialogue system evaluation. In Proc. of ASRU, 1997.","title":"User modeling for spoken dialogue system evaluation","authors":"W Eckert, E Levin, R Pieraccini","year":"1997","venue":"Proc. of ASRU","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] S. Hochreiter and J. Schmid- huber. Long short-term memory. volume 9, pages 1735-1780.","title":"Long short-term memory","authors":"S Hochreiter, J Schmidhuber","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kiela et al., 2016] D. Kiela, L. Bulat, A.Vero, and S. Clark. Vir- tual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research. NIPS workshop in Machine Intelligence, 2016.","title":"Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research","authors":"D Kiela, L Bulat, A Vero, S Clark","year":"2016","venue":"NIPS workshop in Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lemon and Pietquin, 2007] O. Lemon and O. Pietquin. Machine learning for spoken dialogue systems. In Proc. of Interspeech, 2007.","title":"Machine learning for spoken dialogue systems","authors":"O Lemon, O Pietquin","year":"2007","venue":"Proc. of Interspeech","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levin et al., 1997] E. Levin, R. Pieraccini, and W. Eckert. Learn- ing dialogue strategies within the markov decision process frame- work. In Proc. of ASRU, 1997.","title":"Learning dialogue strategies within the markov decision process framework","authors":"E Levin, R Pieraccini, W Eckert","year":"1997","venue":"Proc. of ASRU","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] TY. Lin, M. Maire, S. Belongie, J. Hays, P. Per- ona, D. Ramanan, P. Dollár, and L. Zitnick. Microsoft coco: Common objects in context. In Proc. of ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Lin","year":"2014","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016a] C. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proc. of EMNLP, 2016.","title":"How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation","authors":"Liu","year":"2016","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016b] S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy. Optimization of image description metrics using pol- icy gradient methods. Under review at CVPR, 2016.","title":"Optimization of image description metrics using policy gradient methods","authors":"Liu","year":"2016","venue":"Under review at CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lowe et al., 2015] R. Lowe, N. Pow, I. Serban, and J. Pineau. The ubuntu dialogue corpus: A large dataset for research in unstruc- tured multi-turn dialogue systems. In Proc. of SIGdial, 2015.","title":"The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems","authors":"R Lowe, N Pow, I Serban, J Pineau","year":"2015","venue":"Proc. of SIGdial","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2016] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchi- cal Question-Image Co-Attention for Visual Question Answer- ing. In Proc. of NIPS, 2016.","title":"Hierarchical Question-Image Co-Attention for Visual Question Answering","authors":"J Lu, J Yang, D Batra, D Parikh","year":"2016","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mooney, 2006] R. Mooney. Learning language from perceptual context: A challenge problem for AI. In Proc. of the 2006 AAAI Fellows Symposium, 2006.","title":"Learning language from perceptual context: A challenge problem for AI","authors":"R Mooney","year":"2006","venue":"Proc. of the 2006 AAAI Fellows Symposium","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng et al., 1999] A. Ng, D. Harada, and S. Russell. Policy invari- ance under reward transformations: Theory and application to reward shaping. In Proc. of ICML, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"A Ng, D Harada, S Russell","year":"1999","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pietquin and Hastie, 2013] O. Pietquin and H. Hastie. A survey on metrics for the evaluation of user simulations. The Knowledge Engineering Review, 28(1):59-73, 003 2013.","title":"A survey on metrics for the evaluation of user simulations","authors":"O Pietquin, H Hastie","year":"2013","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pietquin et al., 2011] O. Pietquin, M. Geist, S. Chandramohan, and H. Frezza-Buet. Sample-efficient batch reinforcement learn- ing for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):7, 2011.","title":"Sample-efficient batch reinforcement learning for dialogue management optimization","authors":"O Pietquin, M Geist, S Chandramohan, H Frezza, Buet","year":"2011","venue":"ACM Transactions on Speech and Language Processing (TSLP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ranzato et al., 2016] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural net- works. Proc. of ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"M Ranzato, S Chopra, M Auli, W Zaremba","year":"2016","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Roy, 2002] D. Roy. Learning visually grounded words and syn- tax for a scene description task. Computer speech & language, 16(3):353-385, 2002.","title":"Learning visually grounded words and syntax for a scene description task","authors":"D Roy","year":"2002","venue":"Computer speech & language","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schatzmann et al., 2006] J. Schatzmann, K. Weilhammer, M. Stut- tle, and S. Young. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(2):97-126, 2006.","title":"A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies","authors":"J Schatzmann, K Weilhammer, M Stuttle, S Young","year":"2006","venue":"The knowledge engineering review","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Serban et al., 2016] I.","title":null,"authors":"Serban","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Serban, R. Lowe, L. Charlin, and J. Pineau. Generative Deep Neural Networks for Dialogue: A Short Review. NIPS workshop Learning Methods for Dialogue, 2016.","title":"Generative Deep Neural Networks for Dialogue: A Short Review","authors":"R Serban, L Lowe, J Charlin, Pineau","year":"2016","venue":"NIPS workshop Learning Methods for Dialogue","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silver et al., 2016] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"D Silver, A Huang, C Maddison, A Guez, L Sifre, G Van Den Driessche","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Singh et al., 1999] S. Singh, M. Kearns, D. Litman, and M. Walker. Reinforcement Learning for Spoken Dialogue Sys- tems. In Proc. of NIPS, 1999.","title":"Reinforcement Learning for Spoken Dialogue Systems","authors":"S Singh, M Kearns, D Litman, M Walker","year":"1999","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sordoni et al., 2015] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, JY. Nie, J. Gao, and B. Dolan. A neural net- work approach to context-sensitive generation of conversational responses. In Proc. of NAACL HLT, 2015.","title":"A neural network approach to context-sensitive generation of conversational responses","authors":"A Sordoni, M Galley, M Auli, C Brockett, Y Ji, M Mitchell, J Y Nie, J Gao, B Dolan","year":"2015","venue":"Proc. of NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton et al., 1999] R. Sutton, D. McAllester, S. Singh, Y. Man- sour, et al. Policy gradient methods for reinforcement learning with function approximation. In Proc. of NIPS, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"R Sutton, D Mcallester, S Singh, Y Mansour","year":"1999","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Vinyals and Le, 2015] O. Vinyals and Q. Le. A neural conversa- tional model. ICML Deep Learning Workshop, 2015.","title":"A neural conversational model","authors":"O Vinyals, Q Le","year":"2015","venue":"ICML Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Walker et al., 1997] M. Walker, D. Litman, C. Kamm, and A. Abella. Paradise: A framework for evaluating spoken dia- logue agents. In Proc. of ACL, 1997.","title":"Paradise: A framework for evaluating spoken dialogue agents","authors":"M Walker, D Litman, C Kamm, A Abella","year":"1997","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Weizenbaum, 1966] J. Weizenbaum. Eliza—a computer program for the study of natural language communication be- tween man and machine. Commun. ACM, 9(1):36-45, 1966.","title":"Eliza—a computer program for the study of natural language communication between man and machine","authors":"J Weizenbaum","year":"1966","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Williams and Young, 2007] J. Williams and S. Young. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393-422, 2007.","title":"Partially observable markov decision processes for spoken dialog systems","authors":"J Williams, S Young","year":"2007","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Williams, 1992] R. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"R Williams","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}