{"type":"ScholarlyArticle","title":"Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents","authors":[{"name":"Wenhan Xiong","slug":"wenhan-xiong","given":"Wenhan","family":"Xiong","affiliation":"University of California , Santa Barbara"},{"name":"Xiaoxiao Guo","slug":"xiaoxiao-guo","given":"Xiaoxiao","family":"Guo","affiliation":"IBM Research"},{"name":"Mo Yu","slug":"mo-yu","given":"Mo","family":"Yu","affiliation":"IBM Research"},{"name":"Shiyu Chang","slug":"shiyu-chang","given":"Shiyu","family":"Chang","affiliation":"IBM Research"},{"name":"Bowen Zhou","slug":"bowen-zhou","given":"Bowen","family":"Zhou","affiliation":"JD AI Research"},{"name":"William Yang Wang","slug":"william-yang-wang","given":"William Yang","family":"Wang","affiliation":"University of California , Santa Barbara"}],"abstract":"We investigate the task of learning to interpret natural language instructions by jointly reasoning with visual observations and language inputs. Unlike current methods which start with learning from demonstrations (LfD) and then use reinforcement learning (RL) to fine-tune the model parameters, we propose a novel policy optimization algorithm which can dynamically schedule demonstration learning and RL. The proposed training paradigm provides efficient exploration and generalization beyond existing methods. Comparing to existing ensemble models, the best single model based on our proposed method tremendously decreases the execution error by 55% on a block-world environment. To further illustrate the exploration strategy of our RL algorithm, our paper includes systematic studies on the evolution of policy entropy during training.","doi":"10.24963/ijcai.2018/626","url":"https://ijcai.org/proceedings/2018/626","pdf":"https://ijcai.org/proceedings/2018/0626.pdf","markdown":"https://ijcai.org/proceedings/2018/626.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/626","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":626,"pages":{"first":4503,"last":4509},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Robotics: Human Robot Interaction","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2015] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for se- quence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems, pages 1171-1179, 2015.","title":"Scheduled sampling for sequence prediction with recurrent neural networks","authors":"Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bisk et al., 2016] Yonatan Bisk, Deniz Yuret, and Daniel Marcu. Natural language communication with robots. In HLT-NAACL, pages 751-761, 2016.","title":"Natural language communication with robots","authors":"Yonatan Bisk, Deniz Yuret, Daniel Marcu","year":"2016","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Branavan et al., 2009] Satchuthananthavale RK Branavan, Harr Chen, Luke S Zettlemoyer, and Regina Barzilay. Re- inforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Con- ference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 82-90. Association for Compu- tational Linguistics, 2009.","title":"Reinforcement learning for mapping instructions to actions","authors":"Branavan","year":"2009","venue":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Mooney, 2011] David L Chen and Raymond J Mooney. Learning to interpret natural language naviga- tion instructions from observations. In AAAI, volume 2, pages 1-2, 2011.","title":"Learning to interpret natural language navigation instructions from observations","authors":"L David, Raymond J Chen, Mooney","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Di Eugenio, 1992] Barbara Di Eugenio. Understanding nat- ural language instructions: the case of purpose clauses. In Proceedings of the 30th annual meeting on Association for Computational Linguistics, pages 120-127. Associa- tion for Computational Linguistics, 1992.","title":"Understanding natural language instructions: the case of purpose clauses","authors":"Barbara Di, Eugenio","year":"1992","venue":"Proceedings of the 30th annual meeting on Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim and Mooney, 2013] Joohyun Kim and Raymond J Mooney. Adapting discriminative reranking to grounded language learning. In ACL (1), pages 218-227, 2013.","title":"Adapting discriminative reranking to grounded language learning","authors":"Mooney Kim, Joohyun Kim, Raymond J Mooney","year":"2013","venue":"ACL (1)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Levine et al., 2016] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learn- ing hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, page 0278364917710318, 2016.","title":"Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection","authors":"Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, Deirdre Quillen","year":"2016","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liang et al., 2016] Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. Neural symbolic ma- chines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020, 2016.","title":"Neural symbolic machines: Learning semantic parsers on freebase with weak supervision","authors":"Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, Ni Lao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[MacMahon et al., 2006] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. Def, 2(6):4, 2006.","title":"Walk the talk: Connecting language, knowledge, and action in route instructions","authors":"Matt Macmahon, Brian Stankiewicz, Benjamin Kuipers","year":"2006","venue":"Def","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Matuszek et al., 2010] Cynthia Matuszek, Dieter Fox, and Karl Koscher. Following directions using statistical ma- chine translation. In Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on, pages 251-258. IEEE, 2010.","title":"Following directions using statistical machine translation","authors":"Cynthia Matuszek, Dieter Fox, Karl Koscher","year":"2010","venue":"5th ACM/IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mei et al., 2016] Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In AAAI, volume 1, page 2, 2016.","title":"Listen, attend, and walk: Neural mapping of navigational instructions to action sequences","authors":"Mei","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Misra et al., 2017] Dipendra K Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), Copenhagen, Den- mark, September 2017. ACL.","title":"Mapping instructions and visual observations to actions with reinforcement learning","authors":"K Dipendra, John Misra, Yoav Langford, Artzi","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Peters and Schaal, 2008] Jan Peters and Stefan Schaal. Nat- ural actor-critic. Neurocomputing, 71(7):1180-1190, 2008.","title":"Jan Peters and Stefan Schaal","authors":"Schaal Peters","year":"2008","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pomerleau, 1991] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"A Dean, Pomerleau","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ross et al., 2011] Stéphane Ross, Geoffrey J Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stéphane Ross, Geoffrey J Gordon, Drew Bagnell","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tellex et al., 2011] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. Understanding natu- ral language commands for robotic navigation and mobile manipulation. In AAAI, 2011.","title":"Understanding natural language commands for robotic navigation and mobile manipulation","authors":"Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, Nicholas Roy","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Walsh et al., 2011] Thomas J Walsh, Daniel K Hewlett, and Clayton T Morrison. Blending autonomous exploration and apprenticeship learning. In Advances in Neural Infor- mation Processing Systems, pages 2258-2266, 2011.","title":"Blending autonomous exploration and apprenticeship learning","authors":"J Thomas, Daniel K Walsh, Clayton T Hewlett, Morrison","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Winograd, 1972] Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.","title":"Terry Winograd. Understanding natural language","authors":"Winograd","year":"1972","venue":"Cognitive psychology","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}