{"type":"ScholarlyArticle","title":"Using Natural Language for Reward Shaping in Reinforcement Learning","authors":[{"name":"Prasoon Goyal","slug":"prasoon-goyal","given":"Prasoon","family":"Goyal","affiliation":"The University of Texas at Austin"},{"name":"Scott Niekum","slug":"scott-niekum","given":"Scott","family":"Niekum","affiliation":"The University of Texas at Austin"},{"name":"Raymond J. Mooney","slug":"raymond-j-mooney","given":"Raymond J.","family":"Mooney","affiliation":"The University of Texas at Austin"}],"abstract":"Recent reinforcement learning (RL) approaches have shown strong performance in complex domains, such as Atari games, but are highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. Designing such rewards remains a challenge, though. In this work, we use natural language instructions to perform reward shaping. We propose a framework that maps free-form natural language instructions to intermediate rewards, that can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari video games domain, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that for the same number of interactions with the environment, using language-based rewards can successfully complete the task 60% more often, averaged across all tasks, compared to learning without language.","doi":"10.24963/ijcai.2019/331","url":"https://ijcai.org/proceedings/2019/331","pdf":"https://ijcai.org/proceedings/2019/0331.pdf","markdown":"https://ijcai.org/proceedings/2019/331.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/331","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":331,"pages":{"first":2385,"last":2391},"keywords":["Natural Language Processing: Natural Language Processing","Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Arumugam et al., 2017] Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson LS Wong, and Stefanie Tellex. Accurately and efficiently interpreting human-robot instructions of varying granularities. arXiv preprint arXiv:1704.06616, 2017.","title":"Accurately and efficiently interpreting human-robot instructions of varying granularities","authors":"Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson Ls Wong, Stefanie Tellex","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2018] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning to follow language instruc- tions with adversarial reward induction. arXiv preprint arXiv:1806.01946, 2018.","title":"Learning to follow language instructions with adversarial reward induction","authors":"Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, Edward Grefenstette","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Branavan et al., 2012a] SRK Branavan, Nate Kushman, Tao Lei, and Regina Barzilay. Learning high-level planning from text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 126-135. Association for Com- putational Linguistics, 2012.","title":"Learning high-level planning from text","authors":"Branavan","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Branavan et al., 2012b] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.","title":"Learning to win by reading manuals in a monte-carlo framework","authors":"Branavan","year":"2012","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conneau et al., 2017] Alexis Conneau, Douwe Kiela, Hol- ger Schwenk, Loïc Barrault, and Antoine Bordes. Su- pervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics.","title":"Supervised learning of universal sentence representations from natural language inference data","authors":"Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, Antoine Bordes","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Harnad, 1990] Stevan Harnad. The symbol grounding prob- lem. Physica D: Nonlinear Phenomena, 42(1-3):335-346, 1990.","title":"The symbol grounding problem","authors":"Stevan Harnad","year":"1990","venue":"Physica D: Nonlinear Phenomena","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaplan et al., 2017] Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating atari with natural lan- guage guided reinforcement learning. arXiv preprint arXiv:1704.05539, 2017.","title":"Beating atari with natural language guided reinforcement learning","authors":"Russell Kaplan, Christopher Sauer, Alexander Sosa","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kostrikov, 2018] Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://github.com/ ikostrikov/pytorch-a2c-ppo-acktr, 2018.","title":"Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms","authors":"Kostrikov","year":"2018","venue":null,"doi":null,"url":"https://github.com/ikostrikov/pytorch-a2c-ppo-acktr,2018","internal_url":null},{"position":12,"raw":"[Kuhlmann et al., 2004] Gregory Kuhlmann, Peter Stone, Raymond Mooney, and Jude Shavlik. Guiding a reinforce- ment learner with natural language advice: Initial results in robocup soccer. In The AAAI-2004 workshop on super- visory control of learning and adaptive systems. San Jose, CA, 2004.","title":"Guiding a reinforcement learner with natural language advice: Initial results in robocup soccer","authors":"Kuhlmann","year":"2004","venue":"The AAAI-2004 workshop on supervisory control of learning and adaptive systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kurin et al., 2017] Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset. arXiv preprint arXiv:1705.10998, 2017.","title":"The atari grand challenge dataset","authors":"Sebastian Vitaly Kurin, Katja Nowozin, Lucas Hofmann, Bastian Beyer, Leibe","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforce- ment learning for dialogue generation. arXiv preprint arXiv:1606.01541, 2016.","title":"Deep reinforcement learning for dialogue generation","authors":"Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, Dan Jurafsky","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Misra et al., 2017] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint arXiv:1704.08795, 2017.","title":"Mapping instructions and visual observations to actions with reinforcement learning","authors":"Dipendra Misra, John Langford, Yoav Artzi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paulus et al., 2017] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017.","title":"A deep reinforced model for abstractive summarization","authors":"Romain Paulus, Caiming Xiong, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Večerík et al., 2017] Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Ried- miller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017.","title":"Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards","authors":"Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, Martin Riedmiller","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Xin Wang, Qiuyuan Huang, Asli Ce- likyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Rein- forced cross-modal matching and self-supervised imitation learning for vision-language navigation. arXiv preprint arXiv:1811.10092, 2018.","title":"Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation","authors":"Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang, Wang, Lei Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams et al., 2017] Edward C Williams, Mina Rhee, Nakul Gopalan, and Stefanie Tellex. Learning to parse natural language to grounded reward functions with weak supervision. In AAAI Fall Symposium on Natural Commu- nication for Human-Robot Collaboration, 2017.","title":"Learning to parse natural language to grounded reward functions with weak supervision","authors":"Mina Edward C Williams, Nakul Rhee, Stefanie Gopalan, Tellex","year":"2017","venue":"AAAI Fall Symposium on Natural Communication for Human-Robot Collaboration","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xiong et al., 2017] Caiming Xiong, Victor Zhong, and Richard Socher. Dcn+: Mixed objective and deep resid- ual coattention for question answering. arXiv preprint arXiv:1711.00106, 2017.","title":"Dcn+: Mixed objective and deep residual coattention for question answering","authors":"Xiong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}