{"type":"ScholarlyArticle","title":"Data Augmentation for Learning to Play in Text-Based Games","authors":[{"name":"Jinhyeon Kim","slug":"jinhyeon-kim","given":"Jinhyeon","family":"Kim","affiliation":"Skelter Labs"},{"name":"Kee-Eung Kim","slug":"kee-eung-kim","given":"Kee-Eung","family":"Kim","affiliation":"Skelter Labs"}],"abstract":"Improving generalization in text-based games serves as a useful stepping-stone towards reinforcement learning (RL) agents with generic linguistic ability. Data augmentation for generalization in RL has shown to be very successful in classic control and visual tasks, but there is no prior work for text-based games. We propose Transition-Matching Permutation, a novel data augmentation technique for text-based games, where we identify phrase permutations that match as many transitions in the trajectory data. We show that applying this technique results in state-of-the-art performance in the Cooking Game benchmark suite for text-based games.","doi":"10.24963/ijcai.2022/436","url":"https://ijcai.org/proceedings/2022/436","pdf":"https://ijcai.org/proceedings/2022/0436.pdf","markdown":"https://ijcai.org/proceedings/2022/436.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/436","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":436,"pages":{"first":3143,"last":3149},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning","Planning and Scheduling: POMDPs","Natural Language Processing: Other"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986698","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984841","length_sec":303,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adhikari et al., 2020] Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33, 2020.","title":"Learning dynamic belief graphs to generalize on text-based games","authors":"Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, Will Hamilton","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Campagna et al., 2020] Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica Lam. Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 122-132, 2020.","title":"Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking","authors":"Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, Monica Lam","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhury et al., 2020] Subhajit Chaudhury, Daiki Kimura, Kartik Talamadupula, Michiaki Tatsubori, Asim Munawar, and Ryuki Tachibana. Bootstrapped Q-learning with context relevant observation pruning to generalize in text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3002-3008, 2020.","title":"Bootstrapped Q-learning with context relevant observation pruning to generalize in text-based games","authors":"Subhajit Chaudhury, Daiki Kimura, Kartik Talamadupula, Michiaki Tatsubori, Asim Munawar, Ryuki Tachibana","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cobbe et al., 2019] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying general- ization in reinforcement learning. In International Con- ference on Machine Learning, pages 1282-1289. PMLR, 2019.","title":"Quantifying generalization in reinforcement learning","authors":"Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Côté et al., 2018] Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. TextWorld: A learning environment for text-based games. arXiv preprint arXiv:1806.11532, 2018.","title":"TextWorld: A learning environment for text-based games","authors":"Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, Adam Trischler","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doshi and Roy, 2008] Finale Doshi and Nicholas Roy. The permutable POMDP: Fast solutions to POMDPs for pref- erence elicitation. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems -Volume 1, page 493-500, 2008.","title":"Finale Doshi and Nicholas Roy. The permutable POMDP: Fast solutions to POMDPs for preference elicitation","authors":"Roy Doshi","year":"2008","venue":"Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hausknecht et al., 2020] Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interactive fiction games: A colossal adventure. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, 2020.","title":"Interactive fiction games: A colossal adventure","authors":"Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, Xingdi Yuan","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Honnibal et al., 2020] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in Python. https://spacy.io/, 2020. Accessed: 2022-01-14.","title":"spaCy: Industrial-strength natural language processing in Python","authors":"Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd","year":"2022","venue":null,"doi":null,"url":"https://spacy.io/,2020","internal_url":null},{"position":9,"raw":"[Hou et al., 2018] Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. Sequence-to-sequence data augmentation for dialogue language understanding. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 1234-1245, 2018.","title":"Sequence-to-sequence data augmentation for dialogue language understanding","authors":"Yutai Hou, Yijia Liu, Wanxiang Che, Ting Liu","year":"2018","venue":"Proceedings of the 27th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Laskin et al., 2020] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33, 2020.","title":"Reinforcement learning with augmented data","authors":"Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convo- lutional networks. In European semantic web conference, pages 593-607. Springer, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"European semantic web conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Trischler et al., 2019] Adam Trischler, Marc-Alexandre Côté, and Pedro Lima. First TextWorld problems, the com- petition: Using text-based games to advance capabilities of AI agents. https://www.microsoft.com/en-us/research/ blog/first-textworld-problems-the-competition-using- text-based-games-to-advance-capabilities-of-ai-agents/, 2019. Accessed: 2022-01-14.","title":"First TextWorld problems, the competition: Using text-based games to advance capabilities of AI agents","authors":"Adam Trischler, Marc-Alexandre Côté, Pedro Lima","year":"2022","venue":null,"doi":null,"url":"https://www.microsoft.com/en-us/research/blog/first-textworld-problems-the-competition-using-text-based-games-to-advance-capabilities-of-ai-agents/,2019","internal_url":null},{"position":14,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble Q-learning. In Proceedings of the AAAI conference on artificial intelligence, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yarats et al., 2020] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.","title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","authors":"Denis Yarats, Ilya Kostrikov, Rob Fergus","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}