{"type":"ScholarlyArticle","title":"Extracting Action Sequences from Texts Based on Deep Reinforcement Learning","authors":[{"name":"Wenfeng Feng","slug":"wenfeng-feng","given":"Wenfeng","family":"Feng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Hankz Hankui Zhuo","slug":"hankz-hankui-zhuo","given":"Hankz Hankui","family":"Zhuo","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Subbarao Kambhampati","slug":"subbarao-kambhampati","given":"Subbarao","family":"Kambhampati","affiliation":"Department of Computer Science and Engineering , Arizona State University , Tempe , Arizona , US"}],"abstract":"Extracting action sequences from texts is\r\nchallenging, as it  requires commonsense inferences based on world\r\nknowledge. Although there has been work on extracting action scripts,\r\ninstructions, navigation actions, etc., they require either the\r\nset of candidate actions be provided in advance, or action\r\ndescriptions are restricted to a specific form, e.g., description\r\ntemplates. In this paper we aim to extract action sequences from\r\ntexts in \\emph{free} natural language, i.e., without any restricted\r\ntemplates, provided the set of actions is unknown. We\r\npropose to extract action sequences from texts based on the deep\r\nreinforcement learning framework. Specifically, we view ``selecting''\r\nor ``eliminating'' words from texts as ``actions'', and texts associated with actions as ``states''. We build Q-networks to learn policies of extracting actions and extract plans from the labeled texts. We demonstrate the effectiveness of our approach on several datasets with comparison to state-of-the-art approaches.","doi":"10.24963/ijcai.2018/565","url":"https://ijcai.org/proceedings/2018/565","pdf":"https://ijcai.org/proceedings/2018/0565.pdf","markdown":"https://ijcai.org/proceedings/2018/565.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/565","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":565,"pages":{"first":4064,"last":4070},"keywords":["Planning and Scheduling: Activity and Plan Recognition","Planning and Scheduling: Planning with Incomplete information","Machine Learning: Deep Learning","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Branavan et al., 2009] S. R. K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In ACL, 2009.","title":"Reinforcement learning for mapping instructions to actions","authors":"S R K Branavan, Harr Chen, Luke S Zettlemoyer, Regina Barzilay","year":"2009","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Mooney, 2011] David L. Chen and Raymond J. Mooney. Learning to interpret natural language navigation instructions from observations. In AAAI, 2011.","title":"Learning to interpret natural language navigation instructions from observations","authors":"David L Chen, Raymond J Mooney","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cresswell et al., 2009] Stephen Cresswell, Thomas Leo McCluskey, and Margaret Mary West. Acquisition of object-centred domain models from planning examples. In ICAPS, 2009.","title":"Acquisition of object-centred domain models from planning examples","authors":"Stephen Cresswell, Thomas Leo Mccluskey, Margaret Mary West","year":"2009","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daniele et al., 2017] Andrea F Daniele, Mohit Bansal, and Matthew R Walter. Navigational instruction generation as inverse reinforcement learning with neural machine trans- lation. In HRI, 2017.","title":"Navigational instruction generation as inverse reinforcement learning with neural machine translation","authors":"Mohit Andrea F Daniele, Matthew R Bansal, Walter","year":"2017","venue":"HRI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kim and Mooney, 2013] Joohyun Kim and Raymond J. Mooney. Unsupervised pcfg induction for grounded lan- guage learning with highly ambiguous supervision. In EMNLP, 2013.","title":"Unsupervised pcfg induction for grounded language learning with highly ambiguous supervision","authors":"Mooney Kim, Joohyun Kim, Raymond J Mooney","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lindsay et al., 2017] Alan Lindsay, Jonathon Read, João F. Ferreira, Thomas Hayton, Julie Porteous, and Peter Gre- gory. Framer: Planning models from natural language ac- tion descriptions. In ICAPS, 2017.","title":"Framer: Planning models from natural language action descriptions","authors":"Alan Lindsay, Jonathon Read, João F Ferreira, Thomas Hayton, Julie Porteous, Peter Gregory","year":"2017","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard H. Hovy. End- to-end sequence labeling via bi-directional lstm-cnns-crf. In ACL, 2016.","title":"Endto-end sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy ; Ma, Xuezhe Ma, Eduard H Hovy","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Macmahon et al., 2006] Matt Macmahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In AAAI, 2006.","title":"Walk the talk: Connecting language, knowledge, and action in route instructions","authors":"Matt Macmahon, Brian Stankiewicz, Benjamin Kuipers","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mei et al., 2016] Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. Listen, attend, and walk: neural mapping of navigational instructions to action sequences. In AAAI, 2016.","title":"Listen, attend, and walk: neural mapping of navigational instructions to action sequences","authors":"Hongyuan Mei, Mohit Bansal, Matthew R Walter","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In NIPS, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] V Mnih, K Kavukcuoglu, D Silver, A. A. Rusu, J Veness, M. G. Bellemare, A Graves, M Ried- miller, A. K. Fidjeland, and G Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540):529-33, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih, Kavukcuoglu, A A Silver, Rusu, M G Veness, Bellemare, Graves, A K Riedmiller, G Fidjeland, Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Narasimhan et al., 2015] Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In EMNLP, 2015.","title":"Language understanding for text-based games using deep reinforcement learning","authors":"Karthik Narasimhan, Tejas D Kulkarni, Regina Barzilay","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pomarlan et al., 2017] Mihai Pomarlan, Sebastian Ko- ralewski, and Michael Beetz. From natural language in- structions to structured robot plans. In KI, 2017.","title":"From natural language instructions to structured robot plans","authors":"Mihai Pomarlan, Sebastian Koralewski, Michael Beetz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Reimers and Gurevych, 2017] Nils Reimers and Iryna Gurevych. Reporting score distributions makes a differ- ence: Performance study of lstm-networks for sequence tagging. In EMNLP, 2017.","title":"Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sil and Yates, 2011] Avirup Sil and Alexander Yates. Ex- tracting STRIPS representations of actions and events. In RANLP, 2011.","title":"Avirup Sil and Alexander Yates. Extracting STRIPS representations of actions and events","authors":"Yates Sil","year":"2011","venue":"RANLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sil et al., 2010] Avirup Sil, Fei Huang, and Alexander Yates. Extracting action and event semantics from web text. In AAAI, 2010.","title":"Extracting action and event semantics from web text","authors":"Sil","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2017] Jin Wang, Zhongyuan Wang, Dawei Zhang, and Jun Yan. Combining knowledge with deep convolutional neural networks for short text classification. In IJCAI, 2017.","title":"Combining knowledge with deep convolutional neural networks for short text classification","authors":"Jin Wang, Zhongyuan Wang, Dawei Zhang, Jun Yan","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/406"},{"position":19,"raw":"[Zhang and Wallace, 2015] Ye Zhang and Byron C. Wal- lace. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. CoRR, abs/1510.03820, 2015.","title":"A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification","authors":"Wallace ; ] Ye Zhang, Byron C Zhang, Wallace","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhuo and Kambhampati, 2017] Hankz Hankui Zhuo and Subbarao Kambhampati. Model-lite planning: Case-based vs. model-based approaches. Artif. Intell., 246:1-21, 2017.","title":"Hankz Hankui Zhuo and Subbarao Kambhampati. Model-lite planning: Case-based vs. model-based approaches","authors":"Kambhampati Zhuo","year":"2017","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhuo and Yang, 2014] Hankz Hankui Zhuo and Qiang Yang. Action-model acquisition for planning via transfer learning. Artif. Intell., 212:80-103, 2014.","title":"Action-model acquisition for planning via transfer learning","authors":"Yang ; ] Zhuo, Hankui Hankz, Qiang Zhuo, Yang","year":"2014","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhuo et al., 2014] Hankz Hankui Zhuo, Héctor Muñoz- Avila, and Qiang Yang. Learning hierarchical task network domains from partially observed plan traces. Artif. Intell., 212:134-157, 2014.","title":"Learning hierarchical task network domains from partially observed plan traces","authors":"Zhuo","year":"2014","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}