{"type":"ScholarlyArticle","title":"Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisation","authors":[{"name":"Yang Gao","slug":"yang-gao","given":"Yang","family":"Gao","affiliation":"Dept. of Computer Science , Royal Holloway , University of London"},{"name":"Christian M. Meyer","slug":"christian-m-meyer","given":"Christian M.","family":"Meyer","affiliation":"Ubiquitous Knowledge Processing Lab (UKP-TUDA) , Technische Universität Darmstadt"},{"name":"Mohsen Mesgar","slug":"mohsen-mesgar","given":"Mohsen","family":"Mesgar","affiliation":"Ubiquitous Knowledge Processing Lab (UKP-TUDA) , Technische Universität Darmstadt"},{"name":"Iryna Gurevych","slug":"iryna-gurevych","given":"Iryna","family":"Gurevych","affiliation":"Ubiquitous Knowledge Processing Lab (UKP-TUDA) , Technische Universität Darmstadt"}],"abstract":"Document summarisation can be formulated as a sequential decision-making problem, which can be solved by Reinforcement Learning (RL) algorithms. The predominant RL paradigm for summarisation learns a cross-input policy, which requires considerable time, data and parameter tuning due to the huge search spaces and the delayed rewards. Learning input-specific RL policies is a more efficient alternative, but so far depends on handcrafted rewards, which are difficult to design and yield poor performance. We propose RELIS, a novel RL paradigm that learns a reward function with Learning-to-Rank (L2R) algorithms at training time and uses this reward function to train an input-specific RL policy at test time. We prove that RELIS guarantees to generate near-optimal summaries with appropriate L2R and RL algorithms. Empirically, we evaluate our approach on extractive multi-document summarisation. We show that RELIS reduces the training time by two orders of magnitude compared to the state-of-the-art models while performing on par with them.","doi":"10.24963/ijcai.2019/326","url":"https://ijcai.org/proceedings/2019/326","pdf":"https://ijcai.org/proceedings/2019/0326.pdf","markdown":"https://ijcai.org/proceedings/2019/326.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/326","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":326,"pages":{"first":2350,"last":2356},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Learning Preferences or Rankings","Machine Learning: Reinforcement Learning","Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Summarization"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learn- ing. In ICML, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal and Collins, 2010] Shivani Agarwal and Michael Collins. Maximum margin ranking algorithms for infor- mation retrieval. In ECIR, pages 332-343, 2010.","title":"Maximum margin ranking algorithms for information retrieval","authors":"Shivani Agarwal, Michael Collins","year":"2010","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bosselut et al., 2018] Antoine Bosselut, Asli C ¸elikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. In NAACL-HLT, pages 173-184, 2018.","title":"Discourse-aware neural rewards for coherent text generation","authors":"Antoine Bosselut, Xiaodong Asli C ¸elikyilmaz, Jianfeng He, Po-Sen Gao, Yejin Huang, Choi","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2015] Ziqiang Cao, Furu Wei, Sujian Li, Wenjie Li, Ming Zhou, and Houfeng Wang. Learning summary prior representation for extractive summarization. In ACL, pages 829-833, 2015.","title":"Learning summary prior representation for extractive summarization","authors":"Ziqiang Cao, Furu Wei, Sujian Li, Wenjie Li, Ming Zhou, Houfeng Wang","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2017] Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. Improving multi-document summarization via text classification. In AAAI, pages 3053-3059, 2017.","title":"Improving multi-document summarization via text classification","authors":"Ziqiang Cao, Wenjie Li, Sujian Li, Furu Wei","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaganty et al., 2018] Arun Tejasvi Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing au- tomatic metrics in natural language evalaution. In ACL, pages 643-653, 2018.","title":"The price of debiasing automatic metrics in natural language evalaution","authors":"Arun Tejasvi, Chaganty, Stephen Mussmann, Percy Liang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cheng et al., 2018] Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots. Fast policy learning through imitation and reinforcement. In UAI, pages 845-855, 2018.","title":"Fast policy learning through imitation and reinforcement","authors":"Ching-An Cheng, Xinyan Yan, Nolan Wagener, Byron Boots","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fazel et al., 2018] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gra- dient methods for the linear quadratic regulator. In ICML, pages 1466-1475, 2018.","title":"Global convergence of policy gradient methods for the linear quadratic regulator","authors":"Maryam Fazel, Rong Ge, Sham Kakade, Mehran Mesbahi","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2018] Yang Gao, Christian M. Meyer, and Iryna Gurevych. APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning. In EMNLP, pages 4120-4130, 2018.","title":"APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning","authors":"Yang Gao, Christian M Meyer, Iryna Gurevych","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gillick and Favre, 2009] Dan Gillick and Benoit Favre. A scalable global model for summarization. In ILP, pages 10-18. Association for Computational Linguistics, 2009.","title":"A scalable global model for summarization","authors":"Dan Gillick, Benoit Favre","year":"2009","venue":"ILP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ibarz et al., 2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In NeurIPS, pages 8022-8034, 2018.","title":"Reward learning from human preferences and demonstrations in atari","authors":"Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kreutzer et al., 2018] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In ACL, pages 1777-1788, 2018.","title":"Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning","authors":"Julia Kreutzer, Joshua Uyheng, Stefan Riezler","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kryscinski et al., 2018] Wojciech Kryscinski, Romain Paulus, Caiming Xiong, and Richard Socher. Improving abstraction in text summarization. In EMNLP, pages 1808-1817, 2018.","title":"Improving abstraction in text summarization","authors":"Wojciech Kryscinski, Romain Paulus, Caiming Xiong, Richard Socher","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li, 2011] Hang Li. A short introduction to learning to rank. IEICE Transactions, 94-D(10):1854-1862, 2011.","title":"A short introduction to learning to rank","authors":"Hang Li","year":"2011","venue":"IEICE Transactions","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In ACL Workshop \"Text Summarization Branches Out\", pages 74-81, 2004.","title":"ROUGE: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"ACL Workshop \"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maystre and Grossglauser, 2017] Lucas Maystre and Matthias Grossglauser. Just sort it! A simple and effective approach to active preference learning. In ICML, pages 2344-2353, 2017.","title":"Just sort it! A simple and effective approach to active preference learning","authors":"Lucas Maystre, Matthias Grossglauser","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Narayan et al., 2018] Shisha Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive sum- marization with reinforcement learning. In NAACL-HLT, pages 1747-1759, 2018.","title":"Ranking sentences for extractive summarization with reinforcement learning","authors":"Narayan","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2017] Khanh Nguyen, Hal Daumé III, and Jordan L. Boyd-Graber. Reinforcement learning for ban- dit neural machine translation with simulated human feed- back. In EMNLP, pages 1465-1475, 2017.","title":"Reinforcement learning for bandit neural machine translation with simulated human feedback","authors":"Khanh Nguyen, Hal Daumé, Iii, Jordan L Boyd-Graber","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peyrard and Eckle-Kohler, 2017] Maxime Peyrard and Ju- dith Eckle-Kohler. A principled framework for evaluat- ing summarizers: Comparing models of summary quality against human judgments. In ACL, pages 26-31, 2017.","title":"Maxime Peyrard and Judith Eckle-Kohler. A principled framework for evaluating summarizers: Comparing models of summary quality against human judgments","authors":"Eckle-Kohler Peyrard","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Peyrard and Gurevych, 2018] Maxime Peyrard and Iryna Gurevych. Objective function learning to match human judgements for optimization-based summarization. In NAACL-HLT, pages 654-660, 2018.","title":"Maxime Peyrard and Iryna Gurevych. Objective function learning to match human judgements for optimization-based summarization","authors":"Gurevych Peyrard","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ren et al., 2018] Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Liqiang Nie, Jun Ma, and Maarten de Ri- jke. Sentence relations for extractive summarization with deep neural networks. ACM Trans. Inf. Syst., 36(4):39:1- 39:32, 2018.","title":"Sentence relations for extractive summarization with deep neural networks","authors":"Ren","year":"2018","venue":"ACM Trans. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rioux et al., 2014] Cody Rioux, Sadid A. Hasan, and Yl- lias Chali. Fear the REAPER: A system for automatic multi-document summarization with reinforcement learn- ing. In EMNLP, pages 681-690, 2014.","title":"Fear the REAPER: A system for automatic multi-document summarization with reinforcement learning","authors":"Cody Rioux, Sadid A Hasan, Yllias Chali","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ryang and Abekawa, 2012] Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using reinforcement learning. In EMNLP/CoNLL, pages 256-265, 2012.","title":"Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using reinforcement learning","authors":"Abekawa Ryang","year":"2012","venue":"EMNLP/CoNLL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton et al., 1999] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton et al., 2009] Richard S. Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear func- tion approximation. In ICML, pages 993-1000, 2009.","title":"Fast gradient-descent methods for temporal-difference learning with linear function approximation","authors":"Richard S Sutton, Reza Hamid, Doina Maei, Shalabh Precup, David Bhatnagar, Csaba Silver, Eric Szepesvári, Wiewiora","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu and Hu, 2018] Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learn- ing. In AAAI, pages 5602-5609, 2018.","title":"Learning to extract coherent summary via deep reinforcement learning","authors":"Hu ; ] Yuxiang Wu, Baotian Wu, Hu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2018] Kaichun Yao, Libo Zhang, Tiejian Luo, and Yanjun Wu. Deep reinforcement learning for extrac- tive document summarization. Neurocomputing, 284:52- 62, 2018.","title":"Deep reinforcement learning for extractive document summarization","authors":"Kaichun Yao, Libo Zhang, Tiejian Luo, Yanjun Wu","year":"2018","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Lapata, 2017] Xingxing Zhang and Mirella La- pata. Sentence simplification with deep reinforcement learning. In EMNLP, pages 584-594, 2017.","title":"Sentence simplification with deep reinforcement learning","authors":"Lapata Zhang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2018] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In NeurIPS, pages 4649-4659, 2018.","title":"On learning intrinsic rewards for policy gradient methods","authors":"Zheng","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}