{"type":"ScholarlyArticle","title":"Sequence Prediction with Unlabeled Data by Reward Function Learning","authors":[{"name":"Lijun Wu","slug":"lijun-wu","given":"Lijun","family":"Wu","affiliation":"School of Data and Computer Science , Sun Yat-sen University"},{"name":"Li Zhao","slug":"li-zhao","given":"Li","family":"Zhao","affiliation":"Microsoft Research Asia"},{"name":"Tao Qin","slug":"tao-qin","given":"Tao","family":"Qin","affiliation":"Microsoft Research Asia"},{"name":"Jianhuang Lai","slug":"jianhuang-lai","given":"Jianhuang","family":"Lai","affiliation":"School of Data and Computer Science , Sun Yat-sen University; Guangdong Key Laboratory of Information Security Technology"},{"name":"Tie-Yan Liu","slug":"tie-yan-liu","given":"Tie-Yan","family":"Liu","affiliation":"Microsoft Research Asia"}],"abstract":"Reinforcement learning (RL), which has been successfully applied to sequence prediction, introduces \\textit{reward} as sequence-level supervision signal to evaluate the quality of a generated sequence. Existing RL approaches use the ground-truth sequence to define reward, which limits the application of RL techniques to labeled data. Since labeled data is usually scarce and/or costly to collect, it is desirable to leverage large-scale unlabeled data. In this paper, we extend existing RL methods for sequence prediction to exploit unlabeled data. We propose to learn the reward function from labeled data and use the predicted reward as \\textit{pseudo reward} for unlabeled data so that we can learn from unlabeled data using the pseudo reward. To get good pseudo reward on unlabeled data, we propose a RNN-based reward network with attention mechanism, trained with purposely biased data distribution.\r\nExperiments show that the pseudo reward can provide good supervision and guide the learning process on unlabeled data. We observe significant improvements on both neural machine translation and text summarization.","doi":"10.24963/ijcai.2017/432","url":"https://ijcai.org/proceedings/2017/432","pdf":"https://ijcai.org/proceedings/2017/0432.pdf","markdown":"https://ijcai.org/proceedings/2017/432.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/432","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":432,"pages":{"first":3098,"last":3104},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Machine Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2016] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor- critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086, 2016.","title":"An actorcritic algorithm for sequence prediction","authors":"Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cettolo et al., 2014] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014. In Pro- ceedings of the International Workshop on Spoken Lan- guage Translation, Hanoi, Vietnam, 2014.","title":"Report on the 11th iwslt evaluation campaign, iwslt 2014","authors":"Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Marcello Federico","year":"2014","venue":"Proceedings of the International Workshop on Spoken Language Translation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cettolo et al., 2015] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. The iwslt 2015 evaluation campaign. Proc. of IWSLT, Da Nang, Vietnam, 2015.","title":"The iwslt 2015 evaluation campaign","authors":"Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, Marcello Federico","year":"2015","venue":"Proc. of IWSLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2016] Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi- supervised learning for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, August 2016. Association for Computa- tional Linguistics.","title":"Semisupervised learning for neural machine translation","authors":"Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing (EMNLP), Doha, Qatar, October 2014. Association for Computational Linguistics.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Collobert et al., 2011] R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for ma- chine learning. In BigLearn, NIPS Workshop, 2011.","title":"Torch7: A matlab-like environment for machine learning","authors":"R Collobert, K Kavukcuoglu, C Farabet","year":"2011","venue":"BigLearn, NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai and Le, 2015] Andrew M Dai and Quoc V Le. Semi- supervised sequence learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 2015.","title":"Semisupervised sequence learning","authors":"M Andrew, Quoc V Dai, Le","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daumé et al., 2009] Hal Daumé, Iii, John Langford, and Daniel Marcu. Search-based structured prediction. Mach. Learn., 75(3), June 2009.","title":"Hal Daumé, Iii, John Langford, and Daniel Marcu. Search-based structured prediction","authors":"Daumé","year":"2009","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graff and Cieri, 2003] David Graff and C Cieri. English gi- gaword, linguistic data consortium, 2003.","title":"English gigaword, linguistic data consortium","authors":"David Graff, C Cieri","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29.","title":"Dual learning for machine translation","authors":"Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, Wei-Ying Ma","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8), November 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for auto- matic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, vol- ume 8. Barcelona, Spain, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Lin, Chin-Yew Lin","year":"2004","venue":"Text summarization branches out: Proceedings of the ACL-04 workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ng and Russell, 2000] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Pro- ceedings of the Seventeenth International Conference on Machine Learning, ICML '00, San Francisco, CA, USA, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Andrew Y Ng, Stuart J Russell","year":"2000","venue":"Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computa- tional linguistics. Association for Computational Linguis- tics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rush et al., 2015] Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for ab- stractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Sumit Alexander M Rush, Jason Chopra, Weston","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Berlin, Germany, Au- gust 2016. Association for Computational Linguistics.","title":"Improving neural machine translation models with monolingual data","authors":"Sennrich","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2016] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Mini- mum risk training for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, August 2016. Association for Computa- tional Linguistics.","title":"Minimum risk training for neural machine translation","authors":"Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Confer- ence on Neural Information Processing Systems, NIPS'14, Cambridge, MA, USA, 2014. MIT Press.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS'14","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Van Merriënboer et al., 2015] Bart Van Merriënboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-Farley, Jan Chorowski, and Yoshua Bengio. Blocks and fuel: Frameworks for deep learning. arXiv preprint arXiv:1506.00619, 2015.","title":"Blocks and fuel: Frameworks for deep learning","authors":"Bart Van Merriënboer, Dzmitry Bahdanau, Dmitriy Vincent Dumoulin, David Serdyuk, Jan Warde-Farley, Yoshua Chorowski, Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Mach. Learn., 8(3-4), May 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Zong, 2016] Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural ma- chine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, November 2016. Association for Computa- tional Linguistics.","title":"Exploiting source-side monolingual data in neural machine translation","authors":"Jiajun Zhang, Chengqing Zong","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}