{"type":"ScholarlyArticle","title":"Story Ending Prediction by Transferable BERT","authors":[{"name":"Zhongyang Li","slug":"zhongyang-li","given":"Zhongyang","family":"Li","affiliation":"Research Center for Social Computing and Information Retrieval , Harbin Institute of Technology; Johns Hopkins University."},{"name":"Xiao Ding","slug":"xiao-ding","given":"Xiao","family":"Ding","affiliation":"Research Center for Social Computing and Information Retrieval , Harbin Institute of Technology"},{"name":"Ting Liu","slug":"ting-liu","given":"Ting","family":"Liu","affiliation":"Research Center for Social Computing and Information Retrieval , Harbin Institute of Technology"}],"abstract":"Recent advances, such as GPT and BERT, have shown success in incorporating a pre-trained transformer language model and fine-tuning operation to improve downstream NLP systems. However, this framework still has some fundamental problems in effectively incorporating supervised knowledge from other related tasks. In this study, we investigate a transferable BERT (TransBERT) training framework, which can transfer not only general language knowledge from large-scale unlabeled data but also specific kinds of knowledge from various semantically related supervised tasks, for a target task. Particularly, we propose utilizing three kinds of transfer tasks, including natural language inference, sentiment classification, and next action prediction, to further train BERT based on a pre-trained model. This enables the model to get a better initialization for the target task. We take story ending prediction as the target task to conduct experiments. The final result, an accuracy of 91.8%, dramatically outperforms previous state-of-the-art baseline methods. Several comparative experiments give some helpful suggestions on how to select transfer tasks to improve BERT.","doi":"10.24963/ijcai.2019/249","url":"https://ijcai.org/proceedings/2019/249","pdf":"https://ijcai.org/proceedings/2019/0249.pdf","markdown":"https://ijcai.org/proceedings/2019/249.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/249","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Knowledge Representation and Reasoning","track":"Main track","number":249,"pages":{"first":1800,"last":1806},"keywords":["Machine Learning: Deep Learning","Knowledge Representation and Reasoning: Common-Sense Reasoning","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2017] Zheng Cai, Lifu Tu, and Kevin Gimpel. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL, pages 616-622, 2017.","title":"Pay attention to the ending: Strong neural baselines for the roc story cloze task","authors":"Zheng Cai, Lifu Tu, Kevin Gimpel","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaturvedi et al., 2017] Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. Story comprehension for predicting what happens next. In EMNLP, pages 1603-1614, 2017.","title":"Story comprehension for predicting what happens next","authors":"Snigdha Chaturvedi, Haoruo Peng, Dan Roth","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Jiaao Chen, Jianshu Chen, and Zhou Yu. Incorporating structured commonsense knowledge in story completion. AAAI, 2019.","title":"Incorporating structured commonsense knowledge in story completion","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu et al., 2018] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Ex- ploration and evaluation. In AAAI, 2018.","title":"Style transfer in text: Exploration and evaluation","authors":"Fu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Howard and Ruder, 2018] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text clas- sification. In ACL, volume 1, pages 328-339, 2018.","title":"Universal language model fine-tuning for text classification","authors":"Ruder Howard, Jeremy Howard, Sebastian Ruder","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2013] Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using click- through data. In CIKM, pages 2333-2338. ACM, 2013.","title":"Learning deep structured semantic models for web search using clickthrough data","authors":"Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, Larry Heck","year":"2013","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018a] Qian Li, Ziwei Li, Jin-Mao Wei, Yanhui Gu, Adam Jatowt, and Zhenglu Yang. A multi-attention based neural network with external knowledge for story ending predicting task. In Coling, pages 1754-1762, 2018.","title":"A multi-attention based neural network with external knowledge for story ending predicting task","authors":"Li","year":"2018","venue":"Coling","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018b] Zhongyang Li, Xiao Ding, and Ting Liu. Constructing narrative event evolutionary graph for script event prediction. In IJCAI, pages 4201-4207, 2018.","title":"Constructing narrative event evolutionary graph for script event prediction","authors":"Li","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/584"},{"position":11,"raw":"[Li et al., 2018c] Zhongyang Li, Xiao Ding, and Ting Liu. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Coling, pages 1033-1043. ACL, August 2018.","title":"Generating reasonable and diversified story ending using sequence to sequence model with adversarial training","authors":"Li","year":"2018","venue":"Coling","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2015] Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-yi Wang. Representation learning using multi-task deep neural networks for seman- tic classification and information retrieval. pages 912-921, 2015.","title":"Representation learning using multi-task deep neural networks for semantic classification and information retrieval","authors":"Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural net- works for natural language understanding. arXiv preprint arXiv:1901.11504, 2019.","title":"Multi-task deep neural networks for natural language understanding","authors":"Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maas et al., 2011] Andrew L Maas, Raymond E Daly, Pe- ter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150. ACL, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Raymond E Andrew L Maas, Peter T Daly, Dan Pham, Andrew Y Huang, Christopher Ng, Potts","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mo et al., 2018] Kaixiang Mo, Yu Zhang, Shuangyin Li, Ji- ajun Li, and Qiang Yang. Personalizing a dialogue system with transfer reinforcement learning. In AAAI, 2018.","title":"Personalizing a dialogue system with transfer reinforcement learning","authors":"Mo","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. NAACL, pages 740-750, 2016.","title":"A corpus and cloze evaluation for deeper understanding of commonsense stories","authors":"Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James Allen","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan and Yang, 2009] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 22(10):1345-1359, 2009.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2009","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peters et al., 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representa- tions. In NAACL, volume 1, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Phang et al., 2018] Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supple- mentary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018.","title":"Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks","authors":"Jason Phang, Thibault Févry, Samuel R Bowman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schwartz et al., 2017] Roy Schwartz, Maarten Sap, Ioan- nis Konstas, Leila Zilles, Yejin Choi, and Noah A Smith. Story cloze task: Uw nlp system. In LSDSem, pages 52- 55, 2017.","title":"Story cloze task: Uw nlp system","authors":"Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, Noah A Smith","year":"2017","venue":"LSDSem","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sharma et al., 2018] Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the story ending biases in the story cloze test. In ACL, vol- ume 2, pages 752-757, 2018.","title":"Tackling the story ending biases in the story cloze test","authors":"Rishi Sharma, James Allen, Omid Bakhshandeh, Nasrin Mostafazadeh","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017] Bingning Wang, Kang Liu, and Jun Zhao. Conditional generative adversarial networks for commonsense machine comprehension. In IJCAI, pages 4123-4129, 2017.","title":"Conditional generative adversarial networks for commonsense machine comprehension","authors":"Wang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/576"},{"position":25,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop, pages 353-355, 2018.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel Bowman","year":"2018","venue":"EMNLP Workshop","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams et al., 2018] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, vol- ume 1, pages 1112-1122, 2018.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel Bowman","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zellers et al., 2018] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP, pages 93-104, 2018.","title":"Swag: A large-scale adversarial dataset for grounded commonsense inference","authors":"Rowan Zellers, Yonatan Bisk, Roy Schwartz, Yejin Choi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2019] Mantong Zhou, Minlie Huang, and Xi- aoyan Zhu. Story ending selection by finding hints from pairwise candidate endings. TASLP, 2019.","title":"Story ending selection by finding hints from pairwise candidate endings","authors":"Zhou","year":"2019","venue":"TASLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}