{"type":"ScholarlyArticle","title":"An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension","authors":[{"name":"Xin Liu","slug":"xin-liu","given":"Xin","family":"Liu","affiliation":"Xiamen University , Xiamen , China"},{"name":"Kai Liu","slug":"kai-liu","given":"Kai","family":"Liu","affiliation":"Baidu Inc. , Beijing , China"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"Xiaomi AI Lab , Xiaomi Inc. , Beijing , China"},{"name":"Jinsong Su","slug":"jinsong-su","given":"Jinsong","family":"Su","affiliation":"Xiamen University , Xiamen , China"},{"name":"Yubin Ge","slug":"yubin-ge","given":"Yubin","family":"Ge","affiliation":"University of Illinois at Urbana-Champaign , Urbana , IL 61801 , USA"},{"name":"Bin Wang","slug":"bin-wang","given":"Bin","family":"Wang","affiliation":"Xiaomi AI Lab , Xiaomi Inc. , Beijing , China"},{"name":"Jiebo Luo","slug":"jiebo-luo","given":"Jiebo","family":"Luo","affiliation":"Department of Computer Science , University of Rochester , Rochester NY 14627 , USA"}],"abstract":"The lack of sufficient training data in many domains, poses a major challenge to the construction of domain-specific machine reading comprehension (MRC) models with satisfying performance. In this paper, we propose a novel iterative multi-source mutual knowledge transfer framework for MRC. As an extension of the conventional knowledge transfer with one-to-one correspondence, our framework focuses on the many-to-many mutual transfer, which involves synchronous executions of multiple many-to-one transfers in an iterative manner.Specifically, to update a target-domain MRC model, we first consider other domain-specific MRC models as individual teachers, and employ knowledge distillation to train a multi-domain MRC model, which is differentially required to fit the training data and match the outputs of these individual models according to their domain-level similarities to the target domain. After being initialized by the multi-domain MRC model, the target-domain MRC model is fine-tuned to match both its training data and the output of its previous best model simultaneously via knowledge distillation. Compared with previous approaches, our framework can continuously enhance all domain-specific MRC models by enabling each model to iteratively and differentially absorb the domain-shared knowledge from others. Experimental results and in-depth analyses on several benchmark datasets demonstrate the effectiveness of our framework.","doi":"10.24963/ijcai.2020/525","url":"https://ijcai.org/proceedings/2020/525","pdf":"https://ijcai.org/proceedings/2020/0525.pdf","markdown":"https://ijcai.org/proceedings/2020/525.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/525","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":525,"pages":{"first":3794,"last":3800},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Question Answering"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24031/file_24031.m3u8","length_sec":260,"provider":"hls"}],"references":[{"position":1,"raw":"[Chu et al., 2017] Chenhui Chu, Raj Dabre, and Sadao Kurohashi. An empirical comparison of domain adapta- tion methods for neural machine translation. In ACL, 2017.","title":"An empirical comparison of domain adaptation methods for neural machine translation","authors":"Chu","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung et al., 2018] Yu-An Chung, Hung-yi Lee, and James R. Glass. Supervised and unsupervised transfer learning for question answering. In NAACL-HLT, 2018.","title":"Supervised and unsupervised transfer learning for question answering","authors":"Yu-An Chung, Hung-Yi Lee, James R Glass","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark et al., 2019] Kevin Clark, Minh-Thang Luong, Ur- vashi Khandelwal, Christopher D. Manning, and Quoc V. Le. Bam! born-again multi-task networks for natural lan- guage understanding. In ACL, 2019.","title":"Bam! born-again multi-task networks for natural language understanding","authors":"Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, V Quoc, Le","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2017] Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. Question generation for question answering. In EMNLP, 2017.","title":"Question generation for question answering","authors":"Nan Duan, Duyu Tang, Peng Chen, Ming Zhou","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Furlanello et al., 2018] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural net- works. In ICML, 2018.","title":"Born-again neural networks","authors":"Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Golub et al., 2017] David Golub, Po-Sen Huang, Xiaodong He, and Li Deng. Two-stage synthesis networks for trans- fer learning in machine comprehension. In EMNLP, 2017.","title":"Two-stage synthesis networks for transfer learning in machine comprehension","authors":"David Golub, Po-Sen Huang, Xiaodong He, Li Deng","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural net- work. CoRR, abs/1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey E Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2018] Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. In EMNLP, 2018.","title":"Attention-guided answer distillation for machine reading comprehension","authors":"Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Joshi et al., 2017] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading compre- hension. In ACL, 2017.","title":"Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension","authors":"Mandar Joshi, Eunsol Choi, Daniel S Weld, Luke Zettlemoyer","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kwiatkowski et al., 2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. TACL, 2019.","title":"Natural questions: a benchmark for question answering research","authors":"Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Kai Liu, Xin Liu, An Yang, Jing Liu, Jin- song Su, Sujian Li, and Qiaoqiao She. A robust adversarial training approach to machine reading comprehension. In AAAI, 2020.","title":"A robust adversarial training approach to machine reading comprehension","authors":"Kai Liu, Xin Liu, An Yang, Jing Liu, Jinsong Su, Sujian Li, Qiaoqiao She","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Min et al., 2017] Sewon Min, Min Joon Seo, and Hannaneh Hajishirzi. Question answering through transfer learning from large fine-grained supervision data. In ACL, 2017.","title":"Question answering through transfer learning from large fine-grained supervision data","authors":"Sewon Min, Min Joon Seo, Hannaneh Hajishirzi","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100, 000+ ques- tions for machine comprehension of text. In EMNLP, 2016.","title":"Squad: 100, 000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Seo et al., 2017] Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR, 2017.","title":"Bidirectional attention flow for machine comprehension","authors":"Seo","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2018] Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. Leveraging context information for natural question generation. In NAACL-HLT (Short Papers), 2018.","title":"Leveraging context information for natural question generation","authors":"Song","year":"2018","venue":"NAACL-HLT (Short Papers)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Su et al., 2019] Jinsong Su, Jiali Zeng, Jun Xie, Huating Wen, Yongjing Yin, and Yang Liu. Exploring discrimina- tive word-level domain contexts for multi-domain neural machine translation. TPAMI, 2019.","title":"Exploring discriminative word-level domain contexts for multi-domain neural machine translation","authors":"Jinsong Su, Jiali Zeng, Jun Xie, Huating Wen, Yongjing Yin, Yang Liu","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Talmor and Berant, 2019] Alon Talmor and Jonathan Be- rant. Multiqa: An empirical investigation of generalization and transfer in reading comprehension. In ACL, 2019.","title":"Multiqa: An empirical investigation of generalization and transfer in reading comprehension","authors":"Alon Talmor, Jonathan Berant","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tan et al., 2019] Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. In ICLR, 2019.","title":"Multilingual neural machine translation with knowledge distillation","authors":"Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Trischler et al., 2017] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In ACL, 2017.","title":"Newsqa: A machine comprehension dataset","authors":"Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, and Jianfeng Gao. Multi-task learning with sample re-weighting for machine reading comprehension. In NAACL-HLT, 2019.","title":"Multi-task learning with sample re-weighting for machine reading comprehension","authors":"Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, Jianfeng Gao","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.","title":"Hotpotqa: A dataset for diverse, explainable multi-hop question answering","authors":"Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, Christopher D Manning","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2018] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR, 2018.","title":"Qanet: Combining local convolution with global self-attention for reading comprehension","authors":"Adams Wei, Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, V Quoc, Le","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zeng et al., 2018] Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, and Jianqiang Zhao. Multi-domain neural machine translation with word-level domain context discrimination. In EMNLP, 2018.","title":"Multi-domain neural machine translation with word-level domain context discrimination","authors":"Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, Jianqiang Zhao","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeng et al., 2019] Jiali Zeng, Yang Liu, jinsong su, yubing Ge, Yaojie Lu, Yongjing Yin, and jiebo luo. Iterative dual domain adaptation for neural machine translation. In EMNLP, 2019.","title":"Iterative dual domain adaptation for neural machine translation","authors":"Jiali Zeng, Yang Liu, Yaojie Ge, Yongjing Lu, Yin","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}