{"type":"ScholarlyArticle","title":"Difficulty Controllable Generation of Reading Comprehension Questions","authors":[{"name":"Yifan Gao","slug":"yifan-gao","given":"Yifan","family":"Gao","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Shatin , N.T. , Hong Kong"},{"name":"Lidong Bing","slug":"lidong-bing","given":"Lidong","family":"Bing","affiliation":"R&D Center Singapore , Machine Intelligence Technology , Alibaba DAMO Academy 1"},{"name":"Wang Chen","slug":"wang-chen","given":"Wang","family":"Chen","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Shatin , N.T. , Hong Kong"},{"name":"Michael Lyu","slug":"michael-lyu","given":"Michael","family":"Lyu","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Shatin , N.T. , Hong Kong"},{"name":"Irwin King","slug":"irwin-king","given":"Irwin","family":"King","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , Shatin , N.T. , Hong Kong"}],"abstract":"We investigate the difficulty levels of questions in reading comprehension datasets such as SQuAD, and propose a new question generation setting, named Difficulty-controllable Question Generation (DQG). Taking as input a sentence in the reading comprehension paragraph and some of its text fragments (i.e., answers) that we want to ask questions about, a DQG method needs to generate questions each of which has a given text fragment as its answer, and meanwhile the generation is under the control of specified difficulty labels---the output questions should satisfy the specified difficulty as much as possible. To solve this task, we propose an end-to-end framework to generate questions of designated difficulty levels by exploring a few important intuitions. For evaluation, we prepared the first dataset of reading comprehension questions with difficulty labels. The results show that the question generated by our framework not only have better quality under the metrics like BLEU, but also comply with the specified difficulty labels.","doi":"10.24963/ijcai.2019/690","url":"https://ijcai.org/proceedings/2019/690","pdf":"https://ijcai.org/proceedings/2019/0690.pdf","markdown":"https://ijcai.org/proceedings/2019/690.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/690","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":690,"pages":{"first":4968,"last":4974},"keywords":["Natural Language Processing: Question Answering","Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2019] Wang Chen, Yifan Gao, Jiani Zhang, Ir- win King, and Michael R. Lyu. Title-guided encoding for keyphrase generation. In AAAI, 2019.","title":"Title-guided encoding for keyphrase generation","authors":"Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, Michael R Lyu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Danon and Last, 2017] Guy Danon and Mark Last. A syntactic approach to domain-specific automatic question generation. CoRR, abs/1712.09827, 2017.","title":"A syntactic approach to domain-specific automatic question generation","authors":"Last Danon, Guy Danon, Mark Last","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Du and Cardie, 2018] Xinya Du and Claire Cardie. Harvest- ing paragraph-level question-answer pairs from wikipedia. In ACL, 2018.","title":"Xinya Du and Claire Cardie. Harvesting paragraph-level question-answer pairs from wikipedia","authors":"Cardie Du","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Du et al., 2017] Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In ACL, 2017.","title":"Learning to ask: Neural question generation for reading comprehension","authors":"Du","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2019] Yifan Gao, Lidong Bing, Piji Li, Irwin King, and Michael R. Lyu. Generating distractors for read- ing comprehension questions from real examinations. In AAAI, 2019.","title":"Generating distractors for reading comprehension questions from real examinations","authors":"Yifan Gao, Lidong Bing, Piji Li, Irwin King, Michael R Lyu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heilman and Smith, 2010] Michael Heilman and Noah A. Smith. Good question! statistical ranking for question generation. In HLT-NAACL, 2010.","title":"Good question! statistical ranking for question generation","authors":"Smith ; Heilman, Michael Heilman, Noah A Smith","year":"2010","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kumar et al., 2018] Vishwajeet Kumar, Kireeti Boorla, Yo- gesh Meena, Ganesh Ramakrishnan, and Yuan-Fang Li. Automating reading comprehension by generating ques- tion and answer pairs. In PAKDD, 2018.","title":"Automating reading comprehension by generating question and answer pairs","authors":"Vishwajeet Kumar, Kireeti Boorla, Yogesh Meena, Ganesh Ramakrishnan, Yuan-Fang Li","year":"2018","venue":"PAKDD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Labutov et al., 2015] Igor Labutov, Sumit Basu, and Lucy Vanderwende. Deep questions without deep understand- ing. In ACL, 2015.","title":"Deep questions without deep understanding","authors":"Igor Labutov, Sumit Basu, Lucy Vanderwende","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai et al., 2017] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017.","title":"Race: Large-scale reading comprehension dataset from examinations","authors":"Lai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liao et al., 2018] Yi Liao, Lidong Bing, Piji Li, Shuming Shi, Wai Lam, and Tong Zhang. Quase: Sequence editing under quantifiable guidance. In EMNLP, 2018.","title":"Quase: Sequence editing under quantifiable guidance","authors":"Yi Liao, Lidong Bing, Piji Li, Shuming Shi, Wai Lam, Tong Zhang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lindberg et al., 2013] David Lindberg, Fred Popowich, John C. Nesbit, and Philip H. Winne. Generating natural language questions to support learning on-line. In ENLG, 2013.","title":"Generating natural language questions to support learning on-line","authors":"David Lindberg, Fred Popowich, John C Nesbit, Philip H Winne","year":"2013","venue":"ENLG","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mazidi and Nielsen, 2014] Karen Mazidi and Rodney D. Nielsen. Linguistic considerations in automatic question generation. In ACL, 2014.","title":"Linguistic considerations in automatic question generation","authors":"Karen Mazidi, Rodney D Nielsen","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. In ACL, 2016.","title":"Generating natural questions about an image","authors":"Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, Lucy Vanderwende","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy S. Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.","title":"Squad: 100, 000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy S Liang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[See et al., 2017] Abigail See, Peter J. Liu, and Christo- pher D. Manning. Get to the point: Summarization with pointer-generator networks. In ACL, 2017.","title":"Summarization with pointer-generator networks","authors":"See","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seo et al., 2017] Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. ICLR, 2017.","title":"Bidirectional attention flow for machine comprehension","authors":"Seo","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Serban et al., 2016] Iulian Serban, Alberto García-Durán, C ¸aglar Gülc ¸ehre, Sungjin Ahn, A. P. Sarath Chandar, Aaron C. Courville, and Yoshua Bengio. Generating fac- toid questions with recurrent neural networks: The 30m factoid question-answer corpus. In ACL, 2016.","title":"Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus","authors":"Iulian Serban, Alberto García-Durán, C ¸aglar Gülc ¸ehre, A P Sungjin Ahn, Aaron C Sarath Chandar, Yoshua Courville, Bengio","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2017] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non-parallel text by cross-alignment. In NIPS, pages 6833-6844, 2017.","title":"Style transfer from non-parallel text by cross-alignment","authors":"Shen","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2018] Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. Leveraging context information for natural question generation. In NAACL-HLT, 2018.","title":"Leveraging context information for natural question generation","authors":"Song","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Su et al., 2016] Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. On generating characteristic-rich question sets for qa evalua- tion. In EMNLP, 2016.","title":"On generating characteristic-rich question sets for qa evaluation","authors":"Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, Xifeng Yan","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sugawara et al., 2017] Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa. Evaluation metrics for machine reading comprehension: Prerequisite skills and readability. In ACL, 2017.","title":"Evaluation metrics for machine reading comprehension: Prerequisite skills and readability","authors":"Sugawara","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2018] Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. Answer-focused and position-aware neural question generation. In EMNLP, 2018.","title":"Answer-focused and position-aware neural question generation","authors":"Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, Shi Wang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vanderwende, 2007] Lucy Vanderwende. Answering and questioning for machine reading. In AAAI Spring Sym- posium: Machine Reading, 2007.","title":"Lucy Vanderwende. Answering and questioning for machine reading","authors":"Vanderwende","year":"2007","venue":"AAAI Spring Symposium: Machine Reading","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2017] Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching net- works for reading comprehension and question answering. In ACL, 2017.","title":"Gated self-matching networks for reading comprehension and question answering","authors":"Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, Ming Zhou","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2017] Xingdi Yuan, Tong Wang, C ¸aglar Gülc ¸ehre, Alessandro Sordoni, Philip Bachman, Sandeep Subramanian, Saizheng Zhang, and Adam Trischler. Machine comprehension by text-to-text neural question generation. In Rep4NLP@ACL, 2017.","title":"Machine comprehension by text-to-text neural question generation","authors":"Xingdi Yuan, Tong Wang, C ¸aglar Gülc ¸ehre, Alessandro Sordoni, Philip Bachman, Sandeep Subramanian, Saizheng Zhang, Adam Trischler","year":"2017","venue":"Rep4NLP@ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung. Gaan: Gated at- tention networks for learning on large and spatiotemporal graphs. In UAI, 2018.","title":"Gaan: Gated attention networks for learning on large and spatiotemporal graphs","authors":"Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, Dit-Yan Yeung","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2018] Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-level neural question gen- eration with maxout pointer and gated self-attention net- works. In ENNLP, 2018.","title":"Paragraph-level neural question generation with maxout pointer and gated self-attention networks","authors":"Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, Qifa Ke","year":"2018","venue":"ENNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. Neural ques- tion generation from text: A preliminary study. In NLPCC, 2017.","title":"Neural question generation from text: A preliminary study","authors":"Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, Ming Zhou","year":"2017","venue":"NLPCC","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}