{"type":"ScholarlyArticle","title":"A Question Type Driven Framework to Diversify Visual Question Generation","authors":[{"name":"Zhihao Fan","slug":"zhihao-fan","given":"Zhihao","family":"Fan","affiliation":"School of Data Science , Fudan University , China"},{"name":"Zhongyu Wei","slug":"zhongyu-wei","given":"Zhongyu","family":"Wei","affiliation":"School of Data Science , Fudan University , China"},{"name":"Piji Li","slug":"piji-li","given":"Piji","family":"Li","affiliation":"Department of Systems Engineering and Engineering Management , The Chinese University of Hong Kong , Hong Kong"},{"name":"Yanyan Lan","slug":"yanyan-lan","given":"Yanyan","family":"Lan","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China; School of Data Science , Fudan University , China"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"School of Computer Science , Fudan University , China"}],"abstract":"Visual question generation aims at asking questions about an image automatically. Existing research works on this topic usually generate a single question for each given image without considering the issue of diversity. In this paper, we propose a question type driven framework to produce multiple questions for a given image with different focuses. In our framework, each question is constructed following the guidance of a sampled question type in a sequence-to-sequence fashion. To diversify the generated questions, a novel conditional variational auto-encoder is introduced to generate multiple questions with a specific question type. Moreover, we design a strategy to conduct the question type distribution learning for each image to select the final questions. Experimental results on three benchmark datasets show that our framework outperforms the state-of-the-art approaches in terms of both relevance and diversity.","doi":"10.24963/ijcai.2018/563","url":"https://ijcai.org/proceedings/2018/563","pdf":"https://ijcai.org/proceedings/2018/0563.pdf","markdown":"https://ijcai.org/proceedings/2018/563.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/563","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":563,"pages":{"first":4048,"last":4054},"keywords":["Natural Language Processing: Natural Language Generation","Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowman et al., 2016] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of CoNLL, pages 10-21, 2016.","title":"Generating sentences from a continuous space","authors":"Luke Samuel R Bowman, Oriol Vilnis, Andrew M Vinyals, Rafal Dai, Samy Jozefowicz, Bengio","year":"2016","venue":"Proceedings of CoNLL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Denkowski and Lavie, 2014] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014.","title":"Meteor universal: Language specific translation evaluation for any target language","authors":"Lavie Denkowski, Michael Denkowski, Alon Lavie","year":"2014","venue":"Proceedings of the EACL 2014 Workshop on Statistical Machine Translation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graesser et al., 2008] Art Graesser, Vasile Rus, and Zhiqiang Cai. Question classification schemes. In Proceedings of first Workshop on Question Generation, 2008.","title":"Question classification schemes","authors":"Graesser","year":"2008","venue":"Proceedings of first Workshop on Question Generation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gregor et al., 2015] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In Proceedings of ICML, pages 1462-1471, 2015.","title":"Draw: A recurrent neural network for image generation","authors":"Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra","year":"2015","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jain et al., 2017] Unnat Jain, Ziyu Zhang, and Alexander G. Schwing. Creativity: Generating diverse questions using variational autoencoders. In Proceedings of IEEE CVPR, pages 6485-6494, 2017.","title":"Creativity: Generating diverse questions using variational autoencoders","authors":"Jain","year":"2017","venue":"Proceedings of IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kunichika et al., 2004] Hidenobu Kunichika, Tomoki Katayama, Tsukasa Hirashima, and Akira Takeuchi. Automated question generation methods for intelli- gent english learning systems and its evaluation. In Proceedings of ICCE, 2004.","title":"Automated question generation methods for intelligent english learning systems and its evaluation","authors":"Hidenobu Kunichika, Tomoki Katayama, Tsukasa Hirashima, Akira Takeuchi","year":"2004","venue":"Proceedings of ICCE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. Deep recurrent generative decoder for abstractive text summarization. In Proceedings of EMNLP, pages 2091-2100, 2017.","title":"Deep recurrent generative decoder for abstractive text summarization","authors":"Li","year":"2017","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Proceedings of ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for auto- matic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, vol- ume 8. Barcelona, Spain, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out: Proceedings of the ACL-04 workshop","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mostafazadeh et al., 2016] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. In Proceedings of the 54th ACL, pages 1802-1813. Association for Computational Linguistics, 2016.","title":"Generating natural questions about an image","authors":"Mostafazadeh","year":"2016","venue":"Proceedings of the 54th ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mostafazadeh et al., 2017] Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P. Spithourakis, and Lucy Vanderwende. Image- grounded conversations: Multimodal context for natural question and response generation. In Proceedings of 8th IJCNLP, 2017.","title":"Vanderwende. Imagegrounded conversations: Multimodal context for natural question and response generation","authors":"Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P Spithourakis, Lucy","year":"2017","venue":"Proceedings of 8th IJCNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th ACL, pages 311-318. Association for Compu- tational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2015] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question an- swering. In Proceedings of NIPS, pages 2953-2961, 2015.","title":"Exploring models and data for image question answering","authors":"Ren","year":"2015","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rezende et al., 2014] Danilo Jimenez Rezende, Shakir Mo- hamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th ICML, pages 1278-1286, 2014.","title":"Stochastic backpropagation and approximate inference in deep generative models","authors":"Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra","year":"2014","venue":"Proceedings of the 31th ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shijie et al., 2017] Zhang Shijie, Qu Lizhen, You Shaodi, Yang Zhenglu, and Zhang Jiawan. Automatic generation of grounded visual questions. In Proceedings of the 26th IJCAI, pages 4235-4243, 2017.","title":"Automatic generation of grounded visual questions","authors":"Zhang Shijie, You Qu Lizhen, Yang Shaodi, Zhang Zhenglu, Jiawan","year":"2017","venue":"Proceedings of the 26th IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/592"},{"position":19,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vijayakumar et al., 2018] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. In Proceedings of AAAI, 2018.","title":"Diverse beam search: Decoding diverse solutions from neural sequence models","authors":"K Ashwin, Michael Vijayakumar, Cogswell, Qing Ramprasath R Selvaraju, Stefan Sun, David Lee, Dhruv Crandall, Batra","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In Proceedings of the IEEE CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Zhuhao Wang, Fei Wu, Weiming Lu, Jun Xiao, Xi Li, Zitong Zhang, and Yueting Zhuang. Diverse image captioning via grouptalk. In IJCAI, pages 2957- 2964, 2016.","title":"Diverse image captioning via grouptalk","authors":"Wang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/420"},{"position":23,"raw":"[Wu et al., 2017] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. TACL, 5:339-351, 2017.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu et al., 2016] Yuke Zhu, Oliver Groth, Michael Bern- stein, and Li Fei-Fei. Visual7w: Grounded question an- swering in images. In Proceedings of the IEEE CVPR, pages 4995-5004, 2016.","title":"Visual7w: Grounded question answering in images","authors":"Yuke Zhu, Oliver Groth, Michael Bernstein, Li Fei-Fei","year":"2016","venue":"Proceedings of the IEEE CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}