{"type":"ScholarlyArticle","title":"AmazonQA: A Review-Based Question Answering Task","authors":[{"name":"Mansi Gupta","slug":"mansi-gupta","given":"Mansi","family":"Gupta","affiliation":"Carnegie Mellon University"},{"name":"Nitish Kulkarni","slug":"nitish-kulkarni","given":"Nitish","family":"Kulkarni","affiliation":"Carnegie Mellon University"},{"name":"Raghuveer Chanda","slug":"raghuveer-chanda","given":"Raghuveer","family":"Chanda","affiliation":"Carnegie Mellon University"},{"name":"Anirudha Rayasam","slug":"anirudha-rayasam","given":"Anirudha","family":"Rayasam","affiliation":"Carnegie Mellon University"},{"name":"Zachary C. Lipton","slug":"zachary-c-lipton","given":"Zachary C.","family":"Lipton","affiliation":"Carnegie Mellon University"}],"abstract":"Every day, thousands of customers post questions on Amazon product pages. After some time, if they are fortunate, a knowledgeable customer might answer their question. Observing that many questions can be answered based upon the available product reviews, we propose the task of review-based QA. Given a corpus of reviews and a question, the QA system synthesizes an answer. To this end, we introduce a new dataset and propose a method that combines informational retrieval techniques for selecting relevant reviews (given a question) and \"reading comprehension\" models for synthesizing an answer (given a question and review). Our dataset consists of 923k questions, 3.6M answers and 14M reviews across 156k products. Building on the well-known Amazon dataset, we additionally collect annotations marking each question as either answerable or unanswerable based on the available reviews. A deployed system could first classify a question as answerable before attempting to generate a provisional answer. Notably, unlike many popular QA datasets, here the questions, passages, and answers are extracted from real human interactions. We evaluate a number of models for answer generation and propose strong baselines, demonstrating the challenging nature of this new task.","doi":"10.24963/ijcai.2019/694","url":"https://ijcai.org/proceedings/2019/694","pdf":"https://ijcai.org/proceedings/2019/0694.pdf","markdown":"https://ijcai.org/proceedings/2019/694.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/694","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":694,"pages":{"first":4996,"last":5002},"keywords":["Natural Language Processing: Question Answering","Natural Language Processing: Natural Language Processing","Natural Language Processing: Resources and Evaluation","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Agresti and Coull, 1998] Alan Agresti and Brent A Coull. Approximate is better than \"exact\" for interval estima- tion of binomial proportions. The American Statistician, 52(2):119-126, 1998.","title":"Approximate is better than \"exact\" for interval estimation of binomial proportions","authors":"Coull Agresti, Alan Agresti, Brent A Coull","year":"1998","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075, 2015.","title":"Large-scale simple question answering with memory networks","authors":"Antoine Bordes, Nicolas Usunier, Sumit Chopra, Jason Weston","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark et al., 2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answer- ing? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.","title":"Think you have solved question answering? try arc, the ai2 reasoning challenge","authors":"Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dunn et al., 2017] Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017.","title":"Searchqa: A new q&a dataset augmented with context from a search engine","authors":"Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, Kyunghyun Cho","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Group, 2017] Natural Language Computing Group. R-net: Machine reading comprehension with self-matching net- works. May 2017.","title":"Natural Language Computing Group. R-net: Machine reading comprehension with self-matching networks","authors":null,"year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gururangan et al., 2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bow- man, and Noah A Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 107-112, 2018.","title":"Annotation artifacts in natural language inference data","authors":"Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, Noah A Smith","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He and Dai, 2011] Jing He and Decheng Dai. Summariza- tion of yes/no questions using a feature function model. In Asian Conference on Machine Learning, pages 351-366, 2011.","title":"Summarization of yes/no questions using a feature function model","authors":"Dai ; ] Jing He, Decheng He, Dai","year":"2011","venue":"Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2018] Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. Dureader: a chinese machine reading comprehension dataset from real-world applica- tions. In Proceedings of the Workshop on Machine Read- ing for Question Answering, pages 37-46, 2018.","title":"Dureader: a chinese machine reading comprehension dataset from real-world applications","authors":"Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She","year":"2018","venue":"Proceedings of the Workshop on Machine Reading for Question Answering","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaushik and Lipton, 2018] Divyansh Kaushik and Zachary C Lipton. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Empirical Methods in Natural Language Processing (EMNLP), 2018.","title":"How much reading does reading comprehension require? a critical investigation of popular benchmarks","authors":"Lipton Kaushik, Divyansh Kaushik, Zachary C Lipton","year":"2018","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kočiskỳ et al., 2018] Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317-328, 2018.","title":"The narrativeqa reading comprehension challenge","authors":"Kočiskỳ","year":"2018","venue":"Transactions of the Association of Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lai et al., 2017] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale read- ing comprehension dataset from examinations. In Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, 2017.","title":"Race: Large-scale reading comprehension dataset from examinations","authors":"Lai","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Lin, Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[McAuley and Yang, 2016] Julian McAuley and Alex Yang. Addressing complex and subjective product-related queries with customer reviews. In Proceedings of the 25th International Conference on World Wide Web, pages 625-635. International World Wide Web Conferences Steering Committee, 2016.","title":"Addressing complex and subjective product-related queries with customer reviews","authors":"Yang ; Mcauley, Julian Mcauley, Alex Yang","year":"2016","venue":"Proceedings of the 25th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nguyen et al., 2016] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. choice, 2640:660, 2016.","title":"Ms marco: A human generated machine reading comprehension dataset","authors":"Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng","year":"2016","venue":"choice","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rajpurkar et al., 2018] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswer- able questions for squad. In Proceedings of the 56th An- nual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), volume 2, pages 784- 789, 2018.","title":"Know what you don't know: Unanswerable questions for squad","authors":"Pranav Rajpurkar, Robin Jia, Percy Liang","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Robertson and Zaragoza, 2009] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333-389, April 2009.","title":"The probabilistic relevance framework: Bm25 and beyond","authors":"Stephen Robertson, Hugo Zaragoza","year":"2009","venue":"Found. Trends Inf. Retr","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Trischler et al., 2017] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. ACL 2017, page 191, 2017.","title":"Newsqa: A machine comprehension dataset","authors":"Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman","year":"2017","venue":"ACL 2017","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wan and McAuley, 2016] Mengting Wan and Julian McAuley. Modeling ambiguity, subjectivity, and diverg- ing viewpoints in opinion question answering systems. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, pages 489-498. IEEE, 2016.","title":"Mengting Wan and Julian McAuley. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems","authors":"Wan, Mcauley","year":"2016","venue":"IEEE 16th International Conference on","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2012] Jianxing Yu, Zheng-Jun Zha, and Tat-Seng Chua. Answering opinion questions on products by ex- ploiting hierarchical organization of consumer reviews. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computa- tional Natural Language Learning, pages 391-401. Asso- ciation for Computational Linguistics, 2012.","title":"Answering opinion questions on products by exploiting hierarchical organization of consumer reviews","authors":"Jianxing Yu, Zheng-Jun Zha, Tat-Seng Chua","year":"2012","venue":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}