{"type":"ScholarlyArticle","title":"Answering Binary Causal Questions Through Large-Scale Text Mining: An Evaluation Using Cause-Effect Pairs from Human Experts","authors":[{"name":"Oktie Hassanzadeh","slug":"oktie-hassanzadeh","given":"Oktie","family":"Hassanzadeh","affiliation":"IBM Research"},{"name":"Debarun Bhattacharjya","slug":"debarun-bhattacharjya","given":"Debarun","family":"Bhattacharjya","affiliation":"IBM Research"},{"name":"Mark Feblowitz","slug":"mark-feblowitz","given":"Mark","family":"Feblowitz","affiliation":"IBM Research"},{"name":"Kavitha Srinivas","slug":"kavitha-srinivas","given":"Kavitha","family":"Srinivas","affiliation":"IBM Research"},{"name":"Michael Perrone","slug":"michael-perrone","given":"Michael","family":"Perrone","affiliation":"IBM Research"},{"name":"Shirin Sohrabi","slug":"shirin-sohrabi","given":"Shirin","family":"Sohrabi","affiliation":"IBM Research"},{"name":"Michael Katz","slug":"michael-katz","given":"Michael","family":"Katz","affiliation":"IBM Research"}],"abstract":"In this paper, we study the problem of answering questions of type \"Could X cause Y?\" where X and Y are general phrases without any constraints. Answering such questions will assist with various decision analysis tasks such as verifying and extending presumed causal associations used for decision making. Our goal is to analyze the ability of an AI agent built using state-of-the-art unsupervised methods in answering causal questions derived from collections of cause-effect pairs from human experts. We focus only on unsupervised and weakly supervised methods due to the difficulty of creating a large enough training set with a reasonable quality and coverage. The methods we examine rely on a large corpus of text derived from news articles, and include methods ranging from large-scale application of classic NLP techniques and statistical analysis to the use of neural network based phrase embeddings and state-of-the-art neural language models.","doi":"10.24963/ijcai.2019/695","url":"https://ijcai.org/proceedings/2019/695","pdf":"https://ijcai.org/proceedings/2019/0695.pdf","markdown":"https://ijcai.org/proceedings/2019/695.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/695","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":695,"pages":{"first":5003,"last":5009},"keywords":["Natural Language Processing: Knowledge Extraction","Natural Language Processing: NLP Applications and Tools"],"videos":[],"references":[{"position":1,"raw":"[Asghar, 2016] N. Asghar. Automatic extraction of causal relations from natural language texts: A comprehensive survey. CoRR, abs/1605.07895, 2016.","title":"Automatic extraction of causal relations from natural language texts: A comprehensive survey","authors":"N Asghar","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Y Bengio, R Ducharme, P Vincent, C Janvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chambers and Jurafsky, 2008] N. Chambers and D. Juraf- sky. Unsupervised learning of narrative event chains. In ACL, 2008.","title":"Unsupervised learning of narrative event chains","authors":"N Chambers, D Jurafsky","year":"2008","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Church and Hank, 1990] K. W. Church and P. Hank. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29, 1990.","title":"Word association norms, mutual information, and lexicography","authors":"K W Church, P Hank","year":"1990","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dasgupta et al., 2018] T. Dasgupta, R. Saha, L. Dey, and A. Naskar. Automatic extraction of causal relations from text using linguistically informed deep neural networks. In SIGDIAL, 2018.","title":"Automatic extraction of causal relations from text using linguistically informed deep neural networks","authors":"T Dasgupta, R Saha, L Dey, A Naskar","year":"2018","venue":"SIGDIAL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2018] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirec- tional transformers for language understanding. CoRR, abs/1810.04805, 2018.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"J Devlin, M Chang, K Lee, K Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Do et al., 2011] Q. Do, Y. Seng Chan, and D. Roth. Min- imally supervised event causality identification. In EMNLP, 2011.","title":"Minimally supervised event causality identification","authors":"Q Do, Y Seng Chan, D Roth","year":"2011","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girju and Moldovan, 2002] R. Girju and D. I. Moldovan. Text mining for causal relations. In FLAIRS, 2002.","title":"Text mining for causal relations","authors":"R Girju, D I Moldovan","year":"2002","venue":"FLAIRS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Girju, 2003] R. Girju. Automatic detection of causal rela- tions for question answering. In MultiSumQA, 2003.","title":"Automatic detection of causal relations for question answering","authors":"R Girju","year":"2003","venue":"MultiSumQA","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gordon et al., 2011] A. S. Gordon, C. A. Bejan, and K. Sagae. Commonsense causal reasoning using millions of personal stories. In AAAI, 2011.","title":"Commonsense causal reasoning using millions of personal stories","authors":"A S Gordon, C A Bejan, K Sagae","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Halpern, 2016] J. Y. Halpern. Actual Causality. The MIT Press, 2016.","title":"Actual Causality","authors":"J Y Halpern","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hashimoto et al., 2014] C. Hashimoto, K. Torisawa, J. Kloetzer, M. Sano, I. Varga, J.-H. Oh, and Y. Ki- dawara. Toward future scenario generation: Extracting event causality exploiting semantic relation, context, and association features. In ACL, 2014.","title":"Toward future scenario generation: Extracting event causality exploiting semantic relation, context, and association features","authors":"C Hashimoto, K Torisawa, J Kloetzer, M Sano, I Varga, J.-H Oh, Y Kidawara","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hassanzadeh et al., 2018] O. Hassanzadeh, S. Trewin, and A. Gliozzo. Semantic concept discovery over event databases. In ESWC, 2018.","title":"Semantic concept discovery over event databases","authors":"O Hassanzadeh, S Trewin, A Gliozzo","year":"2018","venue":"ESWC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hassanzadeh et al., 2019] O. Hassanzadeh, D. Bhattachar- jya, M. Feblowitz, K. Srinivas, M. Perrone, S. Sohrabi, and M. Katz. Data sets of cause-effect pairs, May 2019. https://doi.org/10.5281/zenodo.3214925.","title":"Data sets of cause-effect pairs","authors":"O Hassanzadeh, D Bhattacharjya, M Feblowitz, K Srinivas, M Perrone, S Sohrabi, M Katz","year":"2019","venue":null,"doi":"10.5281/zenodo.3214925","url":"https://doi.org/10.5281/zenodo.3214925","internal_url":null},{"position":15,"raw":"[Hendrickx et al., 2010] I.","title":null,"authors":"Hendrickx","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Hendrickx, S. Kim, Z. Kozareva, P. Nakov, D. Ó. Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, and S. Szpakowicz. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In SemEval, 2010.","title":"SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals","authors":"S Hendrickx, Z Kim, P Kozareva, D Ó Nakov, S Séaghdha, M Padó, L Pennacchiotti, S Romano, Szpakowicz","year":"2010","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khoo et al., 2000] C. Khoo, S. Chan, and Y. Niu. Extracting causal knowledge from a medical database using graphical patterns. In ACL, 2000. [Kruengkrai et al., 2017]","title":"Extracting causal knowledge from a medical database using graphical patterns","authors":"C Khoo, S Chan, Y Niu","year":"2000","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"C. Kruengkrai, K. Torisawa, C. Hashimoto, J. Kloetzer, J.-H. Oh, and M. Tanaka. Improving event causality recognition with multiple background knowledge sources using multi-column convolutional neural networks. In AAAI, 2017.","title":"Improving event causality recognition with multiple background knowledge sources using multi-column convolutional neural networks","authors":"C Kruengkrai, K Torisawa, C Hashimoto, J Kloetzer, J.-H Oh, M Tanaka","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leban et al., 2014] G. Leban, B. Fortuna, J. Brank, and M. Grobelnik. Event Registry: Learning about world events from news. In WWW, 2014.","title":"Event Registry: Learning about world events from news","authors":"G Leban, B Fortuna, J Brank, M Grobelnik","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2016] Z. Luo, Y. Sha, K. Q. Zhu, S. Hwang, and Z. Wang. Commonsense causal reasoning between short texts. In KR, 2016.","title":"Commonsense causal reasoning between short texts","authors":"Z Luo, Y Sha, K Q Zhu, S Hwang, Z Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mikolov et al., 2013] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed Representations of Words and Phrases and Their Compositionality. In NIPS, 2013.","title":"Distributed Representations of Words and Phrases and Their Compositionality","authors":"T Mikolov, I Sutskever, K Chen, G S Corrado, J Dean","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[NATO-SFA, 2017] Strategic Foresight Analysis 2017 Re- port. https://www.act.nato.int/publications-ffao, 2017. [Online; accessed February 21, 2019].","title":"Strategic Foresight Analysis 2017 Report","authors":null,"year":"2017","venue":null,"doi":null,"url":"https://www.act.nato.int/publications-ffao,2017","internal_url":null},{"position":23,"raw":"[Pearl, 2009] J. Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2009.","title":"Causality: Models, Reasoning and Inference","authors":"J Pearl","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Radinsky and Horvitz, 2013] K. Radinsky and E. Horvitz. Mining the web to predict future events. In WSDM, 2013.","title":"Mining the web to predict future events","authors":"K Radinsky, E Horvitz","year":"2013","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Radinsky et al., 2012] K. Radinsky, S. Davidovich, and S. Markovitch. Learning to predict from textual data. J. Artif. Intell. Res., 45:641-684, 2012.","title":"Learning to predict from textual data","authors":"K Radinsky, S Davidovich, S Markovitch","year":"2012","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Riaz and Girju, 2010] M. Riaz and R. Girju. Another look at causality: Discovering scenario-specific contingency re- lationships with no supervision. In ICSC, 2010.","title":"Another look at causality: Discovering scenario-specific contingency relationships with no supervision","authors":"M Riaz, R Girju","year":"2010","venue":"ICSC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sap et al., 2019] M. Sap, R. LeBras, E. Allaway, C. Bhaga- vatula, N. Lourie, H. Rashkin, B. Roof, N. A. Smith, and Y. Choi. ATOMIC: An atlas of machine commonsense for if-then reasoning. In AAAI, 2019.","title":"ATOMIC: An atlas of machine commonsense for if-then reasoning","authors":"M Sap, R Lebras, E Allaway, C Bhagavatula, N Lourie, H Rashkin, B Roof, N A Smith, Y Choi","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sharp et al., 2016] R. Sharp, M. Surdeanu, P. Jansen, P. Clark, and M. Hammond. Creating causal embed- dings for question answering with minimal supervision. In EMNLP, 2016.","title":"Creating causal embeddings for question answering with minimal supervision","authors":"R Sharp, M Surdeanu, P Jansen, P Clark, M Hammond","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sohrabi et al., 2018a] S. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, and M. D. Feblowitz. IBM scenario planning advisor: Plan recognition as AI planning in practice. In IJCAI, 2018.","title":"IBM scenario planning advisor: Plan recognition as AI planning in practice","authors":"Sohrabi","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/864"},{"position":30,"raw":"[Sohrabi et al., 2018b] S. Sohrabi, A. V. Riabov, M. Katz, and O. Udrea. An AI planning solution to scenario gen- eration for enterprise risk management. In AAAI, 2018.","title":"An AI planning solution to scenario generation for enterprise risk management","authors":"Sohrabi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sohrabi et al., 2019] S. Sohrabi, M. Katz, O. Hassanzadeh, O. Udrea, M. D. Feblowitz, and A. Riabov. IBM scenario planning advisor: Plan recognition as AI planning in prac- tice. AI Commun., 32(1):1-13, 2019.","title":"IBM scenario planning advisor: Plan recognition as AI planning in practice","authors":"S Sohrabi, M Katz, O Hassanzadeh, O Udrea, M D Feblowitz, A Riabov","year":"2019","venue":"AI Commun","doi":null,"url":null,"internal_url":"/proceedings/2018/864"},{"position":32,"raw":"[Tetlock and Gardner, 2015] P. E. Tetlock and D. Gardner. Superforecasting: The Art and Science of Prediction. Crown Publishing Group, New York, USA, 2015. [Wikipedia, 2019] https://en.wikipedia.org/wiki/ List of concept-and mind-mapping software, 2019. [Online; accessed February 21, 2019].","title":"Superforecasting: The Art and Science of Prediction","authors":"P E Tetlock, D Gardner","year":"2015","venue":null,"doi":null,"url":"https://en.wikipedia.org/wiki/Listofconcept-andmind-mappingsoftware,2019","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}