{"type":"ScholarlyArticle","title":"Predicting the Quality of Short Narratives from Social Media","authors":[{"name":"Tong Wang","slug":"tong-wang","given":"Tong","family":"Wang","affiliation":"University of Massachusetts Boston; Disney Research"},{"name":"Ping Chen","slug":"ping-chen","given":"Ping","family":"Chen","affiliation":"University of Massachusetts Boston"},{"name":"Boyang Li","slug":"boyang-li","given":"Boyang","family":"Li","affiliation":"Disney Research"}],"abstract":"An important and difficult challenge in building computational models for narratives is the automatic evaluation of narrative quality. Quality evaluation connects narrative understanding and generation as generation systems need to evaluate their own products. To circumvent difficulties in acquiring annotations, we employ upvotes in social media as an approximate measure for story quality. We collected 54,484 answers from a crowd-powered question-and-answer website, Quora, and then used active learning to build a classifier that labeled 28,320 answers as stories. To predict the number of upvotes without the use of social network features, we create neural networks that model textual regions and the interdependence among regions, which serve as strong benchmarks for future research. To our best knowledge, this is the first large-scale study for automatic evaluation of narrative quality.","doi":"10.24963/ijcai.2017/539","url":"https://ijcai.org/proceedings/2017/539","pdf":"https://ijcai.org/proceedings/2017/0539.pdf","markdown":"https://ijcai.org/proceedings/2017/539.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/539","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":539,"pages":{"first":3859,"last":3865},"keywords":["Multidisciplinary Topics and Applications: Interactive Entertainment","Natural Language Processing: NLP Applications and Tools","Multidisciplinary Topics and Applications: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Alikaniotis et al., 2016] Dimitrios Alikaniotis, Helen Yan- nakoudakis, and Marek Rei. Automatic text scoring us- ing neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016.","title":"Automatic text scoring using neural networks","authors":"Dimitrios Alikaniotis, Helen Yannakoudakis, Marek Rei","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bamman et al., 2013] David Bamman, Brendan O'Connor, and Noah A. Smith. Learning latent personas of film char- acters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013.","title":"Learning latent personas of film characters","authors":"David Bamman, O' Brendan, Noah A Connor, Smith","year":"2013","venue":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barzilay and Lapata, 2008] Regina Barzilay and Mirella Lapata. Modeling local coherence: An entity-based ap- proach. Computational Linguistics, 34(1):1-34, 2008.","title":"Modeling local coherence: An entity-based approach","authors":"Lapata Barzilay, Regina Barzilay, Mirella Lapata","year":"2008","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2014] Justin Cheng, Lada Adamic, P. Alex Dow, Jon Michael Kleinberg, and Jure Leskovec. Can cas- cades be predicted? In Proceedings of the 23rd Interna- tional Conference on World Wide Web, 2014.","title":"Can cascades be predicted?","authors":"Justin Cheng, Lada Adamic, P Alex Dow, Jon Michael Kleinberg, Jure Leskovec","year":"2014","venue":"Proceedings of the 23rd International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheong and Young, 2015] Yun-Gyung Cheong and R. Michael Young. Suspenser: A story generation system for suspense. IEEE Transactions on Com- putational Intelligence and AI in Games, (1):39-52, 2015.","title":"Suspenser: A story generation system for suspense","authors":"Yun-Gyung Cheong, R Michael Young","year":"2015","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cummins et al., 2016] Ronan Cummins, Meng Zhang, and Ted Briscoe. Constrained multi-task learning for auto- mated essay scoring. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.","title":"Constrained multi-task learning for automated essay scoring","authors":"Ronan Cummins, Meng Zhang, Ted Briscoe","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2010] Lijun Feng, Martin Jansche, Matt Huen- erfauth, and Noémie Elhadad. A comparison of features for automatic readability assessment. In Proceedings of the 23rd International Conference on Computational Lin- guistics, 2010.","title":"A comparison of features for automatic readability assessment","authors":"Lijun Feng, Martin Jansche, Matt Huenerfauth, Noémie Elhadad","year":"2010","venue":"Proceedings of the 23rd International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ferraro and Van Durme, 2016] Francis Ferraro and Ben- jamin Van Durme. A unified Bayesian model of scripts, frames and language. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, 2016.","title":"Francis Ferraro and Benjamin Van Durme. A unified Bayesian model of scripts, frames and language","authors":"Van Ferraro, Durme","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganguly et al., 2014] Debasis Ganguly, Johannes Leveling, and Gareth J.F. Jones. Automatic prediction of text aes- thetics and interestingness. In Proceedings of the Interna- tional Conference on Computational Linguistics. 2014.","title":"Automatic prediction of text aesthetics and interestingness","authors":"Debasis Ganguly, Johannes Leveling, J F Gareth, Jones","year":"2014","venue":"Proceedings of the International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and J urgen Schmidhuber. A neural approach to automated es- say scoring. Neural Computation, (8):17351780, 1997.","title":"A neural approach to automated essay scoring","authors":"Sepp Hochreiter, Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kao and Jurafsky, 2012] Justine Kao and Dan Jurafsky. A computational analysis of style, affect, and imagery in contemporary poetry. In Proceedings of the 1st Workshop on Computational Linguistics for Literature. 2012.","title":"A computational analysis of style, affect, and imagery in contemporary poetry","authors":"Justine Kao, Dan Jurafsky","year":"2012","venue":"Proceedings of the 1st Workshop on Computational Linguistics for Literature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koncel-Kedziorski et al., 2016] Rik Koncel-Kedziorski, Ioannis Konstas, Luke Zettlemoyer, , and Hannaneh Hajishirzi. A theme-rewriting approach for generating algebra word problems. In Proceedings of EMNLP. 2016.","title":"A theme-rewriting approach for generating algebra word problems","authors":"Rik Koncel-Kedziorski, Ioannis Konstas, Luke Zettlemoyer, Hannaneh Hajishirzi","year":"2016","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2013] Boyang Li, Stephen Lee-Urban, George Johnston, and Mark O. Riedl. Story generation with crowdsourced plot graphs. In The AAAI Conferece on Ar- tificial Intelligence, 2013.","title":"Story generation with crowdsourced plot graphs","authors":"Boyang Li, Stephen Lee-Urban, George Johnston, Mark O Riedl","year":"2013","venue":"The AAAI Conferece on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luan et al., 2016] Yi Luan, Yangfeng Ji, Hannaneh Ha- jishirzi, and Boyang Li. Multiplicative representations for unsupervised semantic role induction. In Proceedings of the 54th ACL Conference, 2016.","title":"Multiplicative representations for unsupervised semantic role induction","authors":"Yi Luan, Yangfeng Ji, Hannaneh Hajishirzi, Boyang Li","year":"2016","venue":"Proceedings of the 54th ACL Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McIntyre and Lapata, 2009] Neil McIntyre and Mirella La- pata. Learning to tell tales: A data-driven approach to story generation. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics. 2009.","title":"Learning to tell tales: A data-driven approach to story generation","authors":"Lapata Mcintyre","year":"2009","venue":"Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed rep- resentations of words and phrases and their composition- ality. In Proceedings of Neural Information Processing Systems. 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Proceedings of Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Milli and Bamman, 2016] Smitha Milli and David Bam- man. Beyond canonical texts: A computational analysis of fanfiction. In Proceedings of the 2016 Conference on Em- pirical Methods in Natural Language Processing. 2016.","title":"Smitha Milli and David Bamman. Beyond canonical texts: A computational analysis of fanfiction","authors":"Bamman Milli","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[O'Neill and Riedl, 2014] Brian O'Neill and Mark Riedl. Dramatis: A computational model of suspense. In Pro- ceedings of the AAAI conference, 2014.","title":"Dramatis: A computational model of suspense","authors":"O' Neill, O' Riedl ; Brian, Mark Neill, Riedl","year":"2014","venue":"Proceedings of the AAAI conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ouyang and McKeown, 2015] Jessica Ouyang and Kath- leen McKeown. Modeling reportable events as turning points in narrative. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 2015.","title":"Modeling reportable events as turning points in narrative","authors":"Mckeown Ouyang, Jessica Ouyang, Kathleen Mckeown","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pichotta and Mooney, 2016] Karl Pichotta and Raymond J. Mooney. Learning statistical scripts with lstm recurrent neural networks. In Proceedings of the 30th AAAI Confer- ence on Artificial Intelligence. 2016.","title":"Learning statistical scripts with lstm recurrent neural networks","authors":"Mooney Pichotta, Karl Pichotta, Raymond J Mooney","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pitler and Nenkova, 2008] Emily Pitler and Ani Nenkova. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, 2008.","title":"Revisiting readability: A unified framework for predicting text quality","authors":"Nenkova Pitler, Emily Pitler, Ani Nenkova","year":"2008","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Prince, 1987] Gerald Prince. A Dictionary of Narratology. University of Nebraska Press, 1987.","title":"A Dictionary of Narratology","authors":"Gerald Prince","year":"1987","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Riedl and Young, 2010] Mark O. Riedl and R. Michael Young. Narrative planning: Balancing plot and character. Journal of Artificial Intelligence Research, 39, 2010.","title":"Narrative planning: Balancing plot and character","authors":"Mark O Riedl, R Michael Young","year":"2010","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srivastava et al., 2016] Shashank Srivastava, Snigdha Chaturvedi, and Tom Mitchell. Inferring interpersonal relations in narrative summaries. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016.","title":"Inferring interpersonal relations in narrative summaries","authors":"Shashank Srivastava, Snigdha Chaturvedi, Tom Mitchell","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stone et al., 1966] Philip J. Stone, Dexter C. Dunphy, and Marshall S. Smith. The General Inquirer: A Computer Approach to Content Analysis. MIT Press, Cambridge, MA, USA, 1966.","title":"The General Inquirer: A Computer Approach to Content Analysis","authors":"Philip J Stone, Dexter C Dunphy, Marshall S Smith","year":"1966","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Swanson and Gordon, 2012] Reid Swanson and Andrew Gordon. Say anything: Using textual case-based reason- ing to enable open-domain interactive storytelling. ACM Transactions on Interactive Intelligent Systems, 2, 2012.","title":"Say anything: Using textual case-based reasoning to enable open-domain interactive storytelling","authors":"Gordon ; ] Reid Swanson, Andrew Swanson, Gordon","year":"2012","venue":"ACM Transactions on Interactive Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taghipour and Ng, 2016] Kaveh Taghipour and Hwee Tou Ng. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. 2016.","title":"A neural approach to automated essay scoring","authors":"Ng Taghipour, Kaveh Taghipour, Hwee Tou Ng","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tatar et al., 2014] Alexandru Tatar, Marcelo Dias de Amorim, Serge Fdida, and Panayotis Antoniadis. A survey on predicting the popularity of web content. Journal of Internet Services and Applications, 5(1), 2014.","title":"A survey on predicting the popularity of web content","authors":"Alexandru Tatar, Marcelo Dias De Amorim, Serge Fdida, Panayotis Antoniadis","year":"2014","venue":"Journal of Internet Services and Applications","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}