{"type":"ScholarlyArticle","title":"Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading","authors":[{"name":"Sachin Kumar","slug":"sachin-kumar","given":"Sachin","family":"Kumar","affiliation":"Conduent Labs India , Bangalore"},{"name":"Soumen Chakrabarti","slug":"soumen-chakrabarti","given":"Soumen","family":"Chakrabarti","affiliation":"IIT Bombay"},{"name":"Shourya Roy","slug":"shourya-roy","given":"Shourya","family":"Roy","affiliation":"American Express Big Data Labs"}],"abstract":"Automatic short answer grading (ASAG) can reduce tedium for instructors, but is complicated by free-form student inputs. An important ASAG task is to assign  ordinal scores to student answers, given some “model” or ideal answers. Here we introduce a novel framework for ASAG by cascading three neural building blocks: Siamese bidirectional LSTMs applied to a model and a student answer, a novel pooling layer based on earth-mover distance (EMD) across all hidden states from  both LSTMs, and a flexible final regression layer to output scores. On standard ASAG data sets, our system shows substantial reduction in grade estimation error compared to competitive baselines. We demonstrate that EMD pooling results in substantial accuracy gains, and that a support vector ordinal regression (SVOR) output layer helps outperform softmax. Our system also outperforms recent attention mechanisms on LSTM states.","doi":"10.24963/ijcai.2017/284","url":"https://ijcai.org/proceedings/2017/284","pdf":"https://ijcai.org/proceedings/2017/0284.pdf","markdown":"https://ijcai.org/proceedings/2017/284.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/284","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":284,"pages":{"first":2046,"last":2052},"keywords":["Natural Language Processing: Natural Language Processing","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Alikaniotis et al., 2016] Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. Automatic Text Scoring Using Neural Networks. CoRR, abs/1606.04289, 2016.","title":"Automatic Text Scoring Using Neural Networks","authors":"Dimitrios Alikaniotis, Helen Yannakoudakis, Marek Rei","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christo- pher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burrows et al., 2015] Steven Burrows, Iryna Gurevych, and Benno Stein. The Eras and Trends of Automatic Short Answer Grading. In- ternational Journal of Artificial Intelligence in Education, 25(1):60- 117, 2015.","title":"The Eras and Trends of Automatic Short Answer Grading","authors":"Steven Burrows, Iryna Gurevych, Benno Stein","year":"2015","venue":"International Journal of Artificial Intelligence in Education","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng, 2007] Jianlin Cheng. A neural network approach to ordinal regression. CoRR, abs/0704.1028, 2007.","title":"A neural network approach to ordinal regression","authors":"Jianlin Cheng","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chu and Keerthi, 2007] Wei Chu and S. Sathiya Keerthi. Support Vec- tor Ordinal Regression. Neural Comp., 19:792-815, 2007.","title":"Support Vector Ordinal Regression","authors":"Wei Chu, S Sathiya Keerthi","year":"2007","venue":"Neural Comp","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conole and Warburton, 2005] Gráinne Conole and Bill Warburton. A review of computer-assisted assessment. Research in learning tech- nology, 13(1), 2005.","title":"A review of computer-assisted assessment","authors":"Conole, Warburton ; Conole, Bill Warburton","year":"2005","venue":"Research in learning technology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2015] Xiaodong Cui, Vaibhava Goel, and Brian Kings- bury. Data augmentation for deep neural network acoustic mod- eling. Transactions on Audio, Speech and Language Processing, 23(9):1469-1477, 2015.","title":"Data augmentation for deep neural network acoustic modeling","authors":"Xiaodong Cui, Vaibhava Goel, Brian Kingsbury","year":"2015","venue":"Transactions on Audio, Speech and Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Lightspeed compu- tation of optimal transport. In Advances in Neural Information Pro- cessing Systems, pages 2292-2300, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dzikovska et al., 2013] Myroslava O. Dzikovska, Rodney D. Nielsen, Chris Brew, Claudia Leacock, Danilo Giampiccolo, Luisa Bentivogli, Peter Clark, Ido Dagan, and Hoa T Dang. SemEval-2013 task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge. Technical report, DTIC Document, 2013.","title":"SemEval-2013 task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge","authors":"O Myroslava, Rodney D Dzikovska, Chris Nielsen, Claudia Brew, Danilo Leacock, Luisa Giampiccolo, Peter Bentivogli, Ido Clark, Hoa T Dagan, Dang","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gutiérrez et al., 2016] Pedro Antonio Gutiérrez, Maria Perez-Ortiz, Javier Sanchez-Monedero, Francisco Fernandez-Navarro, and Cesar Hervas-Martinez. Ordinal regression methods: survey and experi- mental study. IEEE TKDE, 28(1):127-146, 2016.","title":"Ordinal regression methods: survey and experimental study","authors":"Pedro Antonio Gutiérrez, Maria Perez-Ortiz, Javier Sanchez-Monedero, Francisco Fernandez-Navarro, Cesar Hervas-Martinez","year":"2016","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He and Lin, 2016] Hua He and Jimmy Lin. Pairwise word interaction modeling with deep neural networks for semantic similarity measure- ment. In NAACL-HLT, pages 937-948, 2016.","title":"Pairwise word interaction modeling with deep neural networks for semantic similarity measurement","authors":"Lin ; ] Hua He, Jimmy He, Lin","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heilman and Madnani, 2013] Michael Heilman and Nitin Madnani. ETS: Domain Adaptation and Stacking for Short Answer Scoring. In Proceedings of the 2nd joint conference on lexical and computa- tional semantics, volume 2, pages 275-279, 2013.","title":"ETS: Domain Adaptation and Stacking for Short Answer Scoring","authors":"Michael Heilman, Nitin Madnani","year":"2013","venue":"Proceedings of the 2nd joint conference on lexical and computational semantics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Herbrich et al., 1999] Ralf Herbrich, Thore Graepel, and Klaus Ober- mayer. Support vector learning for ordinal regression. In ICANN, pages 97-102, 1999.","title":"Support vector learning for ordinal regression","authors":"Ralf Herbrich, Thore Graepel, Klaus Obermayer","year":"1999","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jurgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2016] Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, and Kilian Q Weinberger. Supervised word mover's dis- tance. In NIPS, pages 4862-4870, 2016.","title":"Supervised word mover's distance","authors":"Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, Kilian Q Weinberger","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kaggle, 2012a] Kaggle. The Hewlett Foundation: Automated Essay Scoring. https://www.kaggle.com/c/asap-aes, 2012.","title":"Kaggle. The Hewlett Foundation: Automated Essay Scoring","authors":"Kaggle","year":"2012","venue":null,"doi":null,"url":"https://www.kaggle.com/c/asap-aes,2012","internal_url":null},{"position":17,"raw":"[Kaggle, 2012b] Kaggle. The Hewlett Foundation: Short Answer Scor- ing. http://www.kaggle.com/c/asap-sas, 2012.","title":"Kaggle. The Hewlett Foundation: Short Answer Scoring","authors":"Kaggle","year":"2012","venue":null,"doi":null,"url":"http://www.kaggle.com/c/asap-sas,2012","internal_url":null},{"position":18,"raw":"[Knight, 2008] Philip A Knight. The sinkhorn-knopp algorithm: con- vergence and applications. SIAM Journal on Matrix Analysis and Applications, 30(1):261-275, 2008.","title":"The sinkhorn-knopp algorithm: convergence and applications","authors":"A Philip, Knight","year":"2008","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kusner et al., 2015] Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. From Word Embeddings To Document Dis- tances. In ICML, volume 37, pages 957-966, 2015.","title":"From Word Embeddings To Document Distances","authors":"Matt J Kusner, Yu Sun, Nicholas I Kolkin, Kilian Q Weinberger","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Levina and Bickel, 2001] Elizaveta Levina and Peter J. Bickel. The earth mover's distance is the mallows distance: Some insights from statistics. In ICCV, pages 251-256, 2001.","title":"The earth mover's distance is the mallows distance: Some insights from statistics","authors":"Elizaveta Levina, Peter J Bickel","year":"2001","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2016] Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. Learning natural language inference using bidirectional LSTM model and inner-attention. CoRR, abs/1605.09090, 2016.","title":"Learning natural language inference using bidirectional LSTM model and inner-attention","authors":"Yang Liu, Chengjie Sun, Lei Lin, Xiaolong Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[MacCartney, 2009] Bill MacCartney. Natural language inference. PhD thesis, 2009.","title":"Natural language inference","authors":"Maccartney","year":"2009","venue":"Bill MacCartney","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mikolov et al., 2013a] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mohler et al., 2011] Michael Mohler, Razvan C. Bunescu, and Rada Mihalcea. Learning to Grade Short Answer Questions using Seman- tic Similarity Measures and Dependency Graph Alignments. In ACL, pages 752-762, 2011.","title":"Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments","authors":"Michael Mohler, Razvan C Bunescu, Rada Mihalcea","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mueller and Thyagarajan, 2016] Jonas Mueller and Aditya Thyagara- jan. Siamese recurrent architectures for learning sentence similarity. In AAAI, pages 2786-2792. AAAI Press, 2016.","title":"Jonas Mueller and Aditya Thyagarajan. Siamese recurrent architectures for learning sentence similarity","authors":"Thyagarajan Mueller","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ostermann et al., 2015] Simon Ostermann, Andrea Horbach, and Manfred Pinkal. Annotating Entailment Relations for Shortanswer Questions. In ACL-IJCNLP, page 49, 2015.","title":"Annotating Entailment Relations for Shortanswer Questions","authors":"Simon Ostermann, Andrea Horbach, Manfred Pinkal","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pulman and Sukkarieh, 2005] Stephen G. Pulman and Jana Z. Sukkarieh. Automatic Short Answer Marking. In Proceedings of the Second Workshop on Building Educational Applications Using NLP (EdAppsNLP), pages 9-16, 2005.","title":"Automatic Short Answer Marking","authors":"Stephen G Pulman, Jana Z Sukkarieh","year":"2005","venue":"Proceedings of the Second Workshop on Building Educational Applications Using NLP (EdAppsNLP)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ramachandran et al., 2015] Lakshmi Ramachandran, Jian Cheng, and Peter Foltz. Identifying Patterns for Short Answer Scoring using Graph-based Lexico-semantic Text Matching. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 97-106, 2015.","title":"Identifying Patterns for Short Answer Scoring using Graph-based Lexico-semantic Text Matching","authors":"Lakshmi Ramachandran, Jian Cheng, Peter Foltz","year":"2015","venue":"Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rocktäschel et al., 2015] Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. Rea- soning about entailment with neural attention. arXiv preprint arXiv:1509.06664, 2015.","title":"Reasoning about entailment with neural attention","authors":"Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Phil Blunsom","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Roy et al., 2015] Shourya Roy, Y. Narahari, and Om D. Deshmukh. A Perspective on Computer Assisted Assessment Techniques for Short Free-Text Answers. In Computer Assisted Assessment, pages 96-109. Springer, 2015.","title":"A Perspective on Computer Assisted Assessment Techniques for Short Free-Text Answers","authors":"Y Shourya Roy, Om D Narahari, Deshmukh","year":"2015","venue":"Computer Assisted Assessment","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sakaguchi et al., 2015] Keisuke Sakaguchi, Michael Heilman, and Nitin Madnani. Effective Feature Integration for Automated Short Answer Scoring. In NAACL-HLT, pages 1049-1054, 2015.","title":"Effective Feature Integration for Automated Short Answer Scoring","authors":"Keisuke Sakaguchi, Michael Heilman, Nitin Madnani","year":"2015","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sammons et al., 2011] Mark Sammons, VG Vydiswaran, and Dan Roth. Recognizing textual entailment. Multilingual Natural Lan- guage Applications: From Theory to Practice. Prentice-Hall, 2011.","title":"Recognizing textual entailment","authors":"Mark Sammons, Dan Vg Vydiswaran, Roth","year":"2011","venue":"Multilingual Natural Language Applications: From Theory to Practice","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Severyn and Moschitti, 2015] Aliaksei Severyn and Alessandro Mos- chitti. Learning to rank short text pairs with convolutional deep neu- ral networks. In ACM SIGIR, pages 373-382, 2015.","title":"Learning to rank short text pairs with convolutional deep neural networks","authors":"Aliaksei Severyn, Alessandro Moschitti","year":"2015","venue":"ACM SIGIR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sinkhorn and Knopp, 1967] Richard Sinkhorn and Paul Knopp. Con- cerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-348, 1967.","title":"Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices","authors":"Knopp Sinkhorn","year":"1967","venue":"Pacific Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Socher et al., 2011] Richard Socher, Eric H. Huang, Jeffrey Penning- ton, Andrew Y. Ng, and Christopher D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, pages 801-809, 2011.","title":"Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection","authors":"Richard Socher, Eric H Huang, Jeffrey Pennington, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Sultan et al., 2016] Md. Arafat Sultan, Cristobal Salazar, and Tamara Sumner. Fast and Easy Short Answer Grading with High Accuracy. In NAACL-HLT, pages 1070-1075, 2016.","title":"Fast and Easy Short Answer Grading with High Accuracy","authors":"Arafat Md, Cristobal Sultan, Tamara Salazar, Sumner","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang and Jiang, 2015] Shuohang Wang and Jing Jiang. Learn- ing natural language inference with LSTM. arXiv preprint arXiv:1512.08849, 2015.","title":"Learning natural language inference with LSTM","authors":"Wang, Jiang, Shuohang Wang, Jing Jiang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Willmott and Matsuura, 2005] Cort J. Willmott and Kenji Matsuura. Advantages of the Mean Absolute Error (MAE) over the Root Mean Square Error (RMSE) in Assessing Average Model Performance. Cli- mate research, 30(1):79-82, 2005.","title":"Advantages of the Mean Absolute Error (MAE) over the Root Mean Square Error (RMSE) in Assessing Average Model Performance","authors":"J Cort, Kenji Willmott, Matsuura","year":"2005","venue":"Climate research","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yin et al., 2015] Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. ABCNN: Attention-based convolutional neural net- work for modeling sentence pairs. arXiv preprint arXiv:1512.05193, 2015.","title":"ABCNN: Attention-based convolutional neural network for modeling sentence pairs","authors":"Yin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang and LeCun, 2015] Xiang Zhang and Yann LeCun. Text under- standing from scratch. arXiv preprint arXiv:1502.01710, 2015.","title":"Text understanding from scratch","authors":"Xiang Zhang, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}