{"type":"ScholarlyArticle","title":"Automated Essay Scoring Using Discourse External Knowledge","authors":[{"name":"Nisrine Ait Khayi","slug":"nisrine-ait-khayi","given":"Nisrine","family":"Ait Khayi","affiliation":""},{"name":"Vasile Rus","slug":"vasile-rus","given":"Vasile","family":"Rus","affiliation":""}],"abstract":"The Automated Essay Scoring (AES) task is an important NLP research problem given its significance for the education ecosystem. Recently, researchers started to apply a hybrid approach to this task. This hybrid approach incorporates into a deep learning model expert features that assess a particular dimension of the essay. Motivated by these successes, we propose to automatically assess essays using a hybrid approach that relies on external discourse knowledge. Our proposed model consists of using transformer-based embeddings to generate semantic representations of essays. Then, we incorporate several discourse features into these representations. Finally, we apply a linear classifier to generate the final score. To evaluate the effectiveness of this approach, we have conducted extensive experiments using the Automated Student Assessment Prize dataset (ASAP). The performance of the proposed model has been evaluated using the Quadratic Weighted Kappa (QWK) metric. The experimental results demonstrate the effectiveness of this approach in comparison with several existing solutions in literature.","doi":"10.24963/ijcai.2024/791","url":"https://ijcai.org/proceedings/2024/791","pdf":"https://ijcai.org/proceedings/2024/0791.pdf","markdown":"https://ijcai.org/proceedings/2024/791.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/791","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI for Good","track":"AI for Good","number":791,"pages":{"first":7154,"last":7160},"keywords":["Multidisciplinary Topics and Applications: General","Natural Language Processing: General","Knowledge Representation and Reasoning: General"],"videos":[],"references":[{"position":1,"raw":"[Amorim et al.,2018] Evelin Amorim, Marcia Cançado, and Adriano Veloso. 2018. Automated Essay Scoring in the Presence of Biased Ratings. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.New Orleans, Louisiana: Association for Computational Linguistics [","title":"Automated Essay Scoring in the Presence of Biased Ratings","authors":"Evelin Amorim, Marcia Cançado, Adriano Veloso","year":"2018","venue":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Burstein et al., 2013] Jill Burstein, Joel Tetreault, and Martin Chodorow. 2013. Holistic Discourse Coherence Annotation for Noisy Essay Writing. Dialogue and Discourse, 4(2).","title":"Holistic Discourse Coherence Annotation for Noisy Essay Writing","authors":"Jill Burstein, Joel Tetreault, Martin Chodorow","year":"2013","venue":"Dialogue and Discourse","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dasgupta et al.,2018] Tirthankar Dasgupta, Abir Naskar, Lipika Dey, and Rupsa Saha. 2018. Augmenting Textual Qualitative Features in Deep Convolution Recurrent Neural Network for Automatic Essay Scoring. In Proceedings of the Workshop on NLP Techniques for Educational Applications, Association for Computational Linguistics, pages 93-102","title":"Augmenting Textual Qualitative Features in Deep Convolution Recurrent Neural Network for Automatic Essay Scoring","authors":"Tirthankar Dasgupta, Abir Naskar, Lipika Dey, Rupsa Saha","year":"2018","venue":"Proceedings of the Workshop on NLP Techniques for Educational Applications, Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong and Zhang,2016] Fei Dong and Yue Zhang. 2016. Automatic Features for Essay Scoring -An Empirical Study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics.","title":"Automatic Features for Essay Scoring -An Empirical Study","authors":"Fei Dong, Yue Zhang","year":"2016","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al.,2017] Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-Based Recurrent Convolutional Neural Network for Automatic Essay Scoring. In Proceedings of the Conference on Computational Natural Language Learning. Vancouver, Canada: Association for Computational Linguistics.","title":"Attention-Based Recurrent Convolutional Neural Network for Automatic Essay Scoring","authors":"Fei Dong, Yue Zhang, Jie Yang","year":"2017","venue":"Proceedings of the Conference on Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graesser et al.,2005] Arthur C. Graesser and Sarah Petschonek. 2005. Automated Systems that Analyze Text and Discourse: QUAID, Coh-Metrix, and AutoTutor. Ad- vancing Health Outcomes Research Methods and Clinical Applications, McLean, VA: Degnon Associates.","title":"Automated Systems that Analyze Text and Discourse: QUAID, Coh-Metrix, and AutoTutor. Advancing Health Outcomes Research Methods and Clinical Applications","authors":"Arthur C Graesser, Sarah Petschonek","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Halliday and Hasan, 1976] Michael Halliday and Ruqaiya Hasan. 1976. Cohesion in English London: Longman.","title":"Cohesion in English London","authors":"Michael Halliday, Ruqaiya Hasan","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ke and Ng, 2019] Zixuan Ke and Vincent Ng.2019. Auto- mated Essay Scoring: A Survey of the State of the Art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence.","title":"Automated Essay Scoring: A Survey of the State of the Art","authors":"Ng Ke, Zixuan Ke, Vincent Ng","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/879"},{"position":9,"raw":"[Ke et al.,2019] Zixuan Ke, Hrishikesh Inamdar, Hui Lin, and Vincent Ng. 2019. Give Me More Feedback II: Annotat- ing Thesis Strength and Related Attributes in Student Es- says. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, It- aly: Association for Computational Linguistics.","title":"Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays","authors":"Zixuan Ke, Hrishikesh Inamdar, Hui Lin, Vincent Ng","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma et al.,2014] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al.,2019] Jiawei Liu, Yang Xu, and Yaguang Zhu. 2019. Automated Essay Scoring Based on Two-Stage Learning. ArXiv, abs/1901.07744.","title":"Automated Essay Scoring Based on Two-Stage Learning","authors":"Jiawei Liu, Yang Xu, Yaguang Zhu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mathias et al.,2018] Sandeep Mathias and Pushpak Bhattacharyya. 2018. Thank \"Goodness\"! A Way to Measure Style in Student Essays. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications. Association for Computational Linguistics.","title":"Thank \"Goodness\"! A Way to Measure Style in Student Essays","authors":"Sandeep Mathias, Pushpak Bhattacharyya","year":"2018","venue":"Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mayfield and Black,2020] Elijah Mayfield and Alan W Black. 2020. Should You Fine-Tune BERT for Automated Essay Scoring?. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications. Seattle, WA: Association for Computational Linguistics.","title":"Should You Fine-Tune BERT for Automated Essay Scoring?","authors":"Elijah Mayfield, Alan W Black","year":"2020","venue":"Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mathias and Bhattacharya, 2018] Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores. In Proceedings of the International Conference on Language Resources and Evaluation.","title":"ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores","authors":"Sandeep Mathias, Pushpak Bhattacharyya","year":"2018","venue":"Proceedings of the International Conference on Language Resources and Evaluation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McNamara et al., 2014] Danielle S. McNamara, Arthur C. Graesser, Philip M. McCarthy, and Zhiqiang Cai. 2014. Automated Evaluation of Text and Discourse with Coh- Metrix. Cambridge University Press.","title":"Automated Evaluation of Text and Discourse with Coh-Metrix","authors":"Danielle S Mcnamara, Arthur C Graesser, Philip M Mccarthy, Zhiqiang Cai","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Morris and Hirst,1991] Jane Morris and Graeme Hirst. 1991. Lexical Cohesion Computed by Thesaural Relations as an Indicator of the Structure of Text. Computational Linguistics 17(1).","title":"Lexical Cohesion Computed by Thesaural Relations as an Indicator of the Structure of Text","authors":"Jane Morris, Graeme Hirst","year":"1991","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen and Litman,2018] Huy V. Nguyen and Diane J. Litman. 2018. Argument Mining for Improving the Automated Scoring of Persuasive Essays. In Proceedings of the AAI Conference on Artificial Intelligence, New Orleans, Louisiana: Association for the Advancement of Artificial Intelligence.","title":"Argument Mining for Improving the Automated Scoring of Persuasive Essays","authors":"V Huy, Diane J Nguyen, Litman","year":"2018","venue":"Proceedings of the AAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ridley et al.,2020] Robert Ridley, Liang He, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2020. Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-Prompt Automated Essay Scoring. ArXiv, abs/2008.01441.","title":"Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-Prompt Automated Essay Scoring","authors":"Robert Ridley, Liang He, Xinyu Dai, Shujian Huang, Jiajun Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rodriguez et al.,2019] Pedro Uria Rodriguez, Amir Jafari, and Christopher M. Ormerod. 2019. Language models and Automated Essay Scoring. ArXiv, abs/1909.09482.","title":"Language models and Automated Essay Scoring","authors":"Pedro Uria Rodriguez, Amir Jafari, Christopher M Ormerod","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rus and Niraula, 2012] Rus, Vasile, and Nobal Niraula.2012. \"Automated Detection of Local Coherence in Short Argumentative Essays based on Centering Theory.\" Computational Linguistics and Intelligent Text Processing: 13th International Conference, CICLing 2012, New Delhi, India, March 11-17, 2012, Proceedings, Part I 13. Springer Berlin Heidelberg, 2012.","title":"Automated Detection of Local Coherence in Short Argumentative Essays based on Centering Theory","authors":"Vasile Rus, Nobal Niraula","year":"2012","venue":"Computational Linguistics and Intelligent Text Processing: 13th International Conference, CICLing 2012","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Somasundaran et al.,2018] Swapna Somasundaran, Michael Flor, Martin Chodorow, Hillary Molloy, Binod Gyawali, and Laura McCulla. 2018. Towards Evaluating Narrative Quality in Student Writing. Computational Linguistics 6(0).","title":"Towards Evaluating Narrative Quality in Student Writing","authors":"Swapna Somasundaran, Michael Flor, Martin Chodorow, Hillary Molloy, Binod Gyawali, Laura Mcculla","year":"2018","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2020] Wei Song, Ziyao Song, Lizhen Liu, and Ruiji Fu.2020. Hierarchical Multi-task Learning for Organization Evaluation of Argumentative Student Essays. In Bessiere, C., ed., Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, 3875-38. International Joint Conferences on Artificial Intelligence Organization.","title":"Hierarchical Multi-task Learning for Organization Evaluation of Argumentative Student Essays","authors":"Wei Song, Ziyao Song, Lizhen Liu, Ruiji Fu","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20","doi":null,"url":null,"internal_url":"/proceedings/2020/536"},{"position":23,"raw":"[Taghipour et al., 2016] Kaveh Taghipour and Hwee Tou Ng. 2016. A Neural Approach to Automated Essay Scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1882-1891.","title":"A Neural Approach to Automated Essay Scoring","authors":"Kaveh Taghipour, Hwee Tou Ng","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Austin, Texas: Association for Computational Linguistics.","title":"Association for Computational Linguistics","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Uto et al.,2020] Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural Automated Essay Scoring Incorporating Handcrafted Features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077-6088, Barcelona, Spain (Online). International Committee on Computational Linguistics.","title":"Neural Automated Essay Scoring Incorporating Handcrafted Features","authors":"Masaki Uto, Yikuan Xie, Maomi Ueno","year":"2020","venue":"Proceedings of the 28th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al.,2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNET: Generalized Autoregressive Pretraining for Language Understanding. Advances in Neural Information Processing Systems, 32.","title":"XLNET: Generalized Autoregressive Pretraining for Language Understanding","authors":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, V Quoc, Le","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al.,2020] Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing Automated Essay Scoring Performance via Fine-Tuning Pre-trained Language Models with Combination of Regression and Ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560-1569, Online. Association for Computational Linguistics.","title":"Enhancing Automated Essay Scoring Performance via Fine-Tuning Pre-trained Language Models with Combination of Regression and Ranking","authors":"Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, Xiaodong He","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al.,2018] Haoran Zhang and Diane Litman. 2018. Co-Attention Based Neural Network for Source- Dependent Essay Scoring. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, 399-409. New Orleans, Louisiana: Association for Computational Linguistics.","title":"Co-Attention Based Neural Network for Source-Dependent Essay Scoring","authors":"Haoran Zhang, Diane Litman","year":"2018","venue":"Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al.,2020] Haoran Zhang and Diane Litman. 2020. Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8569-8584.","title":"Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring","authors":"Haoran Zhang, Diane Litman","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Seattle, WA, USA (Virtual Event): Association for Computational Linguistics.","title":"Association for Computational Linguistics","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al.,2022] Xuandong Zhao, Zhiguo Yu, Ming Wu, and Lei Li. 2022.Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation. arXiv preprint arXiv:2203.07687. .","title":"Compressing Sentence Representation for Semantic Retrieval via Homomorphic Projective Distillation","authors":"Xuandong Zhao, Zhiguo Yu, Ming Wu, Lei Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}