{"type":"ScholarlyArticle","title":"Enhancing Length Generalization for Attention Based Knowledge Tracing Models with Linear Biases","authors":[{"name":"Xueyi Li","slug":"xueyi-li","given":"Xueyi","family":"Li","affiliation":"Guangdong Institute of Smart Education , Jinan University; School of Intelligent Systems Science and Engineering , Jinan University"},{"name":"Youheng Bai","slug":"youheng-bai","given":"Youheng","family":"Bai","affiliation":"Guangdong Institute of Smart Education , Jinan University"},{"name":"Teng Guo","slug":"teng-guo","given":"Teng","family":"Guo","affiliation":"Guangdong Institute of Smart Education , Jinan University"},{"name":"Zitao Liu","slug":"zitao-liu","given":"Zitao","family":"Liu","affiliation":"Guangdong Institute of Smart Education , Jinan University"},{"name":"Yaying Huang","slug":"yaying-huang","given":"Yaying","family":"Huang","affiliation":"Guangdong Institute of Smart Education , Jinan University"},{"name":"Xiangyu Zhao","slug":"xiangyu-zhao","given":"Xiangyu","family":"Zhao","affiliation":"School of Data Science , City University of Hong Kong"},{"name":"Feng Xia","slug":"feng-xia","given":"Feng","family":"Xia","affiliation":"School of Computing Technologies , RMIT University"},{"name":"Weiqi Luo","slug":"weiqi-luo","given":"Weiqi","family":"Luo","affiliation":"Guangdong Institute of Smart Education , Jinan University"},{"name":"Jian Weng","slug":"jian-weng","given":"Jian","family":"Weng","affiliation":"College of Information Science and Technology , Jinan University"}],"abstract":"Knowledge tracing (KT) is the task of predicting students' future performance based on their historical learning interaction data. With the rapid advancement of attention mechanisms, many attention based KT models are developed. However, existing attention based KT models exhibit performance drops as the number of student interactions increases beyond the number of interactions on which the KT models are trained. We refer to this as the length generalization of KT model. In this paper, we propose stableKT to enhance length generalization that is able to learn from short sequences and maintain high prediction performance when generalizing on long sequences. Furthermore, we design a multi-head aggregation module to capture the complex relationships between questions and the corresponding knowledge components (KCs) by combining dot-product attention and hyperbolic attention. Experimental results on three public educational datasets show that our model exhibits robust capability of length generalization and outperforms all baseline models in terms of AUC. To encourage reproducible research, we make our data and code publicly available at https://pykt.org.","doi":"10.24963/ijcai.2024/654","url":"https://ijcai.org/proceedings/2024/654","pdf":"https://ijcai.org/proceedings/2024/0654.pdf","markdown":"https://ijcai.org/proceedings/2024/654.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/654","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":654,"pages":{"first":5918,"last":5926},"keywords":["Humans and AI: HAI: Computer-aided education","Multidisciplinary Topics and Applications: MTA: Education"],"videos":[],"references":[{"position":1,"raw":"[Chi et al., 2023] Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Associa- tion for Computational Linguistics, Toronto, Canada, July 2023.","title":"Dissecting transformer length extrapolation via the lens of receptive field analysis","authors":"Ta-Chung Chi, Alexander Ting-Han Fan, Peter Rudnicky, Ramadge","year":"2023","venue":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Choi et al., 2020] Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of the 7th ACM Conference on Learning At Scale, Virtual Conference, August 2020.","title":"Towards an appropriate query, key, and value computation for knowledge tracing","authors":"Choi","year":"2020","venue":"Proceedings of the 7th ACM Conference on Learning At Scale, Virtual Conference","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2023] Jiajun Cui, Zeyuan Chen, Aimin Zhou, Jianyong Wang, and Wei Zhang. Fine-grained interac- tion modeling with multi-relational transformer for knowl- edge tracing. ACM Transactions on Information Systems, 41(4):1-26, January 2023.","title":"Fine-grained interaction modeling with multi-relational transformer for knowledge tracing","authors":"Jiajun Cui, Zeyuan Chen, Aimin Zhou, Jianyong Wang, Wei Zhang","year":"2023","venue":"ACM Transactions on Information Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghosh et al., 2020] Aritra Ghosh, Neil Heffernan, and An- drew S Lan. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Vir- tual Conference, August 2020.","title":"Context-aware attentive knowledge tracing","authors":"Aritra Ghosh, Neil Heffernan, Andrew S Lan","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gulcehre et al., 2019] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. Hyperbolic attention networks. In Proceedings of the 7th International Con- ference on Learning Representations, New Orleans, LA, USA, April 2019.","title":"Hyperbolic attention networks","authors":"Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro","year":"2019","venue":"Proceedings of the 7th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2021] Xiaopeng Guo, Zhijie Huang, Jie Gao, Mingyu Shang, Maojing Shu, and Jun Sun. Enhancing knowledge tracing via adversarial training. In Proceedings of the 29th ACM International Conference on Multimedia, Virtual Conference, October 2021.","title":"Enhancing knowledge tracing via adversarial training","authors":"Xiaopeng Guo, Zhijie Huang, Jie Gao, Mingyu Shang, Maojing Shu, Jun Sun","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia, Virtual Conference","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2023] Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, and Jian Weng. Towards robust knowl- edge tracing models via k-sparse attention. In Proceedings of the 46th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, Taipei, Taiwan, China, July 2023.","title":"Towards robust knowledge tracing models via k-sparse attention","authors":"Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, Jian Weng","year":"2023","venue":"Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Im et al., 2023] Yoonjin Im, Eunseong Choi, Heejin Kook, and Jongwuk Lee. Forgetting-aware linear bias for atten- tive knowledge tracing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, Birmingham, UK, October 2023.","title":"Forgetting-aware linear bias for attentive knowledge tracing","authors":"Im","year":"2023","venue":"Proceedings of the 32nd ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jacob et al., 2019] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. BERT: pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Min- neapolis, MN, USA, June 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob","year":"2019","venue":"Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2023] Guanxin Li, Jingang Shi, Yuan Zong, Fei Wang, Tian Wang, and Yihong Gong. Learning attention from attention: efficient self-refinement transformer for face super-resolution. In Proceedings of the 32nd Interna- tional Joint Conference on Artificial Intelligence, Macao, SAR, China, August 2023.","title":"Learning attention from attention: efficient self-refinement transformer for face super-resolution","authors":"Guanxin Li, Jingang Shi, Yuan Zong, Fei Wang, Tian Wang, Yihong Gong","year":"2023","venue":"Proceedings of the 32nd International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/115"},{"position":11,"raw":"[Liu et al., 2022] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang, and Weiqi Luo. pykt: A python library to benchmark deep learning based knowl- edge tracing models. In Proceedings of 36th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, New Orleans, LA, USA, November 2022.","title":"pykt: A python library to benchmark deep learning based knowledge tracing models","authors":"Liu","year":"2022","venue":"Proceedings of 36th Conference on Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023a] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Boyu Gao, Weiqi Luo, and Jian Weng. En- hancing deep knowledge tracing with auxiliary tasks. In Proceedings of the ACM Web Conference 2023, Austin, TX, USA, April 2023.","title":"Enhancing deep knowledge tracing with auxiliary tasks","authors":"Liu","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2023b] Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, and Weiqi Luo. simpleKT: A simple but tough-to-beat baseline for knowledge tracing. In Proceed- ings of the 11th International Conference on Learning Representations, Kigali, Rwanda, May 2023.","title":"simpleKT: A simple but tough-to-beat baseline for knowledge tracing","authors":"Liu","year":"2023","venue":"Proceedings of the 11th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nakagawa et al., 2019] Hiromi Nakagawa, Yusuke Iwa- sawa, and Yutaka Matsuo. Graph-based knowledge trac- ing: Modeling student proficiency using graph neural net- work. In Proceedings of the 2019 IEEE/WIC/ACM In- ternational Conference on Web Intelligence, Thessaloniki, Greece, October 2019.","title":"Graph-based knowledge tracing: Modeling student proficiency using graph neural network","authors":"Hiromi Nakagawa, Yusuke Iwasawa, Yutaka Matsuo","year":"2019","venue":"Proceedings of the 2019 IEEE/WIC/ACM International Conference on Web Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pan et al., 2022] Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. In Proceedings of the 36th Conference on Neural Information Processing Systems, New Orleans, LA, USA, November 2022.","title":"Fast vision transformers with hilo attention","authors":"Pan","year":"2022","venue":"Proceedings of the 36th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pandey and Karypis, 2019] Shalini Pandey and George Karypis. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Edu- cational Data Mining, Montréal, Canada, July 2019.","title":"A self-attentive model for knowledge tracing","authors":"Shalini Pandey, George Karypis","year":"2019","venue":"Proceedings of the 12th International Conference on Educational Data Mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Piech et al., 2015] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge trac- ing. In Proceedings of the 28th Conference on Neural In- formation Processing Systems, Montreal, Quebec, Canada, December 2015.","title":"Deep knowledge tracing","authors":"Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, Jascha Sohl-Dickstein","year":"2015","venue":"Proceedings of the 28th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Press et al., 2022] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of the 10th In- ternational Conference on Learning Representations, Vir- tual Conference, April 2022.","title":"Train short, test long: Attention with linear biases enables input length extrapolation","authors":"Ofir Press, Noah Smith, Mike Lewis","year":"2022","venue":"Proceedings of the 10th International Conference on Learning Representations, Virtual Conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qin et al., 2024] Zhen Qin, Yiran Zhong, and Hui Deng. Exploring transformer extrapolation. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, Vancou- ver, Canada, February 2024.","title":"Exploring transformer extrapolation","authors":"Zhen Qin, Yiran Zhong, Hui Deng","year":"2024","venue":"Proceedings of the 38th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(140):1- 67, February 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2021] Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, and Shi- jin Wang. Learning process-consistent knowledge trac- ing. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual Confer- ence, August 2021.","title":"Learning process-consistent knowledge tracing","authors":"Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, Shijin Wang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Virtual Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Su et al., 2021] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.","title":"Roformer: Enhanced transformer with rotary position embedding","authors":"Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, Yunfeng Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tseng et al., 2023] Albert Tseng, Tao Yu, Toni J.B. Liu, and Christopher De Sa. Coneheads: Hierarchy aware atten- tion. In Proceedings of the 37th Conference on Neural Information Processing Systems, New Orleans, LA, USA, December 2023.","title":"Coneheads: Hierarchy aware attention","authors":"Albert Tseng, Tao Yu, Toni J B Liu, Christopher De Sa","year":"2023","venue":"Proceedings of the 37th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, December 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of the 31st Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2023] Zhiwei Wang, Junlin Xian, Kangyi Liu, Xin Li, Qiang Li, and Xin Yang. Dual-view correlation hybrid attention network for robust holistic mammogram classification. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, Macao, SAR, China, August 2023.","title":"Dual-view correlation hybrid attention network for robust holistic mammogram classification","authors":"Zhiwei Wang, Junlin Xian, Kangyi Liu, Xin Li, Qiang Li, Xin Yang","year":"2023","venue":"Proceedings of the 32nd International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/168"},{"position":26,"raw":"[Yin et al., 2023] Yu Yin, Le Dai, Zhenya Huang, Shuanghong Shen, Fei Wang, Qi Liu, Enhong Chen, and Xin Li. Tracing knowledge instead of patterns: Stable knowledge tracing with diagnostic transformer. In Proceedings of the ACM Web Conference 2023, Austin, TX, USA, April 2023.","title":"Tracing knowledge instead of patterns: Stable knowledge tracing with diagnostic transformer","authors":"Yu Yin, Le Dai, Zhenya Huang, Shuanghong Shen, Fei Wang, Qi Liu, Enhong Chen, Xin Li","year":"2023","venue":"Proceedings of the ACM Web Conference 2023","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2023] Tao Yu, Toni JB Liu, Albert Tseng, and Christopher De Sa. Shadow cones: Unveiling partial orders in hyperbolic space. arXiv preprint arXiv:2305.15215, 2023.","title":"Shadow cones: Unveiling partial orders in hyperbolic space","authors":"Tao Yu, Toni Jb Liu, Albert Tseng, Christopher De Sa","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th Inter- national Conference on World Wide Web, Perth, Canada, April 2017.","title":"Dynamic key-value memory networks for knowledge tracing","authors":"Jiani Zhang, Xingjian Shi, Irwin King, Dit-Yan Yeung","year":"2017","venue":"Proceedings of the 26th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}