{"type":"ScholarlyArticle","title":"HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens","authors":[{"name":"Ivan Karpukhin","slug":"ivan-karpukhin","given":"Ivan","family":"Karpukhin","affiliation":"Sber AI Lab , Russia"},{"name":"Maksim Polesskii","slug":"maksim-polesskii","given":"Maksim","family":"Polesskii","affiliation":"Sber AI Lab , Russia; Lomonosov Moscow State University , Russia"},{"name":"Andrey Savchenko","slug":"andrey-savchenko","given":"Andrey","family":"Savchenko","affiliation":"Sber AI Lab , Russia; HSE University , Laboratory for Theoretical Foundations of AI Models , Russia"}],"abstract":"Deep learning has achieved strong results in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform recurrent networks in classification tasks that aim to predict future targets, such as churn, user reactions, or treatment response. The reason behind this performance gap remains largely underexplored. In this paper, we identify a key limitation of transformers: the lack of a single vector representation that compactly summarizes the evolving state of a sequence. We further show that commonly used contrastive embeddings are poorly suited to capturing the local context needed for accurate forward-looking prediction. To address these challenges, we introduce history tokens, a novel concept that enables the accumulation of historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks. The code is publicly available: https://github.com/ivan-chai/pretpp.","doi":"10.24963/ijcai.2026/719","url":"https://ijcai.org/proceedings/2026/719","pdf":"https://ijcai.org/proceedings/2026/0719.pdf","markdown":"https://ijcai.org/proceedings/2026/719.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/719","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"AI4Tech: AI Enabling Technologies","track":"AI4Tech: AI Enabling Technologies","number":719,"pages":{"first":6462,"last":6470},"keywords":["Domain-specific AI4Tech: AI4Care and AI4Health","Domain-specific AI4Tech: AI4Finance","Advanced AI4Tech: Deep AI4Tech","Domain-specific AI4Tech: AI4Customer and AI4Market"],"videos":[],"references":[{"position":1,"raw":"[Babaev et al., 2022] Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Maria Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin. ColLES: Contrastive learning for event sequences with self-supervision. In Proceedings of the 2022 International Conference on Management of Data, pages 1190-1199, 2022.","title":"ColLES: Contrastive learning for event sequences with self-supervision","authors":"Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Maria Ivanova, Gleb Gusev, Ivan Nazarov, Alexander Tuzhilin","year":"2022","venue":"Proceedings of the 2022 International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[BehnamGhader et al., 2024] Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024.","title":"LLM2Vec: Large language models are secretly powerful text encoders","authors":"Behnamghader","year":"2024","venue":"First Conference on Language Modeling","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beltagy et al., 2020] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020.","title":"Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer","authors":"Beltagy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bulatov et al., 2022] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079-11091, 2022.","title":"Recurrent memory transformer","authors":"Aydar Bulatov, Yury Kuratov, Mikhail Burtsev","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chopra et al., 2005] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recog- nition (CVPR'05), volume 1, pages 539-546. IEEE, 2005.","title":"Learning a similarity metric discriminatively, with application to face verification","authors":"Sumit Chopra, Raia Hadsell, Yann Lecun","year":"2005","venue":"IEEE computer society conference on computer vision and pattern recognition (CVPR'05)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 conference of the North American chapter of the association for computational lin- guistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gorishniy et al., 2021] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural in- formation processing systems, 34:18932-18943, 2021.","title":"Revisiting deep learning models for tabular data","authors":"Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, Artem Babenko","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson et al., 2016] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo An- thony Celi, and Roger G Mark. Mimic-iii, a freely ac- cessible critical care database. Scientific data, 3(1):1-9, 2016.","title":"Mimic-iii, a freely accessible critical care database","authors":"E W Alistair, Tom J Johnson, Lu Pollard, Li-Wei H Shen, Mengling Lehman, Mohammad Feng, Benjamin Ghassemi, Peter Moody, Leo Szolovits, Roger G Anthony Celi, Mark","year":"2016","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karpukhin and Savchenko, 2026] Ivan Karpukhin and An- drey Savchenko. Detecting the future: All-at-once event sequence forecasting with horizon matching. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 40, pages 22536-22544, 2026.","title":"Ivan Karpukhin and Andrey Savchenko. Detecting the future: All-at-once event sequence forecasting with horizon matching","authors":"Savchenko Karpukhin","year":"2026","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. Advances in neural information process- ing systems, 30, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The Twelfth International Conference on Learning Represen- tations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McDermott et al., 2023] Matthew McDermott, Bret Nestor, Peniel Argaw, and Isaac S Kohane. Event stream gpt: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events. Advances in Neural Information Pro- cessing Systems, 36:24322-24334, 2023.","title":"Event stream gpt: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events","authors":"Matthew Mcdermott, Bret Nestor, Peniel Argaw, Isaac S Kohane","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Muennighoff et al., 2023] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Confer- ence of the European Chapter of the Association for Com- putational Linguistics, pages 2014-2037, 2023.","title":"Mteb: Massive text embedding benchmark","authors":"Muennighoff","year":"2023","venue":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Osin et al., 2025] Dmitry Osin, Igor Udovichenko, Egor Shvetsov, Viktor Moskvoretskii, and Evgeny Burnaev. Ebes: Easy benchmarking for event sequences. In Pro- ceedings of the 31st ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining V. 2, pages 5730-5741, 2025.","title":"Ebes: Easy benchmarking for event sequences","authors":"Dmitry Osin, Igor Udovichenko, Egor Shvetsov, Viktor Moskvoretskii, Evgeny Burnaev","year":"2025","venue":"Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Padhi et al., 2021] Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3565-3569. IEEE, 2021.","title":"Tabular transformers for modeling multivariate time series","authors":"Padhi","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rizoiu et al., 2017] Marian-Andrei Rizoiu, Young Lee, Swapnil Mishra, and Lexing Xie. Hawkes processes for events in social media. In Frontiers of multimedia re- search, pages 191-218. 2017.","title":"Hawkes processes for events in social media","authors":"Rizoiu","year":"2017","venue":"Frontiers of multimedia research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Robinson et al., 2021] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solu- tions? Advances in neural information processing systems, 34:4974-4986, 2021.","title":"Can contrastive learning avoid shortcut solutions?","authors":"Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, Suvrit Sra","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sakhno et al., 2025] Artem Sakhno, Ivan Kireev, Dmitrii Babaev, Maxim Savchenko, Gleb Gusev, and Andrey Savchenko. PyTorch-Lifestream: Learning embeddings on discrete event sequences. In James Kwok, editor, Pro- ceedings of the Thirty-Fourth International Joint Confer- ence on Artificial Intelligence, IJCAI-25, pages 11104- 11108. International Joint Conferences on Artificial Intel- ligence Organization, 8 2025. Demo Track.","title":"PyTorch-Lifestream: Learning embeddings on discrete event sequences","authors":"Artem Sakhno, Ivan Kireev, Dmitrii Babaev, Maxim Savchenko, Gleb Gusev, Andrey Savchenko","year":"2025","venue":"Proceedings of the Thirty-Fourth International Joint Confer-ence on Artificial Intelligence, IJCAI-25","doi":null,"url":null,"internal_url":"/proceedings/2025/1272"},{"position":23,"raw":"[Shchur et al., 2020] Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of tempo- ral point processes. International Conference on Learning Representations (ICLR), 2020.","title":"Intensity-free learning of temporal point processes","authors":"Oleksandr Shchur, Marin Biloš, Stephan Günnemann","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Snoek et al., 2012] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of ma- chine learning algorithms. Advances in neural information processing systems, 25, 2012.","title":"Practical bayesian optimization of machine learning algorithms","authors":"Snoek","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Stankevičius and Lukoševičius, 2024] Lukas Stankevičius and Mantas Lukoševičius. Extracting sentence embed- dings from pretrained transformer models. Applied Sci- ences, 14(19):8887, 2024.","title":"Lukas Stankevičius and Mantas Lukoševičius. Extracting sentence embeddings from pretrained transformer models","authors":"Lukoševičius Stankevičius","year":"2024","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Synerise, 2025] Synerise. Recsys challenge 2025. https: //recsys.synerise.com/, 2025. Accessed: 2025-07-25.","title":"Synerise. Recsys challenge 2025","authors":"Synerise","year":"2025","venue":null,"doi":null,"url":"https://recsys.synerise.com/,2025","internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2019] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Confer- ence on Learning Representations, 2019.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R Bowman","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xue et al., 2024] Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Caigao JIANG, Chen Pan, James Y Zhang, Qingsong Wen, et al. Easytpp: To- wards open benchmarking temporal point processes. In The Twelfth International Conference on Learning Repre- sentations, 2024.","title":"Easytpp: Towards open benchmarking temporal point processes","authors":"Siqiao Xue, Xiaoming Shi, Zhixuan Chu, Yan Wang, Hongyan Hao, Fan Zhou, Jiang Caigao, Chen Pan, James Y Zhang, Qingsong Wen","year":"2024","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2022] Chenghao Yang, Hongyuan Mei, and Ja- son Eisner. Transformer embeddings of irregularly spaced events and their participants. In Proceedings of the Tenth International Conference on Learning Representa- tions (ICLR), 2022.","title":"Transformer embeddings of irregularly spaced events and their participants","authors":"Yang","year":"2022","venue":"Proceedings of the Tenth International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yenduri et al., 2024] Gokul Yenduri, Muthukumaran Ra- malingam, G Chemmalar Selvi, Y Supriya, Gautam Sri- vastava, Praveen Kumar Reddy Maddikunta, G Deepti Raj, Rutvij H Jhaveri, B Prabadevi, Weizheng Wang, et al. Gpt (generative pre-trained transformer)-a comprehen- sive review on enabling technologies, potential applica- tions, emerging challenges, and future directions. IEEE access, 12:54608-54649, 2024.","title":"Gpt (generative pre-trained transformer)-a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions","authors":"Gokul Yenduri, Muthukumaran Ramalingam, G Chemmalar Selvi, Y Supriya, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, G Deepti Raj, H Rutvij, B Jhaveri, Weizheng Prabadevi, Wang","year":"2024","venue":"IEEE access","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2025] Caojin Zhang, Qiang Zhang, Ke Li, Sai Vidyaranya Nuthalapati, Benyu Zhang, Jason Liu, Ser- ena Li, Lizhu Zhang, and Xiangjun Fan. Gem: Empower- ing llm for both embedding generation and language un- derstanding. arXiv preprint arXiv:2506.04344, 2025.","title":"Gem: Empowering llm for both embedding generation and language understanding","authors":"Caojin Zhang, Qiang Zhang, Ke Li, Sai Vidyaranya Nuthalapati, Benyu Zhang, Jason Liu, Serena Li, Lizhu Zhang, Xiangjun Fan","year":"2025","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zuo et al., 2020] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes pro- cess. In International conference on machine learning, pages 11692-11702. PMLR, 2020.","title":"Transformer hawkes process","authors":"Zuo","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}