{"type":"ScholarlyArticle","title":"History Repeats Itself: A Baseline for Temporal Knowledge Graph Forecasting","authors":[{"name":"Julia Gastinger","slug":"julia-gastinger","given":"Julia","family":"Gastinger","affiliation":"NEC Laboratories Europe; University of Mannheim"},{"name":"Christian Meilicke","slug":"christian-meilicke","given":"Christian","family":"Meilicke","affiliation":"University of Mannheim"},{"name":"Federico Errica","slug":"federico-errica","given":"Federico","family":"Errica","affiliation":"NEC Laboratories Europe"},{"name":"Timo Sztyler","slug":"timo-sztyler","given":"Timo","family":"Sztyler","affiliation":"NEC Laboratories Europe"},{"name":"Anett Schülke","slug":"anett-schulke","given":"Anett","family":"Schülke","affiliation":"NEC Laboratories Europe"},{"name":"Heiner Stuckenschmidt","slug":"heiner-stuckenschmidt","given":"Heiner","family":"Stuckenschmidt","affiliation":"University of Mannheim"}],"abstract":"Temporal Knowledge Graph (TKG) Forecasting aims at predicting links in Knowledge Graphs for future timesteps based on a history of Knowledge Graphs. To this day, standardized evaluation protocols and rigorous comparison across TKG models are available, but the importance of simple baselines is often neglected in the evaluation, which prevents researchers from discerning actual and fictitious progress. We propose to close this gap by designing an intuitive baseline for TKG Forecasting based on predicting recurring facts. Compared to most TKG models, it requires little hyperparameter tuning and no iterative training. Further, it can help to identify failure modes in existing approaches. The empirical findings are quite unexpected: compared to 11 methods on five datasets, our baseline ranks first or third in three of them, painting a radically different picture of the predictive quality of the state of the art.","doi":"10.24963/ijcai.2024/444","url":"https://ijcai.org/proceedings/2024/444","pdf":"https://ijcai.org/proceedings/2024/0444.pdf","markdown":"https://ijcai.org/proceedings/2024/444.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/444","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":444,"pages":{"first":4016,"last":4024},"keywords":["Machine Learning: ML: Evaluation","Data Mining: DM: Knowledge graphs and knowledge base completion","Machine Learning: ML: Sequence and graph learning"],"videos":[],"references":[{"position":1,"raw":"[Dacrema et al., 2019] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really mak- ing much progress? A worrying analysis of recent neu- ral recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys), pages 101-109, 2019.","title":"Are we really making much progress? A worrying analysis of recent neural recommendation approaches","authors":"Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach","year":"2019","venue":"Proceedings of the 13th ACM Conference on Recommender Systems (RecSys)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Errica et al., 2020] Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. In 8th Interna- tional Conference on Learning Representations (ICLR), 2020.","title":"A fair comparison of graph neural networks for graph classification","authors":"Federico Errica, Marco Podda, Davide Bacciu, Alessio Micheli","year":"2020","venue":"8th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Errica, 2023] Federico Errica. On class distributions in- duced by nearest neighbor graphs for node classification of tabular data. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS), 2023.","title":"Federico Errica. On class distributions induced by nearest neighbor graphs for node classification of tabular data","authors":"Errica","year":"2023","venue":"Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gastinger et al., 2023] Julia Gastinger, Timo Sztyler, Lokesh Sharma, Anett Schuelke, and Heiner Stuck- enschmidt. Comparing apples and oranges? On the evaluation of methods for temporal knowledge graph forecasting. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pages 533-549, 2023.","title":"Comparing apples and oranges? On the evaluation of methods for temporal knowledge graph forecasting","authors":"Julia Gastinger, Timo Sztyler, Lokesh Sharma, Anett Schuelke, Heiner Stuckenschmidt","year":"2023","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2021a] Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Explainable subgraph reasoning for fore- casting on temporal knowledge graphs. In 9th Interna- tional Conference on Learning Representations (ICLR), 2021.","title":"Explainable subgraph reasoning for forecasting on temporal knowledge graphs","authors":"Han","year":"2021","venue":"9th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2021b] Zhen Han, Zifeng Ding, Yunpu Ma, Yu- jia Gu, and Volker Tresp. Learning neural ordinary equa- tions for forecasting future links on temporal knowledge graphs. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 8352-8364, 2021.","title":"Learning neural ordinary equations for forecasting future links on temporal knowledge graphs","authors":"Han","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han, 2022] Zhen Han. Relational learning on temporal knowledge graphs. Phd thesis, Lud- wig-Maximilians-University, Munich, Germany, 2022.","title":"Relational learning on temporal knowledge graphs","authors":"Han ;, Zhen Han","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2023] Shenyang Huang, Farimah Poursafaei, Jacob Danovitch, Matthias Fey, Weihua Hu, Emanuele Rossi, Jure Leskovec, Michael Bronstein, Guillaume Rabusseau, and Reihaneh Rabbany. Temporal graph benchmark for machine learning on temporal graphs. In 37th Conference on Neural Information Processing Sys- tems (NeurIPS), Datasets and Benchmarks Track, 2023.","title":"Temporal graph benchmark for machine learning on temporal graphs","authors":"Huang","year":"2023","venue":"37th Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2020] Woojeong Jin, Meng Qu, Xisen Jin, and Xi- ang Ren. Recurrent event network: Autoregressive struc- ture inferenceover temporal knowledge graphs. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6669- 6683, 2020.","title":"Recurrent event network: Autoregressive structure inferenceover temporal knowledge graphs","authors":"Jin","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiran et al., 2023] Rage Uday Kiran, Abinash Maharana, and Krishna Reddy Polepalli. A novel explainable link forecasting framework for temporal knowledge graphs us- ing time-relaxed cyclic and acyclic rules. In Proceedings of the 27th Pacific-Asia Conference on Knowledge Discov- ery and Data Mining (PAKDD), Part I, pages 264-275, 2023.","title":"A novel explainable link forecasting framework for temporal knowledge graphs using time-relaxed cyclic and acyclic rules","authors":"Kiran","year":"2023","venue":"Proceedings of the 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), Part I","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2021a] Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng. Search from history and reason for future: Two-stage rea- soning on temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (ACL/IJCNLP), Volume 1: Long Papers, pages 4732-4743, 2021.","title":"Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs","authors":"Li","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021b] Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. In The 44th Inter- national ACM SIGIR Conference on Research and Devel- opment in Information Retrieval (SIGIR), 2021.","title":"Temporal knowledge graph reasoning based on evolutional representation learning","authors":"Li","year":"2021","venue":"The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2022a] Yujia Li, Shiliang Sun, and Jing Zhao. TiRGN: Time-guided recurrent graph network with local- global historical patterns for temporal knowledge graph reasoning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI), pages 2152- 2158, 2022.","title":"TiRGN: Time-guided recurrent graph network with localglobal historical patterns for temporal knowledge graph reasoning","authors":"Li","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2022b] Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. Complex evolutional pat- tern learning for temporal knowledge graph reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 2: Short Pa- pers, pages 290-296, 2022.","title":"Complex evolutional pattern learning for temporal knowledge graph reasoning","authors":"Li","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2023] Qika Lin, Jun Liu, Rui Mao, Fangzhi Xu, and Erik Cambria. TECHS: Temporal logical graph net- works for explainable extrapolation reasoning. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 1281-1293, 2023.","title":"TECHS: Temporal logical graph networks for explainable extrapolation reasoning","authors":"Lin","year":"2023","venue":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lipton and Steinhardt, 2019] Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning schol- arship: Some ML papers suffer from flaws that could mislead the public and stymie future research. Queue, 17(1):45-77, 2019.","title":"Troubling trends in machine learning scholarship: Some ML papers suffer from flaws that could mislead the public and stymie future research","authors":"C Zachary, Jacob Lipton, Steinhardt","year":"2019","venue":"Queue","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022] Yushan Liu, Yunpu Ma, Marcel Hilde- brandt, Mitchell Joblin, and Volker Tresp. TLogic: Tem- poral logical rules for explainable link forecasting on tem- poral knowledge graphs. In 36th Conference on Artificial Intelligence (AAAI), pages 4120-4127, 2022.","title":"TLogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs","authors":"Yushan Liu, Yunpu Ma, Marcel Hildebrandt, Mitchell Joblin, Volker Tresp","year":"2022","venue":"36th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023a] Kangzheng Liu, Feng Zhao, Guandong Xu, Xianzhi Wang, and Hai Jin. RETIA: Relation-entity twin-interact aggregation for temporal knowledge graph extrapolation. In 39th IEEE International Conference on Data Engineering (ICDE), pages 1761-1774, 2023.","title":"RETIA: Relation-entity twin-interact aggregation for temporal knowledge graph extrapolation","authors":"Liu","year":"2023","venue":"39th IEEE International Conference on Data Engineering (ICDE)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2023b] Yuxuan Liu, Yijun Mo, Zhengyu Chen, and Huiyu Liu. LogE-Net: Logic evolution network for temporal knowledge graph forecasting. In International Conference on Artificial Neural Networks (ICANN), pages 472-485, 2023.","title":"LogE-Net: Logic evolution network for temporal knowledge graph forecasting","authors":"Liu","year":"2023","venue":"International Conference on Artificial Neural Networks (ICANN)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mei et al., 2022] Xin Mei, Libin Yang, Xiaoyan Cai, and Zuowei Jiang. An adaptive logical rule embedding model for inductive reasoning over temporal knowledge graphs. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 7304-7316, 2022.","title":"An adaptive logical rule embedding model for inductive reasoning over temporal knowledge graphs","authors":"Xin Mei, Libin Yang, Xiaoyan Cai, Zuowei Jiang","year":"2022","venue":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Meilicke et al., 2018] Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule- and embedding-based systems for knowledge graph com- pletion. In Proceedings of the 17th International Semantic Web Conference (ISWC), Part I 17, pages 3-20, 2018.","title":"Fine-grained evaluation of ruleand embedding-based systems for knowledge graph completion","authors":"Meilicke","year":"2018","venue":"Proceedings of the 17th International Semantic Web Conference (ISWC)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Micheli, 2009] Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498-511, 2009.","title":"Neural network for graphs: A contextual constructive approach","authors":"Alessio Micheli","year":"2009","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Poursafaei et al., 2022] Farimah Poursafaei, Shenyang Huang, Kellin Pelrine, and Reihaneh Rabbany. Towards better evaluation for dynamic link prediction. In 36th Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, pages 32928-32941, 2022.","title":"Towards better evaluation for dynamic link prediction","authors":"Poursafaei","year":"2022","venue":"36th Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rossi et al., 2021] Andrea Rossi, Denilson Barbosa, Do- natella Firmani, Antonio Matinata, and Paolo Merialdo. Knowledge graph embedding for link prediction: A com- parative analysis. ACM Transactions on Knowledge Dis- covery from Data (TKDD), 15(2):1-49, 2021.","title":"Knowledge graph embedding for link prediction: A comparative analysis","authors":"Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, Paolo Merialdo","year":"2021","venue":"ACM Transactions on Knowledge Discovery from Data (TKDD)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Scarselli et al., 2009] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009.","title":"The graph neural network model","authors":"Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, Gabriele Monfardini","year":"2009","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shchur et al., 2018] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evalu- ation. Workshop on Relational Representation Learning, 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.","title":"Pitfalls of graph neural network evaluation","authors":"Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann","year":"2018","venue":"Workshop on Relational Representation Learning, 32nd Conference on Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2020] Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha P. Talukdar, and Yiming Yang. A re- evaluation of knowledge graph completion methods. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5516-5522, 2020.","title":"A reevaluation of knowledge graph completion methods","authors":"Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha P Talukdar, Yiming Yang","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sun et al., 2021] Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. Timetraveler: Reinforcement learning for temporal knowledge graph forecasting. In Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 8306-8319, 2021.","title":"Timetraveler: Reinforcement learning for temporal knowledge graph forecasting","authors":"Sun","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2023] Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. Temporal knowledge graph reasoning with historical con- trastive learning. In 37th Conference on Artificial Intelli- gence (AAAI), pages 4765-4773, 2023.","title":"Temporal knowledge graph reasoning with historical contrastive learning","authors":"Yi Xu, Junjie Ou, Hui Xu, Luoyi Fu","year":"2023","venue":"37th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2023] Mengqi Zhang, Yuwei Xia, Qiang Liu, Shu Wu, and Liang Wang. Learning latent relations for temporal knowledge graph reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (ACL), Volume 1: Long Papers, pages 12617-12631, 2023.","title":"Learning latent relations for temporal knowledge graph reasoning","authors":"Zhang","year":"2023","venue":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2021] Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhang. Learning from history: Modeling temporal knowledge graphs with se- quential copy-generation networks. In 35th Conference on Artificial Intelligence (AAAI), pages 4732-4740, 2021.","title":"Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks","authors":"Zhu","year":"2021","venue":"35th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}