{"type":"ScholarlyArticle","title":"DyGRAIN: An Incremental Learning Framework for Dynamic Graphs","authors":[{"name":"Seoyoon Kim","slug":"seoyoon-kim","given":"Seoyoon","family":"Kim","affiliation":"LG AI Research"},{"name":"Seongjun Yun","slug":"seongjun-yun","given":"Seongjun","family":"Yun","affiliation":"Department of Computer Science and Engineering , Korea University"},{"name":"Jaewoo Kang","slug":"jaewoo-kang","given":"Jaewoo","family":"Kang","affiliation":"Department of Computer Science and Engineering , Korea University"}],"abstract":"Graph-structured data provide a powerful representation of complex relations or interactions. Many variants of graph neural networks (GNNs) have emerged to learn graph-structured data where underlying graphs are static, although graphs in various real-world applications are dynamic (e.g., evolving structure). To consider the dynamic nature that a graph changes over time, the need for applying incremental learning (i.e., continual learning or lifelong learning) to the graph domain has been emphasized. However, unlike incremental learning on Euclidean data, graph-structured data contains dependency between the existing nodes and newly appeared nodes, resulting in the phenomenon that receptive fields of existing nodes vary by new inputs (e.g., nodes and edges). In this paper, we raise a crucial challenge of incremental learning for dynamic graphs as time-varying receptive fields, and propose a novel incremental learning framework, DyGRAIN, to mitigate time-varying receptive fields and catastrophic forgetting. Specifically, our proposed method incrementally learns dynamic graph representations by reflecting the influential change in receptive fields of existing nodes and maintaining previous knowledge of informational nodes prone to be forgotten. Our experiments on large-scale graph datasets demonstrate that our proposed method improves the performance by effectively capturing pivotal nodes and preventing catastrophic forgetting.","doi":"10.24963/ijcai.2022/438","url":"https://ijcai.org/proceedings/2022/438","pdf":"https://ijcai.org/proceedings/2022/0438.pdf","markdown":"https://ijcai.org/proceedings/2022/438.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/438","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":438,"pages":{"first":3157,"last":3163},"keywords":["Data Mining: Mining Graphs","Machine Learning: Incremental Learning","Machine Learning: Sequence and Graph Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986870","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985013","length_sec":335,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aljundi et al., 2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154, 2018.","title":"Memory aware synapses: Learning what (not) to forget","authors":"Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, Tinne Tuytelaars","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berg et al., 2017] Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv:1706.02263, 2017.","title":"Graph convolutional matrix completion","authors":"Rianne Van Den, Thomas N Berg, Max Kipf, Welling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chaudhry et al., 2019a] Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Ef- ficient lifelong learning with a-gem. In ICLR, 2019.","title":"Efficient lifelong learning with a-gem","authors":"Chaudhry","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaudhry et al., 2019b] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajan- than, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv:1902.10486, 2019.","title":"Continual learning with tiny episodic memories","authors":"Chaudhry","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Cao and Kipf, 2018] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv:1805.11973, 2018.","title":"Molgan: An implicit generative model for small molecular graphs","authors":"Nicola De, Cao, Thomas Kipf","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grossberg, 2012] Stephen T Grossberg. Studies of mind and brain: Neural principles of learning, perception, develop- ment, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.","title":"Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control","authors":"Stephen T Grossberg","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hamilton et al., 2017] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Confer- ence on Neural Information Processing Systems, 2017.","title":"Inductive representation learning on large graphs","authors":"Rex William L Hamilton, Jure Ying, Leskovec","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2020] Weihua Hu, Matthias Fey, Marinka Zit- nik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.","title":"Open graph benchmark: Datasets for machine learning on graphs","authors":"Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Rep- resentations, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceed- ings of the national academy of sciences, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klicpera et al., 2019] Johannes Klicpera, Aleksandar Bo- jchevski, and Stephan Günnemann. Predict then propa- gate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019.","title":"Predict then propagate: Graph neural networks meet personalized pagerank","authors":"Klicpera","year":"2019","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neu- ral networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.","title":"Overcoming catastrophic forgetting in graph neural networks","authors":"Huihui Liu, Yiding Yang, Xinchao Wang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ming Chen et al., 2020] Zhewei Wei Ming Chen, Bolin Ding Zengfeng Huang, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, 2020.","title":"Simple and deep graph convolutional networks","authors":"Ming Chen","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qiu et al., 2018] Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jie Tang. Deepinf: Social influ- ence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 2110-2119, 2018.","title":"Deepinf: Social influence prediction with deep learning","authors":"Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, Jie Tang","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rolnick et al., 2019] D. Rolnick, Arun Ahuja, Jonathan Schwarz, T. Lillicrap, and Greg Wayne. Experience re- play for continual learning. In Proceedings of the 33rd In- ternational Conference on Neural Information Processing Systems, 2019.","title":"Experience replay for continual learning","authors":"D Rolnick, Arun Ahuja, Jonathan Schwarz, T Lillicrap, Greg Wayne","year":"2019","venue":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sen et al., 2008] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93-93, 2008.","title":"Collective classification in network data","authors":"Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, Tina Eliassi-Rad","year":"2008","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shin et al., 2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep gen- erative replay. Advances in neural information processing systems, 30, 2017.","title":"Continual learning with deep generative replay","authors":"Hanul Shin, Jung Kwon Lee, Jaehong Kim, Jiwon Kim","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vitter, 1985] Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.","title":"Random sampling with a reservoir","authors":"S Jeffrey, Vitter","year":"1985","venue":"ACM Transactions on Mathematical Software (TOMS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2020] Yishi Xu, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, and Mark Coates. Graph- sail: Graph structure aware incremental learning for rec- ommender systems. In Proceedings of the 29th ACM Inter- national Conference on Information & Knowledge Man- agement, pages 2861-2868, 2020.","title":"Graphsail: Graph structure aware incremental learning for recommender systems","authors":"Yishi Xu, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, Mark Coates","year":"2020","venue":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zenke et al., 2017] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intel- ligence. In International Conference on Machine Learn- ing, pages 3987-3995. PMLR, 2017.","title":"Continual learning through synaptic intelligence","authors":"Friedemann Zenke, Ben Poole, Surya Ganguli","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou and Cao, 2021] Fan Zhou and Chengtai Cao. Over- coming catastrophic forgetting in graph neural networks with experience replay. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 4714- 4722, 2021.","title":"Overcoming catastrophic forgetting in graph neural networks with experience replay","authors":"Cao ; Zhou, Fan Zhou, Chengtai Cao","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}