{"type":"ScholarlyArticle","title":"Temporal Domain Generalization via Learning Instance-level Evolving Patterns","authors":[{"name":"Yujie Jin","slug":"yujie-jin","given":"Yujie","family":"Jin","affiliation":"School of Computer Science , Peking University , Beijing , China; Key Laboratory of High Confidence Software Technologies , Ministry of Education , Beijing , China"},{"name":"Zhibang Yang","slug":"zhibang-yang","given":"Zhibang","family":"Yang","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education , Beijing , China"},{"name":"Xu Chu","slug":"xu-chu","given":"Xu","family":"Chu","affiliation":"School of Computer Science , Peking University , Beijing , China; Key Laboratory of High Confidence Software Technologies , Ministry of Education , Beijing , China; Center on Frontiers of Computing Studies , Peking University , Beijing , China"},{"name":"Liantao Ma","slug":"liantao-ma","given":"Liantao","family":"Ma","affiliation":"National Engineering Research Center of Software Engineering , Peking University , Beijing , China"}],"abstract":"Temporal Domain Generalization (TDG) aims at learning models under temporally evolving data distributions and achieving generalization to unseen future data distributions following the evolving trend. Existing advanced TDG methods learn the evolving patterns through the collective behaviors observed at the population-level of instances, such as time-varying statistics and parameters, tending to overlook the impact of individual-level instance evolving processes on the decision boundary. However, a major obstacle is that datasets at different timestamps may comprise unrelated instances and there is no inherent existence of the instance-level evolving trajectories, which hinders us from learning how the decision boundary changes. To address the above challenges, we propose a Continuous-Time modelling Optimal Transport trajectories (CTOT) framework in this paper. Specifically, we utilize optimal transport to align the data distributions between each pair of adjacent source domains to construct instance evolving trajectories. Subsequently, they are modelled by a continuous-time model and extrapolated to generate future virtual instances, which help the model to adapt its decision boundary to the future domain. Extensive experiments on multiple classification and regression benchmarks  demonstrate the effectiveness of the proposed CTOT framework. The code and appendix are both available on https://github.com/JinYujie99/CTOT.","doi":"10.24963/ijcai.2024/470","url":"https://ijcai.org/proceedings/2024/470","pdf":"https://ijcai.org/proceedings/2024/0470.pdf","markdown":"https://ijcai.org/proceedings/2024/470.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/470","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":470,"pages":{"first":4255,"last":4263},"keywords":["Machine Learning: ML: Regression","Machine Learning: ML: Multi-task and transfer learning","Machine Learning: ML: Incremental learning","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2023] Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In The Eleventh International Confer- ence on Learning Representations, 2023.","title":"Temporal domain generalization with drift-aware dynamic neural networks","authors":"Guangji Bai, Chen Ling, Liang Zhao","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Biloš et al., 2023] Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In International Conference on Machine Learning, pages 2452-2470. PMLR, 2023.","title":"Modeling temporal data as continuous functions with stochastic process diffusion","authors":"Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, Stephan Günnemann","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blanchard et al., 2021] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):46-100, 2021.","title":"Domain generalization by marginal transfer learning","authors":"Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, Clayton Scott","year":"2021","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bui et al., 2021] Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to en- hance domain generalization. Advances in Neural Infor- mation Processing Systems, 34:21189-21201, 2021.","title":"Exploiting domain-specific features to enhance domain generalization","authors":"Manh-Ha Bui, Toan Tran, Anh Tran, Dinh Phung","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cha et al., 2021] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22418, 2021.","title":"Swad: Domain generalization by seeking flat minima","authors":"Cha","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in neural information pro- cessing systems, 31, 2018.","title":"Neural ordinary differential equations","authors":"Yulia Ricky Tq Chen, Jesse Rubanova, David K Bettencourt, Duvenaud","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chu et al., 2022] Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. Dna: Domain generalization with diversified neural averaging. In International Conference on Machine Learning, pages 4010-4034. PMLR, 2022.","title":"Dna: Domain generalization with diversified neural averaging","authors":"Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, Hong Mei","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Courty et al., 2014] Nicolas Courty, Rémi Flamary, and De- vis Tuia. Domain adaptation with regularized optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14, pages 274-289. Springer, 2014.","title":"Domain adaptation with regularized optimal transport","authors":"Courty","year":"2014","venue":"Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Courty et al., 2017] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 2017.","title":"Joint distribution optimal transportation for domain adaptation","authors":"Courty","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. Advances in neu- ral information processing systems, 26, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Damodaran et al., 2018] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV), pages 447-463, 2018.","title":"Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation","authors":"Benjamin Bharath Bhushan Damodaran, Rémi Kellenberger, Devis Flamary, Nicolas Tuia, Courty","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Deng et al., 2020] Ruizhi Deng, Bo Chang, Marcus A Brubaker, Greg Mori, and Andreas Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. Advances in Neural Information Processing Sys- tems, 33:7805-7815, 2020.","title":"Modeling continuous stochastic processes with dynamic normalizing flows","authors":"Deng","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Flamary et al., 2016] Rémi Flamary, Nicholas Courty, Davis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell, 1(1-40):2, 2016.","title":"Optimal transport for domain adaptation","authors":"Rémi Flamary, Nicholas Courty, Davis Tuia, Alain Rakotomamonjy","year":"2016","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Garg et al., 2021] Vikas Garg, Adam Tauman Kalai, Kat- rina Ligett, and Steven Wu. Learn to expect the unex- pected: Probably approximately correct domain general- ization. In International Conference on Artificial Intelli- gence and Statistics, pages 3574-3582. PMLR, 2021.","title":"Learn to expect the unexpected: Probably approximately correct domain generalization","authors":"Vikas Garg, Adam Tauman Kalai, Katrina Ligett, Steven Wu","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gulrajani and Lopez-Paz, 2021] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.","title":"Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization","authors":"Lopez-Paz Gulrajani","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jin et al., 2022] Yujie Jin, Xu Chu, Yasha Wang, and Wenwu Zhu. Domain generalization through the lens of angular invariance. In International Joint Conference on Artificial Intelligence, 2022.","title":"Domain generalization through the lens of angular invariance","authors":"Yujie Jin, Xu Chu, Yasha Wang, Wenwu Zhu","year":"2022","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/139"},{"position":17,"raw":"[Kantorovich, 1942] Leonid V Kantorovich. On the translo- cation of masses. In Dokl. Akad. Nauk. USSR (NS), vol- ume 37, pages 199-201, 1942.","title":"On the translocation of masses","authors":"Leonid V Kantorovich","year":"1942","venue":"Dokl. Akad. Nauk. USSR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kidger et al., 2021] Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infinite-dimensional gans. In International conference on machine learning, pages 5453-5463. PMLR, 2021.","title":"Neural sdes as infinite-dimensional gans","authors":"Patrick Kidger, James Foster, Xuechen Li, Terry J Lyons","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kidger, 2021] Patrick Kidger. On neural differential equa- tions. PhD thesis, University of Oxford, 2021.","title":"On neural differential equations","authors":"Patrick Kidger","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2018] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.","title":"Learning to generalize: Metalearning for domain generalization","authors":"Da Li, Yongxin Yang, Yi-Zhe Song, Timothy Hospedales","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2020] Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradi- ents for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pages 3870-3882. PMLR, 2020.","title":"Scalable gradients for stochastic differential equations","authors":"Li","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lu et al., 2018] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346-2363, 2018.","title":"Learning under concept drift: A review","authors":"Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, Guangquan Zhang","year":"2018","venue":"IEEE transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mahajan et al., 2021] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal match- ing. In International Conference on Machine Learning, pages 7313-7324. PMLR, 2021.","title":"Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching","authors":"Mahajan","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Monge, 1781] Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pages 666-704, 1781.","title":"Mémoire sur la théorie des déblais et des remblais","authors":"Gaspard Monge","year":"1781","venue":"Mem. Math. Phys. Acad. Royale Sci.","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nasery et al., 2021] Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to gen- eralize along time. Advances in Neural Information Pro- cessing Systems, 34:19198-19209, 2021.","title":"Training for the future: A simple gradient interpolation loss to generalize along time","authors":"Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, Sunita Sarawagi","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ortiz-Jimenez et al., 2019] Guillermo Ortiz-Jimenez, Mireille El Gheche, Effrosyni Simou, Hermina Petric Maretic, and Pascal Frossard. Cdot: Continuous do- main adaptation using optimal transport. arXiv preprint arXiv:1909.11448, 2019.","title":"Cdot: Continuous domain adaptation using optimal transport","authors":"Guillermo Ortiz-Jimenez, Mireille El Gheche, Effrosyni Simou, Hermina Petric Maretic, Pascal Frossard","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peyré et al., 2019] Gabriel Peyré, Marco Cuturi, et al. Com- putational optimal transport: With applications to data sci- ence. Foundations and Trends® in Machine Learning, 11(5-6):355-607, 2019.","title":"Computational optimal transport: With applications to data science","authors":"Gabriel Peyré, Marco Cuturi","year":"2019","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qin et al., 2022] Tiexin Qin, Shiqi Wang, and Haoliang Li. Generalizing to evolving domains with latent structure- aware sequential autoencoder. In International Conference on Machine Learning, pages 18062-18082. PMLR, 2022.","title":"Generalizing to evolving domains with latent structureaware sequential autoencoder","authors":"Qin","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rubanova et al., 2019] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equa- tions for irregularly-sampled time series. Advances in neu- ral information processing systems, 32, 2019.","title":"Latent ordinary differential equations for irregularly-sampled time series","authors":"Yulia Rubanova, Ricky Tq Chen, David K Duvenaud","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Santambrogio, 2015] Filippo Santambrogio. Optimal trans- port for applied mathematicians. Birkäuser, NY, 55(58- 63):94, 2015.","title":"Optimal transport for applied mathematicians","authors":"Filippo Santambrogio","year":"2015","venue":"Birkäuser","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Schirmer et al., 2022] Mona Schirmer, Mazin Eltayeb, Ste- fan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In Interna- tional Conference on Machine Learning, pages 19388- 19405. PMLR, 2022.","title":"Modeling irregular time series with continuous recurrent units","authors":"Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, Maja Rudolph","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tang et al., 2024] Yugui Tang, Kuo Yang, Shujing Zhang, and Zhen Zhang. Wind power forecasting: A temporal do- main generalization approach incorporating hybrid model and adversarial relationship-based training. Applied En- ergy, 355:122266, 2024.","title":"Wind power forecasting: A temporal domain generalization approach incorporating hybrid model and adversarial relationship-based training","authors":"Yugui Tang, Kuo Yang, Shujing Zhang, Zhen Zhang","year":"2024","venue":"Applied Energy","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Terrell and Scott, 1992] George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, pages 1236-1265, 1992.","title":"Variable kernel density estimation","authors":"R George, David W Terrell, Scott","year":"1992","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Torralba and Efros, 2011] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.","title":"Unbiased look at dataset bias","authors":"Efros Torralba, Antonio Torralba, Alexei A Efros","year":"2011","venue":"CVPR 2011","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Torres et al., 2021] Luis Caicedo Torres, Luiz Manella Pereira, and M Hadi Amini. A survey on optimal trans- port for machine learning: Theory and applications. arXiv preprint arXiv:2106.01963, 2021.","title":"A survey on optimal transport for machine learning: Theory and applications","authors":"Luis Caicedo Torres, Luiz Manella Pereira, M Hadi Amini","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2020a] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01807, 2020.","title":"Continuously indexed domain adaptation","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2020b] Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 3622-3626. IEEE, 2020.","title":"Heterogeneous domain generalization via domain mixup","authors":"Wang","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2022] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wen- jun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.","title":"Generalizing to unseen domains: A survey on domain generalization","authors":"Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, Philip Yu","year":"2022","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":"/proceedings/2021/628"},{"position":39,"raw":"[Xie et al., 2023] Mixue Xie, Shuang Li, Longhui Yuan, Chi Harold Liu, and Zehui Dai. Evolving standardization for continual domain generalization over temporal drift. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023.","title":"Evolving standardization for continual domain generalization over temporal drift","authors":"Xie","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zeng et al., 2023] Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, and Boyu Wang. Foresee what you will learn: Data augmentation for domain generalization in non-stationary environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11147-11155, 2023.","title":"Foresee what you will learn: Data augmentation for domain generalization in non-stationary environments","authors":"Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, Boyu Wang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang et al., 2022] Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain general- ization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8024- 8034, 2022.","title":"Towards principled disentanglement for domain generalization","authors":"Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, Eric P Xing","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhao, 2023] Liang Zhao. Influenza outbreak event predic- tion via Twitter. UCI Machine Learning Repository, 2023. DOI: https://doi.org/10.24432/C5CP7V.","title":"Influenza outbreak event prediction via Twitter","authors":"Liang Zhao","year":"2023","venue":"UCI Machine Learning Repository","doi":"10.24432/c5cp7v","url":"https://doi.org/10.24432/C5CP7V","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}