{"type":"ScholarlyArticle","title":"Decoupled Invariant Attention Network for Multivariate Time-series Forecasting","authors":[{"name":"Haihua Xu","slug":"haihua-xu","given":"Haihua","family":"Xu","affiliation":"Department of Computer and Information Science , University of Macau , China; The State Key Laboratory of Internet of Things for Smart City , University of Macau , China"},{"name":"Wei Fan","slug":"wei-fan","given":"Wei","family":"Fan","affiliation":"University of Central Florida , USA"},{"name":"Kun Yi","slug":"kun-yi","given":"Kun","family":"Yi","affiliation":"Beijing Institute of Technology , China"},{"name":"Pengyang Wang","slug":"pengyang-wang","given":"Pengyang","family":"Wang","affiliation":"Department of Computer and Information Science , University of Macau , China; The State Key Laboratory of Internet of Things for Smart City , University of Macau , China"}],"abstract":"To achieve more accurate prediction results in Time Series Forecasting (TSF), it is essential to distinguish between the valuable patterns (invariant patterns) of the spatial-temporal relationship and the patterns that are prone to generate distribution shift (variant patterns), then combine them for forecasting.The existing works, such as transformer-based models and GNN-based models, focus on capturing main forecasting dependencies whether it is stable or not, and they tend to overlook patterns that carry both useful information and distribution shift. In this paper, we propose a model for better forecasting time series: Decoupled Invariant Attention Network (DIAN), which contains two modules to learn spatial and  temporal relationships respectively: 1) Spatial Decoupled Invariant-Variant Learning (SDIVL) to decouple the spatial invariant and variant attention scores, and then leverage convolutional networks to effectively integrate them for subsequent layers; 2) Temporal Augmented Invariant-Variant Learning (TAIVL) to decouple temporal invariant and variant patterns and combine them for further forecasting.In this module, we also design Temporal Intervention Mechanism to create multiple intervened samples by reassembling variant patterns across time stamps to eliminate the spurious impacts of variant patterns.In addition, we propose Joint Optimization to minimize the loss function considering all invariant patterns, variant patterns and intervened patterns so that our model can gain a more stable predictive ability.Extensive experiments on five datasets have demonstrated our superior performance with higher efficiency compared with state-of-the-art methods.","doi":"10.24963/ijcai.2024/275","url":"https://ijcai.org/proceedings/2024/275","pdf":"https://ijcai.org/proceedings/2024/0275.pdf","markdown":"https://ijcai.org/proceedings/2024/275.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/275","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Data Mining","track":"Main Track","number":275,"pages":{"first":2487,"last":2495},"keywords":["Data Mining: DM: Mining spatial and/or temporal data"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2018] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018.","title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","authors":"Bai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2020] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. 2020.","title":"Adaptive graph convolutional recurrent network for traffic forecasting","authors":"Lei Bai, Lina Yao, Can Li, Xianzhi Wang, Can Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beccali et al., 2008] M Beccali, M Cellura, V Lo Brano, and Antonino Marvuglia. Short-term prediction of house- hold electricity consumption: Assessing weather sensitiv- ity in a mediterranean area. Renewable and Sustainable Energy Reviews, 12(8):2040-2065, 2008.","title":"Short-term prediction of household electricity consumption: Assessing weather sensitivity in a mediterranean area","authors":"M Beccali, V Cellura, Antonino Lo Brano, Marvuglia","year":"2008","venue":"Renewable and Sustainable Energy Reviews","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2020a] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecast- ing. Advances in neural information processing systems, 33:17766-17778, 2020.","title":"Spectral temporal graph neural network for multivariate time-series forecasting","authors":"Cao","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2020b] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time-series forecast- ing. Advances in neural information processing systems, 33:17766-17778, 2020.","title":"Spectral temporal graph neural network for multivariate time-series forecasting","authors":"Cao","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Yuzhou Chen, Ignacio Segovia- Dominguez, Baris Coskunuzer, and Yulia Gel. Tamp- s2gcnets: coupling time-aware multipersistence knowl- edge representation with spatio-supra graph convolutional networks for time-series forecasting. In International Conference on Learning Representations, 2021.","title":"Tamps2gcnets: coupling time-aware multipersistence knowledge representation with spatio-supra graph convolutional networks for time-series forecasting","authors":"Chen","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2023] Hao Dong, Zhiyuan Ning, Pengyang Wang, Ziyue Qiao, Pengfei Wang, Yuanchun Zhou, and Yanjie Fu. Adaptive path-memory network for temporal knowledge graph reasoning. arXiv preprint arXiv:2304.12604, 2023.","title":"Adaptive path-memory network for temporal knowledge graph reasoning","authors":"Hao Dong, Zhiyuan Ning, Pengyang Wang, Ziyue Qiao, Pengfei Wang, Yuanchun Zhou, Yanjie Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/232"},{"position":8,"raw":"[Dong et al., 2024] Hao Dong, Pengyang Wang, Meng Xiao, Zhiyuan Ning, Pengfei Wang, and Yuanchun Zhou. Tem- poral inductive path neural network for temporal knowl- edge graph reasoning. Artificial Intelligence, 329:104085, 2024.","title":"Temporal inductive path neural network for temporal knowl-edge graph reasoning","authors":"Hao Dong, Pengyang Wang, Meng Xiao, Zhiyuan Ning, Pengfei Wang, Yuanchun Zhou","year":"2024","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fan et al., 2022] Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. Depts: deep expansion learning for periodic time series forecast- ing. arXiv preprint arXiv:2203.07681, 2022.","title":"Depts: deep expansion learning for periodic time series forecasting","authors":"Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, Tie-Yan Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fan et al., 2023] Wei Fan, Pengyang Wang, Dongkun Wang, Dongjie Wang, Yuanchun Zhou, and Yanjie Fu. Dish-ts: A general paradigm for alleviating distribution shift in time series forecasting, 2023.","title":"Dish-ts: A general paradigm for alleviating distribution shift in time series forecasting","authors":"Wei Fan, Pengyang Wang, Dongkun Wang, Dongjie Wang, Yuanchun Zhou, Yanjie Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Flunkert et al., 2020] Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with au- toregressive recurrent networks. International Journal of Forecasting, 36(3), 2020.","title":"Deepar: Probabilistic forecasting with autoregressive recurrent networks","authors":"Valentin Flunkert, David Salinas, Jan Gasthaus","year":"2020","venue":"International Journal of Forecasting","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2023] Xuanming Hu, Wei Fan, Kun Yi, Pengfei Wang, Yuanbo Xu, Yanjie Fu, and Pengyang Wang. Boost- ing urban prediction via addressing spatial-temporal dis- tribution shift. In 2023 IEEE International Conference on Data Mining (ICDM), pages 160-169. IEEE, 2023.","title":"Boosting urban prediction via addressing spatial-temporal distribution shift","authors":"Xuanming Hu, Wei Fan, Kun Yi, Pengfei Wang, Yuanbo Xu, Yanjie Fu, Pengyang Wang","year":"2023","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2022] Song Jiang, Tahin Syed, Xuan Zhu, Joshua Levy, Boris Aronchik, and Yizhou Sun. Bridging self-attention and time series decomposition for periodic forecasting. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 3202-3211, 2022.","title":"Bridging self-attention and time series decomposition for periodic forecasting","authors":"Song Jiang, Tahin Syed, Xuan Zhu, Joshua Levy, Boris Aronchik, Yizhou Sun","year":"2022","venue":"Proceedings of the 31st ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2018] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks, 2018.","title":"Modeling long-and short-term temporal patterns with deep neural networks","authors":"Lai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lim and Zohren, 2021] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a sur- vey. Philosophical Transactions of the Royal Society A, 379(2194):20200209, 2021.","title":"Time-series forecasting with deep learning: a survey","authors":"Bryan Lim, Stefan Zohren","year":"2021","venue":"Philosophical Transactions of the Royal Society A","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oreshkin et al., 2019] B. N. Oreshkin, D. Carpov, N. Cha- pados, and Y. Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. 2019.","title":"N-beats: Neural basis expansion analysis for interpretable time series forecasting","authors":"B N Oreshkin, D Carpov, N Chapados, Y Bengio","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2022] Weijieying Ren, Pengyang Wang, Xiaolin Li, Charles E Hughes, and Yanjie Fu. Semi-supervised drifted stream learning with short lookback. In Proceed- ings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1504-1513, 2022.","title":"Semi-supervised drifted stream learning with short lookback","authors":"Ren","year":"2022","venue":"Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sen et al., 2019] Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series fore- casting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32.","title":"Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting","authors":"Rajat Sen, Hsiang-Fu Yu, Inderjit S Dhillon","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2022] Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time se- ries forecasting. Advances in Neural Information Process- ing Systems, 35:38775-38787, 2022.","title":"Learning latent seasonal-trend representations for time series forecasting","authors":"Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, Fan Zhou","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Watson, 1993] Mark W. Watson. Vector autoregressions and cointegration. Working Paper Series, Macroeconomic Is- sues, 4, 1993.","title":"Vector autoregressions and cointegration","authors":"Mark W Watson","year":"1993","venue":"Working Paper Series, Macroeconomic Issues","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Woo et al., 2022a] Gerald Woo, Chenghao Liu, Doyen Sa- hoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575, 2022.","title":"Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting","authors":"Woo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Woo et al., 2022b] Gerald Woo, Chenghao Liu, Doyen Sa- hoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575, 2022.","title":"Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting","authors":"Woo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2019] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence, IJCAI-19, pages 1907-1913. Interna- tional Joint Conferences on Artificial Intelligence Orga- nization, 7 2019.","title":"Graph wavenet for deep spatial-temporal graph modeling","authors":"Wu","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/264"},{"position":26,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discov- ery & data mining, pages 753-763, 2020.","title":"Connecting the dots: Multivariate time series forecasting with graph neural networks","authors":"Wu","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2021] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A com- prehensive survey on graph neural networks. IEEE trans- actions on neural networks and learning systems, (1):32, 2021.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2021","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2022] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting, 2022.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yi et al., 2024a] Kun Yi, Qi Zhang, Wei Fan, Hui He, Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhen- dong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. Advances in Neural Information Processing Systems, 36, 2024.","title":"Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective","authors":"Yi","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yi et al., 2024b] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecast- ing. Advances in Neural Information Processing Systems, 36, 2024.","title":"Frequency-domain mlps are more effective learners in time series forecasting","authors":"Yi","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2017] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875, 2017.","title":"Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting","authors":"Yu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/505"},{"position":32,"raw":"[Zeng et al., 2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series fore- casting? 2023.","title":"Are transformers effective for time series forecasting?","authors":"Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 2141-2149, 2017.","title":"Stock price prediction via discovering multifrequency trading patterns","authors":"Zhang","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2022] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In S. Koyejo, S. Mohamed, A. Agarwal, D. Bel- grave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 6074- 6089.","title":"Dynamic graph neural networks under spatio-temporal distribution shift","authors":"Zhang","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"Curran Associates, Inc., 2022.","title":null,"authors":"Curran Associates, Inc","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2024] Zhaofan Zhang, Yanan Xiao, Lu Jiang, Dingqi Yang, Minghao Yin, and Pengyang Wang. Spatial- temporal interplay in human mobility: A hierarchical rein- forcement learning approach with hypergraph representa- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 9396-9404, 2024.","title":"Spatialtemporal interplay in human mobility: A hierarchical reinforcement learning approach with hypergraph representation","authors":"Zhang","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zheng et al., 2015] Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In Proceedings of the 21th ACM SIGKDD international con- ference on knowledge discovery and data mining, pages 2267-2276, 2015.","title":"Forecasting fine-grained air quality based on big data","authors":"Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, Tianrui Li","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2022a] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2022.","title":"Domain generalization: A survey","authors":"Zhou","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhou et al., 2022b] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. 2022.","title":"Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting","authors":"Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}