{"type":"ScholarlyArticle","title":"SDformer: Transformer with Spectral Filter and Dynamic Attention for Multivariate Time Series Long-term Forecasting","authors":[{"name":"Ziyu Zhou","slug":"ziyu-zhou","given":"Ziyu","family":"Zhou","affiliation":"Faculty of Information Technology , Beijing University of Technology , Beijing , China"},{"name":"Gengyu Lyu","slug":"gengyu-lyu","given":"Gengyu","family":"Lyu","affiliation":"Faculty of Information Technology , Beijing University of Technology , Beijing , China"},{"name":"Yiming Huang","slug":"yiming-huang","given":"Yiming","family":"Huang","affiliation":"Faculty of Information Technology , Beijing University of Technology , Beijing , China"},{"name":"Zihao Wang","slug":"zihao-wang","given":"Zihao","family":"Wang","affiliation":"Faculty of Information Technology , Beijing University of Technology , Beijing , China"},{"name":"Ziyu Jia","slug":"ziyu-jia","given":"Ziyu","family":"Jia","affiliation":"Institute of Automation , Chinese Academy of Sciences , Beijing , China"},{"name":"Zhen Yang","slug":"zhen-yang","given":"Zhen","family":"Yang","affiliation":"Faculty of Information Technology , Beijing University of Technology , Beijing , China"}],"abstract":"Transformer has gained widespread adoption in modeling time series due to the exceptional ability of its self-attention mechanism in capturing long-range dependencies. However, when processing time series data with numerous variates, the vanilla self-attention mechanism tends to distribute attention weights evenly and smoothly, causing row-homogenization in attention maps and further hampering time series forecasting. To tackle this issue, we propose an advanced Transformer architecture entitled SDformer, which designs two novel modules, Spectral-Filter-Transform (SFT) and Dynamic-Directional-Attention (DDA), and integrates them into the encoder of Transformer to achieve more intensive attention allocation. Specifically, the SFT module utilizes the Fast Fourier Transform to select the most prominent frequencies, along with a Hamming Window to smooth and denoise the filtered series data; The DDA module applies a specialized kernel function to the query and key vectors projected from the denoised data, concentrating this innovative attention mechanism more effectively on the most informative variates to obtain a sharper attention distribution. These two modules jointly enable attention weights to be more salient among numerous variates, which in turn enhances the attention's ability to capture multivariate correlations, improving the performance in forecasting. Extensive experiments on public datasets demonstrate its superior performance over other state-of-the-art models. Code is available at https://github.com/zhouziyu02/SDformer.","doi":"10.24963/ijcai.2024/629","url":"https://ijcai.org/proceedings/2024/629","pdf":"https://ijcai.org/proceedings/2024/0629.pdf","markdown":"https://ijcai.org/proceedings/2024/629.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/629","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":629,"pages":{"first":5689,"last":5697},"keywords":["Machine Learning: ML: Attention models","Data Mining: DM: Mining spatial and/or temporal data","Machine Learning: ML: Time series and data streams"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2020] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, et al. Spectral temporal graph neural network for multivariate time-series forecasting. Advances in Neural Information Processing Systems, 33:17766- 17778, 2020.","title":"Spectral temporal graph neural network for multivariate time-series forecasting","authors":"Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2023] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contif- ormer: Continuous-time transformer for irregular time se- ries modeling. Advances in Neural Information Processing Systems, pages 1-33, 2023.","title":"Contiformer: Continuous-time transformer for irregular time series modeling","authors":"Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, Dongsheng Li","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2023] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. In Advances in Neural Information Processing Systems, pages 1-30, 2023.","title":"Simmtm: A simple pre-training framework for masked time-series modeling","authors":"Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2023] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vi- sion transformer using focused linear attention. In IEEE/CVF International Conference on Computer Vision, pages 5961-5971, 2023.","title":"Flatten transformer: Vision transformer using focused linear attention","authors":"Han","year":"2023","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2023] Yiming Huang, Ziyu Zhou, Zihao Wang, et al. Timesnet-pm2. 5: Interpretable timesnet for disentangling intraperiod and interperiod variations in pm2. 5 prediction. Atmosphere, 14(11):1604, 2023.","title":"Timesnet-pm2. 5: Interpretable timesnet for disentangling intraperiod and interperiod variations in pm2. 5 prediction","authors":"Yiming Huang, Ziyu Zhou, Zihao Wang","year":"2023","venue":"Atmosphere","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, pages 1-15, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kitaev et al., 2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.","title":"Reformer: The efficient transformer","authors":"Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2021] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, et al. Pyraformer: Low-complexity pyramidal atten- tion for long-range time series modeling and forecasting. In International Conference on Learning Representations, pages 1-20, 2021.","title":"Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting","authors":"Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2023] Liu, Tengge Hu, Haoran Zhang, Haixu Wu, et al. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, pages 1-24, 2023.","title":"itransformer: Inverted transformers are effective for time series forecasting","authors":"Tengge Liu, Haoran Hu, Haixu Zhang, Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mottaghi-Kashtiban and Shayesteh, 2011] Mahdi Mottaghi-Kashtiban and Mahrokh G Shayesteh. New efficient window function, replacement for the hamming window. IET Signal Processing, 5(5):499-505, 2011.","title":"Mahdi Mottaghi-Kashtiban and Mahrokh G Shayesteh. New efficient window function, replacement for the hamming window","authors":"Shayesteh Mottaghi-Kashtiban","year":"2011","venue":"IET Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nie et al., 2023] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, pages 1-24, 2023.","title":"A time series is worth 64 words: Long-term forecasting with transformers","authors":"Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, Jayant Kalagnanam","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[O'Connor et al., 2010] Brendan O'Connor, Ramnath Bala- subramanyan, Bryan Routledge, and Noah Smith. From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the international AAAI con- ference on web and social media, volume 4, pages 122- 129, 2010.","title":"From tweets to polls: Linking text sentiment to public opinion time series","authors":"O' Brendan, Ramnath Connor, Bryan Balasubramanyan, Noah Routledge, Smith","year":"2010","venue":"Proceedings of the international AAAI conference on web and social media","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention is all you need. Advances in Neural Information Processing Systems, 30:6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wen et al., 2023] Qingsong Wen, Tian Zhou, Chaoli Zhang, et al. Transformers in time series: A survey. In Inter- national Joint Conference on Artificial Intelligence, pages 6778-6786, 2023.","title":"Transformers in time series: A survey","authors":"Qingsong Wen, Tian Zhou, Chaoli Zhang","year":"2023","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/759"},{"position":15,"raw":"[Woo et al., 2022a] Gerald Woo, Chenghao Liu, Doyen Sa- hoo, Akshat Kumar, et al. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Rep- resentations, pages 1-18, 2022.","title":"CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting","authors":"Woo","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Woo et al., 2022b] Gerald Woo, Chenghao Liu, Doyen Sa- hoo, Akshat Kumar, and Steven Hoi. Etsformer: Expo- nential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, pages 1-18, 2022.","title":"Etsformer: Exponential smoothing transformers for time-series forecasting","authors":"Woo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecast- ing. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Wu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wu et al., 2023] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Rep- resentations, pages 1-23, 2023.","title":"Timesnet: Temporal 2d-variation modeling for general time series analysis","authors":"Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, Mingsheng Long","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2020] Zhangjing Yang, Weiwu Yan, Xiaolin Huang, et al. Adaptive temporal-frequency network for time-series forecasting. IEEE Transactions on Knowledge and Data Engineering, 34(4):1576-1587, 2020.","title":"Adaptive temporal-frequency network for time-series forecasting","authors":"Yang","year":"2020","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yi et al., 2023] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, et al. Frequency-domain MLPs are more effective learners in time series forecasting. In Advances in Neural Informa- tion Processing Systems, pages 1-24, 2023.","title":"Frequency-domain MLPs are more effective learners in time series forecasting","authors":"Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang","year":"2023","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeng et al., 2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series fore- casting? AAAI Conference on Artificial Intelligence, page 11121-11128, 2023.","title":"Are transformers effective for time series forecasting","authors":"Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu","year":"2023","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Min- ing, pages 2141-2149, 2017.","title":"Stock price prediction via discovering multifrequency trading patterns","authors":"Zhang","year":"2017","venue":"ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, et al. Informer: Beyond efficient trans- former for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence, volume 35, pages 11106-11115, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou et al., 2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, et al. Fedformer: Frequency enhanced de- composed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268-27286, 2022.","title":"Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting","authors":"Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}