{"type":"ScholarlyArticle","title":"Skip-Timeformer: Skip-Time Interaction Transformer for Long Sequence Time-Series Forecasting","authors":[{"name":"Wenchang Zhang","slug":"wenchang-zhang","given":"Wenchang","family":"Zhang","affiliation":"School of Computer Science and Technology , Shandong Technology and Business University , Yantai 264005 , China"},{"name":"Hua Wang","slug":"hua-wang","given":"Hua","family":"Wang","affiliation":"School of Information and Electrical Engineering , Ludong University , Yantai 264025 , China"},{"name":"Fan Zhang","slug":"fan-zhang","given":"Fan","family":"Zhang","affiliation":"School of Computer Science and Technology , Shandong Technology and Business University , Yantai 264005 , China; Shandong Future Intelligent Financial Engineering Laboratory , Yantai 264005 , China"}],"abstract":"Recent studies have raised questions about the suitability of the Transformer architecture for long sequence time-series forecasting. These forecasting models leverage Transformers to capture dependencies between multiple time steps in a time series, with embedding tokens composed of data from individual time steps. However, challenges arise when applying Transformers to predict long sequences with strong periodicity, leading to performance degradation and increased computational burden. Furthermore, embedding tokens formed one time step at a time may struggle to reveal meaningful information in long sequences, failing to capture correlations between different time steps. In this study, we propose Skip-Timeformer, a Transformer-based model that utilizes a skip-time interaction for long sequence time-series forecasting. Specifically, we decompose the time series into multiple subsequences based on different time intervals, embedding various time steps into variable tokens across multiple sequences. The skip-time interaction mechanism utilizes these variable tokens to capture dependencies in the skip-time dimension. Additionally, skip-time interaction is employed to learn dependencies between sequences missed by multiple skip time steps. The Skip-Timeformer model demonstrates state-of-the-art performance on various real-world datasets, further enhancing the long sequence forecasting capabilities of the Transformer variations and better adapting to arbitrary lookback windows.","doi":"10.24963/ijcai.2024/608","url":"https://ijcai.org/proceedings/2024/608","pdf":"https://ijcai.org/proceedings/2024/0608.pdf","markdown":"https://ijcai.org/proceedings/2024/608.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/608","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":608,"pages":{"first":5499,"last":5507},"keywords":["Machine Learning: ML: Time series and data streams"],"videos":[],"references":[{"position":1,"raw":"[Angryk et al., 2020] Rafal A Angryk, Petrus C Martens, Berkay Aydin, Dustin Kempton, Sushant S Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, et al. Multi- variate time series dataset for space weather data analytics. Scientific data, 7(1):227, 2020.","title":"Multivariate time series dataset for space weather data analytics","authors":"Rafal A Angryk, C Petrus, Berkay Martens, Dustin Aydin, Kempton, Sunitha Sushant S Mahajan, Azim Basodi, Xumin Ahmadzadeh, Cai, Filali Soukaina, Shah Boubrahimi, Muhammad Hamdi","year":"2020","venue":"Scientific data","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Box and Jenkins, 1968] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91-109, 1968.","title":"Some recent advances in forecasting and control","authors":"Jenkins ; George Ep Box, Gwilym M Box, Jenkins","year":"1968","venue":"Journal of the Royal Statistical Society. Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dao et al., 2022] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.","title":"Flashattention: Fast and memory-efficient exact attention with io-awareness","authors":"Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, Christopher Ré","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Demirel et al., 2012] Ömer Fahrettin Demirel, Selim Zaim, Ahmet C ¸alis ¸kan, and Pinar Özuyar. Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods. Turkish Journal of Elec- trical Engineering and Computer Sciences, 20(5):695- 711, 2012.","title":"Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods","authors":"Demirel","year":"2012","venue":"Turkish Journal of Electrical Engineering and Computer Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2023] Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. arXiv preprint arXiv:2302.00861, 2023.","title":"Simmtm: A simple pre-training framework for masked time-series modeling","authors":"Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, Mingsheng Long","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin et al., 2023] Ming Jin, Guangsi Shi, Yuan-Fang Li, Qingsong Wen, Bo Xiong, Tian Zhou, and Shirui Pan. How expressive are spectral-temporal graph neural net- works for time series forecasting? arXiv preprint arXiv:2305.06587, 2023.","title":"How expressive are spectral-temporal graph neural networks for time series forecasting","authors":"Ming Jin, Guangsi Shi, Yuan-Fang Li, Qingsong Wen, Bo Xiong, Tian Zhou, Shirui Pan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kalyan et al., 2021] Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. Ammus: A sur- vey of transformer-based pretrained models in natural lan- guage processing. arXiv preprint arXiv:2108.05542, 2021.","title":"Ammus: A survey of transformer-based pretrained models in natural language processing","authors":"Kalyan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khan et al., 2022] Salman Khan, Muzammal Naseer, Mu- nawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.","title":"Transformers in vision: A survey","authors":"Salman Khan, Muzammal Naseer, Munawar Hayat, Fahad Syed Waqas Zamir, Mubarak Shahbaz Khan, Shah","year":"2022","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kornblith et al., 2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529.","title":"Similarity of neural network representations revisited","authors":"Kornblith","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"PMLR, 2019.","title":null,"authors":"Pmlr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neu- ral information processing systems, 32, 2019.","title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","authors":"Shiyang Li, Xiaoyong Jin, Xiyou Yao Xuan, Wenhu Zhou, Yu-Xiang Chen, Xifeng Wang, Yan","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lim and Zohren, 2021] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a sur- vey. Philosophical Transactions of the Royal Society A, 379(2194):20200209, 2021.","title":"Time-series forecasting with deep learning: a survey","authors":"Bryan Lim, Stefan Zohren","year":"2021","venue":"Philosophical Transactions of the Royal Society A","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Shizhan Liu, Hang Yu, Cong Liao, Jian- guo Li, Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Inter- national conference on learning representations, 2021.","title":"Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting","authors":"Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, Schahram Dustdar","year":"2021","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Xin Liu, Junhong Guo, Hua Wang, and Fan Zhang. Prediction of stock market index based on issa-bp neural network. Expert Systems with Applications, 204:117604, 2022.","title":"Prediction of stock market index based on issa-bp neural network","authors":"Xin Liu, Junhong Guo, Hua Wang, Fan Zhang","year":"2022","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nie et al., 2022] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.","title":"A time series is worth 64 words: Long-term forecasting with transformers","authors":"Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, Jayant Kalagnanam","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Patton, 2013] Andrew Patton. Copula methods for forecast- ing multivariate time series. Handbook of economic fore- casting, 2:899-960, 2013.","title":"Copula methods for forecasting multivariate time series","authors":"Andrew Patton","year":"2013","venue":"Handbook of economic forecasting","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2023] Min Wang, Hua Wang, and Fan Zhang. Famc-net: Frequency domain parity correction attention and multi-scale dilated convolution for time series fore- casting. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 2554-2563, 2023.","title":"Famc-net: Frequency domain parity correction attention and multi-scale dilated convolution for time series forecasting","authors":"Wang","year":"2023","venue":"Proceedings of the 32nd ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Woo et al., 2022] Gerald Woo, Chenghao Liu, Doyen Sa- hoo, Akshat Kumar, and Steven Hoi. Etsformer: Expo- nential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.01381, 2022.","title":"Etsformer: Exponential smoothing transformers for time-series forecasting","authors":"Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, Steven Hoi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2020] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. Advances in neural information processing systems, 33:17105-17115, 2020.","title":"Adversarial sparse transformer for time series forecasting","authors":"Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, Junzhou Huang","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecast- ing. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2022a] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022.","title":"Timesnet: Temporal 2d-variation modeling for general time series analysis","authors":"Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2022b] Haixu Wu, Jialong Wu, Jiehui Xu, Jian- min Wang, and Mingsheng Long. Flowformer: Lineariz- ing transformers with conservation flows. arXiv preprint arXiv:2202.06258, 2022.","title":"Flowformer: Linearizing transformers with conservation flows","authors":"Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeng et al., 2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series fore- casting? In Proceedings of the AAAI conference on artifi- cial intelligence, volume 37, pages 11121-11128, 2023.","title":"Are transformers effective for time series forecasting?","authors":"Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu","year":"2023","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Yan, 2022] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension de- pendency for multivariate time series forecasting. In The Eleventh International Conference on Learning Represen- tations, 2022.","title":"Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting","authors":"Yunhao Zhang, Junchi Yan","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2023a] Fan Zhang, Gongguan Chen, Hua Wang, Jinjiang Li, and Caiming Zhang. Multi-scale video super-resolution transformer with polynomial approxima- tion. IEEE Transactions on Circuits and Systems for Video Technology, 2023.","title":"Multi-scale video super-resolution transformer with polynomial approximation","authors":"Zhang","year":"2023","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2023b] Fan Zhang, Tiantian Guo, and Hua Wang. Dfnet: Decomposition fusion model for long se- quence time-series forecasting. Knowledge-Based Sys- tems, 277:110794, 2023.","title":"Dfnet: Decomposition fusion model for long sequence time-series forecasting","authors":"Zhang","year":"2023","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2024a] Fan Zhang, Gongguan Chen, Hua Wang, and Caiming Zhang. Cf-dan: Facial-expression recognition based on cross-fusion dual-attention network. Computational Visual Media, pages 1-16, 2024.","title":"Cf-dan: Facial-expression recognition based on cross-fusion dual-attention network","authors":"Zhang","year":"2024","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2024b] Wenchang Zhang, Hua Wang, and Fan Zhang. Spatio-temporal fourier enhanced heterogeneous graph learning for traffic forecasting. Expert Systems with Applications, 241:122766, 2024.","title":"Spatio-temporal fourier enhanced heterogeneous graph learning for traffic forecasting","authors":"Zhang","year":"2024","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Fre- quency enhanced decomposed transformer for long-term series forecasting. In International Conference on Ma- chine Learning, pages 27268-27286. PMLR, 2022.","title":"Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting","authors":"Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, Rong Jin","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}