{"type":"ScholarlyArticle","title":"Transformers in Time Series: A Survey","authors":[{"name":"Qingsong Wen","slug":"qingsong-wen","given":"Qingsong","family":"Wen","affiliation":"DAMO Academy , Alibaba Group , Bellevue , USA"},{"name":"Tian Zhou","slug":"tian-zhou","given":"Tian","family":"Zhou","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Chaoli Zhang","slug":"chaoli-zhang","given":"Chaoli","family":"Zhang","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Weiqi Chen","slug":"weiqi-chen","given":"Weiqi","family":"Chen","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Ziqing Ma","slug":"ziqing-ma","given":"Ziqing","family":"Ma","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Junchi Yan","slug":"junchi-yan","given":"Junchi","family":"Yan","affiliation":"Department of CSE , MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University"},{"name":"Liang Sun","slug":"liang-sun","given":"Liang","family":"Sun","affiliation":"DAMO Academy , Alibaba Group , Bellevue , USA"}],"abstract":"Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance.","doi":"10.24963/ijcai.2023/759","url":"https://ijcai.org/proceedings/2023/759","pdf":"https://ijcai.org/proceedings/2023/0759.pdf","markdown":"https://ijcai.org/proceedings/2023/759.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/759","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Survey Track","track":"Survey Track","number":759,"pages":{"first":6778,"last":6786},"keywords":["Survey: Data Mining","Survey: Machine Learning","Survey: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Bapna et al., 2018] Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In EMNLP, 2018.","title":"Training deeper neural machine translation models with transparent attention","authors":"Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, Yonghui Wu","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benidis et al., 2022] Konstantinos Benidis, Syama Sundar Ranga- puram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, , et al. Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys, 55(6):1-36, 2022.","title":"Deep learning for time series forecasting: Tutorial and literature survey","authors":"Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix","year":"2022","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blázquez-García et al., 2021] Ane Blázquez-García, Angel Conde, Usue Mori, et al. A review on outlier/anomaly detection in time series data. ACM Computing Surveys, 54(3):1-33, 2021.","title":"A review on outlier/anomaly detection in time series data","authors":"Ane Blázquez-García, Angel Conde, Usue Mori","year":"2021","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, et al. Lan- guage models are few-shot learners. NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cai et al., 2020] Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, and Rui Zhu. Traffic transformer: Capturing the conti- nuity and periodicity of time series for traffic forecasting. Trans- actions in GIS, 24(3):736-755, 2020.","title":"Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting","authors":"Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, Rui Zhu","year":"2020","venue":"Transactions in GIS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021a] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, et al. Pre-trained image processing transformer. In CVPR, 2021.","title":"Pre-trained image processing transformer","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021b] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. AutoFormer: Searching transformers for vi- sual recognition. In CVPR, 2021.","title":"AutoFormer: Searching transformers for visual recognition","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021c] Zekai Chen, Dingshuo Chen, Xiao Zhang, Zix- uan Yuan, and Xiuzhen Cheng. Learning graph structures with transformer for multivariate time series anomaly detection in IoT. IEEE Internet of Things Journal, 2021.","title":"Learning graph structures with transformer for multivariate time series anomaly detection in IoT","authors":"Chen","year":"2021","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2022] Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In KDD, 2022.","title":"Learning to rotate: Quaternion transformer for complicated periodical time series forecasting","authors":"Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, Liang Sun","year":"2022","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Choi et al., 2021] Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time- series data: Review, analysis, and guidelines. IEEE Access, 2021.","title":"Deep learning for anomaly detection in timeseries data: Review, analysis, and guidelines","authors":"Choi","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chowdhury et al., 2022] Ranak Roy Chowdhury, Xiyuan Zhang, Jingbo Shang, Rajesh K Gupta, and Dezhi Hong. TARNet: Task- aware reconstruction for time-series transformer. In KDD, 2022.","title":"TARNet: Taskaware reconstruction for time-series transformer","authors":"Chowdhury","year":"2022","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Cirstea et al., 2022] Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Tri- angular, variable-specific attentions for long sequence multivari- ate time series forecasting. In IJCAI, 2022.","title":"Triformer: Triangular, variable-specific attentions for long sequence multivariate time series forecasting","authors":"Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, Shirui Pan","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/277"},{"position":13,"raw":"[Cleveland et al., 1990] Robert Cleveland, William Cleveland, Jean McRae, et al. STL: A seasonal-trend decomposition procedure based on loess. Journal of Official Statistics, 6(1):3-73, 1990.","title":"STL: A seasonal-trend decomposition procedure based on loess","authors":"Robert Cleveland, William Cleveland, Jean Mcrae","year":"1990","venue":"Journal of Official Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, et al. Transformer-XL: Attentive lan- guage models beyond a fixed-length context. In ACL, 2019.","title":"Transformer-XL: Attentive language models beyond a fixed-length context","authors":"Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc V Le","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dehghani et al., 2019] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal trans- formers. In ICLR, 2019.","title":"Universal transformers","authors":"Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Elsken et al., 2019] Elsken, Thomas, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 2019.","title":"Neural architecture search: A survey","authors":"Thomas Elsken, Jan Hendrik Metzen, Frank Hutter","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gao et al., 2022] Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Bernie Wang, Mu Li, et al. Earthformer: Exploring space-time transformers for earth system forecasting. In NeurIPS, 2022.","title":"Earthformer: Exploring space-time transformers for earth system forecasting","authors":"Gao","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grang- ier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In ICML, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, et al. Generative adversarial nets. NeurIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Han et al., 2021] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, et al. Pre- trained models: Past, present and future. AI Open, 2021.","title":"Pretrained models: Past, present and future","authors":"Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang","year":"2021","venue":"AI Open","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Han et al., 2022] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, et al. A survey on vision transformer. IEEE TPAMI, 45(1):87-110, 2022.","title":"A survey on vision transformer","authors":"Han","year":"2022","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Hyndman and Khandakar, 2008] Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast pack- age for r. Journal of statistical software, 27:1-22, 2008.","title":"Automatic time series forecasting: the forecast package for r","authors":"J Rob, Yeasmin Hyndman, Khandakar","year":"2008","venue":"Journal of statistical software","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ismail Fawaz et al., 2019] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery, 2019.","title":"Deep learning for time series classification: a review","authors":"Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller","year":"2019","venue":"Data mining and knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ke et al., 2021] Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In ICLR, 2021.","title":"Rethinking positional encoding in language pre-training","authors":"Guolin Ke, Di He, Tie-Yan Liu","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kenton and others, 2019] Jacob Devlin Ming-Wei Chang Kenton et al. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li et al., 2019] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019.","title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","authors":"Shiyang Li, Xiaoyong Jin, Xiyou Yao Xuan, Wenhu Zhou, Yu-Xiang Chen, Xifeng Wang, Yan","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2021] Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajec- tory prediction. In NeurIPS, 2021.","title":"Grin: Generative relation and intention network for multi-agent trajectory prediction","authors":"Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, Zheng Zhang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Liang et al., 2023] Yuxuan Liang, Yutong Xia, Songyu Ke, Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, and Roger Zim- mermann. AirFormer: Predicting nationwide air quality in china with transformers. In AAAI, 2023.","title":"AirFormer: Predicting nationwide air quality in china with transformers","authors":"Yuxuan Liang, Yutong Xia, Songyu Ke, Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, Roger Zimmermann","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lim and Zohren, 2021] Bryan Lim and Stefan Zohren. Time- series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society, 2021.","title":"Timeseries forecasting with deep learning: a survey","authors":"Bryan Lim, Stefan Zohren","year":"2021","venue":"Philosophical Transactions of the Royal Society","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lim et al., 2021] Bryan Lim, Sercan Ö Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4):1748-1764, 2021.","title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","authors":"Bryan Lim, Ö Sercan, Nicolas Arık, Tomas Loeff, Pfister","year":"2021","venue":"International Journal of Forecasting","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lin et al., 2021] Yang Lin, Irena Koprinska, and Mashud Rana. SSDNet: State space decomposition neural network for time se- ries forecasting. In ICDM, 2021.","title":"SSDNet: State space decomposition neural network for time series forecasting","authors":"Yang Lin, Irena Koprinska, Mashud Rana","year":"2021","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Liu et al., 2021] Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, and Wei Song. Gated trans- former networks for multivariate time series classification. arXiv preprint arXiv:2103.14438, 2021.","title":"Gated transformer networks for multivariate time series classification","authors":"Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, Wei Song","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Liu et al., 2022a] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR, 2022.","title":"Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting","authors":"Liu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Liu et al., 2022b] Yong Liu, Haixu Wu, Jianmin Wang, and Ming- sheng Long. Non-stationary transformers: Exploring the station- arity in time series forecasting. In NeurIPS, 2022.","title":"Non-stationary transformers: Exploring the stationarity in time series forecasting","authors":"Liu","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Mehta et al., 2021] Sachin Mehta, Marjan Ghazvininejad, Srini Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light-weight transformer. In ICLR, 2021.","title":"Delight: Deep and light-weight transformer","authors":"Sachin Mehta, Marjan Ghazvininejad, Srini Iyer, Luke Zettlemoyer, Hannaneh Hajishirzi","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Mei et al., 2022] Hongyuan Mei, Chenghao Yang, and Jason Eis- ner. Transformer embeddings of irregularly spaced events and their participants. In ICLR, 2022.","title":"Transformer embeddings of irregularly spaced events and their participants","authors":"Hongyuan Mei, Chenghao Yang, Jason Eisner","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Nie et al., 2023] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long- term forecasting with transformers. In ICLR, 2023.","title":"A time series is worth 64 words: Longterm forecasting with transformers","authors":"Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, Jayant Kalagnanam","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Rußwurm and Körner, 2020] Marc Rußwurm and Marco Körner. Self-attention for raw optical satellite time series classification. ISPRS J. Photogramm. Remote Sens., 169:421-435, 11 2020.","title":"Self-attention for raw optical satellite time series classification","authors":"Körner Rußwurm, Marc Rußwurm, Marco Körner","year":"2020","venue":"ISPRS J. Photogramm. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Shabani et al., 2023] Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: iterative multi-scale refining trans- formers for time series forecasting. In ICLR, 2023.","title":"Scaleformer: iterative multi-scale refining transformers for time series forecasting","authors":"Amin Shabani, Amir Abdi, Lili Meng, Tristan Sylvain","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Shaw et al., 2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, 2018.","title":"Self-attention with relative position representations","authors":"Peter Shaw, Jakob Uszkoreit, Ashish Vaswani","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Shchur et al., 2021] Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, and Stephan Günnemann. Neural temporal point processes: A review. In IJCAI, 2021.","title":"Neural temporal point processes: A review","authors":"Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, Stephan Günnemann","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/623"},{"position":44,"raw":"[So et al., 2019] David So, Quoc Le, and Chen Liang. The evolved transformer. In ICML, 2019.","title":"The evolved transformer","authors":"David So, Quoc Le, Chen Liang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Tang and Matteson, 2021] Binh Tang and David Matteson. Proba- bilistic transformer for time series analysis. In NeurIPS, 2021.","title":"Probabilistic transformer for time series analysis","authors":"Binh Tang, David Matteson","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Tay et al., 2022] Yi Tay, Mostafa Dehghani, Dara Bahri, and Don- ald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28, 2022.","title":"Efficient transformers: A survey","authors":"Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler","year":"2022","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Torres et al., 2021] José F. Torres, Dalil Hadjout, Abderrazak Se- baa, Francisco Martínez-Álvarez, and Alicia Troncoso. Deep learning for time series forecasting: a survey. Big Data, 2021.","title":"Deep learning for time series forecasting: a survey","authors":"F José, Dalil Torres, Abderrazak Hadjout, Francisco Sebaa, Alicia Martínez-Álvarez, Troncoso","year":"2021","venue":"Big Data","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Tuli et al., 2022] Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. TranAD: Deep transformer networks for anomaly de- tection in multivariate time series data. In VLDB, 2022.","title":"TranAD: Deep transformer networks for anomaly detection in multivariate time series data","authors":"Tuli","year":"2022","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Wang et al., 2020] Xiaoxing Wang, Chao Xue, Junchi Yan, Xi- aokang Yang, Yonggang Hu, et al. MergeNAS: Merge operations into one for differentiable architecture search. In IJCAI, 2020.","title":"MergeNAS: Merge operations into one for differentiable architecture search","authors":"Xiaoxing Wang, Chao Xue, Junchi Yan, Xiaokang Yang, Yonggang Hu","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/424"},{"position":51,"raw":"[Wang et al., 2022] Xixuan Wang, Dechang Pi, Xiangyan Zhang, et al. Variational transformer-based anomaly detection approach for multivariate time series. Measurement, page 110791, 2022.","title":"Variational transformer-based anomaly detection approach for multivariate time series","authors":"Wang","year":"2022","venue":"Measurement","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Wen et al., 2019] Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, et al. RobustSTL: A robust seasonal-trend decomposition algorithm for long time series. In AAAI, 2019.","title":"RobustSTL: A robust seasonal-trend decomposition algorithm for long time series","authors":"Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Wen et al., 2020] Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. Fast RobustSTL: Efficient and robust seasonal-trend decom- position for time series with complex patterns. In KDD, 2020.","title":"Fast RobustSTL: Efficient and robust seasonal-trend decomposition for time series with complex patterns","authors":"Qingsong Wen, Zhe Zhang, Yan Li, Liang Sun","year":"2020","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Wen et al., 2021a] Qingsong Wen, Kai He, Liang Sun, Yingying Zhang, Min Ke, et al. RobustPeriod: Time-frequency mining for robust multiple periodicities detection. In SIGMOD, 2021.","title":"RobustPeriod: Time-frequency mining for robust multiple periodicities detection","authors":"Wen","year":"2021","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Wen et al., 2021b] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In IJCAI, 2021.","title":"Time series data augmentation for deep learning: A survey","authors":"Wen","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/631"},{"position":56,"raw":"[Wen et al., 2022] Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An in- dustrial perspective. In KDD, 2022.","title":"Robust time series analysis and applications: An industrial perspective","authors":"Qingsong Wen, Linxiao Yang, Tian Zhou, Liang Sun","year":"2022","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Wu et al., 2020a] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse trans- former for time series forecasting. In NeurIPS, 2020.","title":"Adversarial sparse transformer for time series forecasting","authors":"Wu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Wu et al., 2020b] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In ICLR, 2020.","title":"Lite transformer with long-short range attention","authors":"Wu","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Ming- sheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.","title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","authors":"Wu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Xin et al., 2020] Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy J. Lin. DeeBERT: Dynamic early exiting for acceler- ating bert inference. In ACL, 2020.","title":"DeeBERT: Dynamic early exiting for accelerating bert inference","authors":"Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy J Lin","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Xu et al., 2020] Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. Spatial- temporal transformer networks for traffic flow forecasting. arXiv preprint arXiv:2001.02908, 2020.","title":"Spatialtemporal transformer networks for traffic flow forecasting","authors":"Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, Hongkai Xiong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Xu et al., 2022] Jiehui Xu, Haixu Wu, Jianmin Wang, and Ming- sheng Long. Anomaly Transformer: Time series anomaly detec- tion with association discrepancy. In ICLR, 2022.","title":"Anomaly Transformer: Time series anomaly detection with association discrepancy","authors":"Xu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Yan et al., 2019] Junchi Yan, Hongteng Xu, and Liangda Li. Mod- eling and applications for temporal point processes. In KDD, 2019.","title":"Modeling and applications for temporal point processes","authors":"Yan","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Yang et al., 2021] Chao-Han Huck Yang, Yun-Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In ICML, 2021.","title":"Voice2series: Reprogramming acoustic models for time series classification","authors":"Chao-Han Huck, Yun-Yun Yang, Pin-Yu Tsai, Chen","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Yu et al., 2020] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedes- trian trajectory prediction. In ECCV, 2020.","title":"Spatio-temporal graph transformer networks for pedestrian trajectory prediction","authors":"Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, Shuai Yi","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Yuan and Lin, 2020] Yuan Yuan and Lei Lin. Self-supervised pre- training of transformers for satellite image time series classifica- tion. IEEE J-STARS, 14:474-487, 2020.","title":"Self-supervised pretraining of transformers for satellite image time series classification","authors":"Yuan Yuan, Lei Lin","year":"2020","venue":"IEEE J-STARS","doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Yun et al., 2020] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, et al. Are transformers universal ap- proximators of sequence-to-sequence functions? In ICLR, 2020.","title":"Are transformers universal approximators of sequence-to-sequence functions?","authors":"Yun","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Zeng et al., 2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, 2023.","title":"Are transformers effective for time series forecasting?","authors":"Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Zerveas et al., 2021] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series repre- sentation learning. In KDD, 2021.","title":"A transformer-based framework for multivariate time series representation learning","authors":"George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, Carsten Eickhoff","year":"2021","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Zhang and Yan, 2023] Yunhao Zhang and Junchi Yan. Cross- former: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR, 2023.","title":"Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting","authors":"Yunhao Zhang, Junchi Yan","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Zhang et al., 2020] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In ICML, 2020.","title":"Self-attentive Hawkes process","authors":"Qiang Zhang, Aldo Lipani, Omer Kirnap, Emine Yilmaz","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Zhang et al., 2021] Hongwei Zhang, Yuanqing Xia, et al. Unsu- pervised anomaly detection in multivariate time series through transformer-based variational autoencoder. In CCDC, 2021.","title":"Unsupervised anomaly detection in multivariate time series through transformer-based variational autoencoder","authors":"Hongwei Zhang, Yuanqing Xia","year":"2021","venue":"CCDC","doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time- series forecasting. In AAAI, 2021.","title":"Informer: Beyond efficient transformer for long sequence timeseries forecasting","authors":"Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":74,"raw":"[Zhou et al., 2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency en- hanced decomposed transformer for long-term series forecasting. In ICML, 2022.","title":"FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting","authors":"Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, Rong Jin","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":75,"raw":"[Zuo et al., 2020] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In ICML, 2020.","title":"Transformer Hawkes process","authors":"Zuo","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}