{"type":"ScholarlyArticle","title":"Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction","authors":[{"name":"Qianggang Ding","slug":"qianggang-ding","given":"Qianggang","family":"Ding","affiliation":"Tsinghua University"},{"name":"Sifan Wu","slug":"sifan-wu","given":"Sifan","family":"Wu","affiliation":"Tsinghua University"},{"name":"Hao Sun","slug":"hao-sun","given":"Hao","family":"Sun","affiliation":"The Chinese University of Hong Kong"},{"name":"Jiadong Guo","slug":"jiadong-guo","given":"Jiadong","family":"Guo","affiliation":""},{"name":"Jian Guo","slug":"jian-guo","given":"Jian","family":"Guo","affiliation":""}],"abstract":"Predicting the price movement of finance securities like stocks is an important but challenging task, due to the uncertainty of financial markets. In this paper, we propose a novel approach based on the Transformer to tackle the stock movement prediction task. Furthermore, we present several enhancements for the proposed basic Transformer. Firstly, we propose a Multi-Scale Gaussian Prior to enhance the locality of Transformer. Secondly, we develop an Orthogonal Regularization to avoid learning redundant heads in the multi-head self-attention mechanism. Thirdly, we design a Trading Gap Splitter for Transformer to learn hierarchical features of high-frequency finance data. Compared with other popular recurrent neural networks such as LSTM, the proposed method has the advantage to mine extremely long-term dependencies from financial time series. Experimental results show our proposed models outperform several competitive methods in stock price prediction tasks for the NASDAQ exchange market and the China A-shares market.","doi":"10.24963/ijcai.2020/640","url":"https://ijcai.org/proceedings/2020/640","pdf":"https://ijcai.org/proceedings/2020/0640.pdf","markdown":"https://ijcai.org/proceedings/2020/640.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/640","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":640,"pages":{"first":4640,"last":4646},"keywords":["AI for trading: AI for novel financial models","AI for trading: General","Foundation for AI in FinTech: Modeling financial structure and hierarchy","AI for trading: AI for predictive trading","Foundation for AI in FinTech: Deep learning and representation for FinTech","AI for trading: AI for algorithmic trading","AI for trading: AI for novel financial mechanisms"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25696/file_25696.m3u8","length_sec":218,"provider":"hls"}],"references":[{"position":1,"raw":"[Altman, 1992] Naomi S Altman. An introduction to ker- nel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175-185, 1992.","title":"An introduction to kernel and nearest-neighbor nonparametric regression","authors":"Naomi S Altman","year":"1992","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Deli Chen, Yanyan Zou, Keiko Hari- moto, Ruihan Bao, Xuancheng Ren, and Xu Sun. Incor- porating fine-grained events in stock movement prediction. arXiv preprint arXiv:1910.05078, 2019.","title":"Incorporating fine-grained events in stock movement prediction","authors":"Deli Chen, Yanyan Zou, Keiko Harimoto, Ruihan Bao, Xuancheng Ren, Xu Sun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cortes and Vapnik, 1995] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.","title":"Corinna Cortes and Vladimir Vapnik. Support-vector networks","authors":"Vapnik Cortes","year":"1995","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2019] Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. Enhancing stock movement prediction with adversarial training. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5843-5849. AAAI Press, 2019.","title":"Enhancing stock movement prediction with adversarial training","authors":"Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, Tat-Seng Chua","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/810"},{"position":6,"raw":"[Guo et al., 2019] Maosheng Guo, Yu Zhang, and Ting Liu. Gaussian transformer: a lightweight approach for natural language inference. In The Thirty-Third AAAI Conference on Artificial Intelligence, 2019.","title":"Gaussian transformer: a lightweight approach for natural language inference","authors":"Guo","year":"2019","venue":"The Thirty-Third AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lee et al., 2019] Mingu Lee, Jinkyu Lee, Hye Jin Jang, Byeonggeun Kim, Wonil Chang, and Kyuwoong Hwang. Orthogonality constrained multi-head attention for key- word spotting. arXiv preprint arXiv:1910.04500, 2019.","title":"Orthogonality constrained multi-head attention for keyword spotting","authors":"Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018] Jian Li, Zhaopeng Tu, Baosong Yang, Michael R Lyu, and Tong Zhang. Multi-head atten- tion with disagreement regularization. arXiv preprint arXiv:1810.10183, 2018.","title":"Multi-head attention with disagreement regularization","authors":"Jian Li, Zhaopeng Tu, Baosong Yang, Tong Michael R Lyu, Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, pages 5244- 5254, 2019.","title":"Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting","authors":"Shiyang Li, Xiaoyong Jin, Xiyou Yao Xuan, Wenhu Zhou, Yu-Xiang Chen, Xifeng Wang, Yan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2013] Yuling Lin, Haixiang Guo, and Jinglu Hu. An svm-based approach for stock market trend prediction. In The 2013 international joint conference on neural net- works (IJCNN), pages 1-7. IEEE, 2013.","title":"An svm-based approach for stock market trend prediction","authors":"Yuling Lin, Haixiang Guo, Jinglu Hu","year":"2013","venue":"The 2013 international joint conference on neural networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nelson et al., 2017] David MQ Nelson, Adriano CM Pereira, and Renato A de Oliveira. Stock market's price movement prediction with lstm neural networks. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 1419-1426. IEEE, 2017.","title":"Stock market's price movement prediction with lstm neural networks","authors":"Adriano Cm David Mq Nelson, Renato A De Pereira, Oliveira","year":"2017","venue":"2017 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual- stage attention-based recurrent neural network for time se- ries prediction. arXiv preprint arXiv:1704.02971, 2017.","title":"A dualstage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Chen, Guofei Cheng, Garrison Jiang, Cottrell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rumelhart et al., 1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533-536, 1986.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1986","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schumaker and Chen, 2009] Robert P Schumaker and Hsinchun Chen. Textual analysis of stock market pre- diction using breaking financial news: The azfin text system. ACM Transactions on Information Systems (TOIS), 27(2):12, 2009.","title":"Textual analysis of stock market prediction using breaking financial news: The azfin text system","authors":"P Robert, Hsinchun Schumaker, Chen","year":"2009","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Selvin et al., 2017] Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan, Vijay Krishna Menon, and KP So- man. Stock price prediction using lstm, rnn and cnn- sliding window model. In 2017 International Conference on Advances in Computing, Communications and Infor- matics (ICACCI), pages 1643-1647. IEEE, 2017.","title":"Stock price prediction using lstm, rnn and cnnsliding window model","authors":"Selvin","year":"2017","venue":"2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tao et al., 2018] Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, and Rui Yan. Get the point of my utterance! learning towards effective re- sponses with multi-head attention mechanism. In IJCAI, pages 4418-4424, 2018.","title":"Get the point of my utterance! learning towards effective responses with multi-head attention mechanism","authors":"Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, Rui Yan","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/614"},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Walczak, 2001] Steven Walczak. An empirical analysis of data requirements for financial forecasting with neural networks. Journal of management information systems, 17(4):203-222, 2001.","title":"An empirical analysis of data requirements for financial forecasting with neural networks","authors":"Steven Walczak","year":"2001","venue":"Journal of management information systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Jia Wang, Tong Sun, Benyuan Liu, Yu Cao, and Hongwei Zhu. Clvsa: A convolutional lstm based variational sequence-to-sequence model with atten- tion for predicting trends of financial markets. In Proceed- ings of the 28th International Joint Conference on Artifi- cial Intelligence, pages 3705-3711. AAAI Press, 2019.","title":"Clvsa: A convolutional lstm based variational sequence-to-sequence model with attention for predicting trends of financial markets","authors":"Jia Wang, Tong Sun, Benyuan Liu, Yu Cao, Hongwei Zhu","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/514"},{"position":21,"raw":"[Weng et al., 2017] Bin Weng, Mohamed A Ahmed, and Fadel M Megahed. Stock market one-day ahead move- ment prediction using disparate data sources. Expert Sys- tems with Applications, 79:153-163, 2017.","title":"Stock market one-day ahead movement prediction using disparate data sources","authors":"Weng","year":"2017","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xingjian et al., 2015] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional lstm network: A machine learning ap- proach for precipitation nowcasting. In Advances in neural information processing systems, pages 802-810, 2015.","title":"Convolutional lstm network: A machine learning approach for precipitation nowcasting","authors":"Xingjian","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu and Cohen, 2018] Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970-1979, 2018.","title":"Stock movement prediction from tweets and historical prices","authors":"Cohen ; ] Yumo Xu, Shay B Xu, Cohen","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi-frequency trading patterns. In Proceedings of the 23rd ACM SIGKDD international conference on knowl- edge discovery and data mining, pages 2141-2149. ACM, 2017.","title":"Stock price prediction via discovering multi-frequency trading patterns","authors":"Zhang","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}