{"type":"ScholarlyArticle","title":"RSAP-DFM: Regime-Shifting Adaptive Posterior Dynamic Factor Model for Stock Returns Prediction","authors":[{"name":"Quanzhou Xiang","slug":"quanzhou-xiang","given":"Quanzhou","family":"Xiang","affiliation":"Fudan University"},{"name":"Zhan Chen","slug":"zhan-chen","given":"Zhan","family":"Chen","affiliation":"Fudan University"},{"name":"Qi Sun","slug":"qi-sun","given":"Qi","family":"Sun","affiliation":"Fudan University"},{"name":"Rujun Jiang","slug":"rujun-jiang","given":"Rujun","family":"Jiang","affiliation":"Fudan University"}],"abstract":"As the latest development of asset pricing research, how to use machine learning to improve the performance of factor models has become a topic of concern in recent years. The variability of the instantaneous macro environment brings great difficulties to quantitative investment, so the extended factor model must learn how to self-adapt to extract the macro pattern from the massive stock volume and price information, and how to continuously map the extracted macro pattern to the stock investment is also an open question. To this end, we propose the first continuous regime-based dynamic factor model, RSAP-DFM, which adaptively extracts continuous macroeconomic information and completes the dynamic explicit mapping of stock returns for the first time through dual regime shifting, while the adversarial posterior factors effectively correct the mapping deviation of prior factors. In addition, our model integrates an innovative two-stage optimization algorithm and normally distributed sampling, which further enhances the robustness of the model. Performance on three real stock datasets validates the validity of our model, which exceeds any previous methods available.","doi":"10.24963/ijcai.2024/676","url":"https://ijcai.org/proceedings/2024/676","pdf":"https://ijcai.org/proceedings/2024/0676.pdf","markdown":"https://ijcai.org/proceedings/2024/676.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/676","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":676,"pages":{"first":6116,"last":6124},"keywords":["Multidisciplinary Topics and Applications: MTA: Finance","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Carhart, 1997] Mark M Carhart. On persistence in mutual fund performance. The Journal of finance, 52(1):57-82, 1997.","title":"On persistence in mutual fund performance","authors":"M Mark, Carhart","year":"1997","venue":"The Journal of finance","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Guestrin, 2016] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowl- edge discovery and data mining, pages 785-794, 2016.","title":"Xgboost: A scalable tree boosting system","authors":"Tianqi Chen, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2020] Qianggang Ding, Sifan Wu, Hao Sun, Ji- adong Guo, and Jian Guo. Hierarchical multi-scale gaus- sian transformer for stock movement prediction. In IJCAI, pages 4640-4646, 2020.","title":"Hierarchical multi-scale gaussian transformer for stock movement prediction","authors":"Ding","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/640"},{"position":4,"raw":"[Du et al., 2021] Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting of time se- ries. In Proceedings of the 30th ACM international con- ference on information & knowledge management, pages 402-411, 2021.","title":"Adarnn: Adaptive learning and forecasting of time series","authors":"Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, Chongjun Wang","year":"2021","venue":"Proceedings of the 30th ACM international conference on information & knowledge management","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2022] Yitong Duan, Lei Wang, Qizhong Zhang, and Jian Li. Factorvae: A probabilistic dynamic fac- tor model based on variational autoencoder for predicting cross-sectional stock returns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4468-4476, 2022.","title":"Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns","authors":"Yitong Duan, Lei Wang, Qizhong Zhang, Jian Li","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fama and French, 1992] Eugene F Fama and Kenneth R French. The cross-section of expected stock returns. the Journal of Finance, 47(2):427-465, 1992.","title":"The cross-section of expected stock returns","authors":"F Eugene, Kenneth R Fama, French","year":"1992","venue":"the Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geweke, 1977] John Geweke. The dynamic factor analy- sis of economic time series. Latent variables in socio- economic models, 1977.","title":"The dynamic factor analysis of economic time series","authors":"John Geweke","year":"1977","venue":"Latent variables in socioeconomic models","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2021] Shihao Gu, Bryan Kelly, and Dacheng Xiu. Autoencoder asset pricing models. Journal of Economet- rics, 222(1):429-450, 2021.","title":"Autoencoder asset pricing models","authors":"Shihao Gu, Bryan Kelly, Dacheng Xiu","year":"2021","venue":"Journal of Econometrics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. Advances in neural information process- ing systems, 30, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kelly et al., 2019] Bryan T Kelly, Seth Pruitt, and Yinan Su. Characteristics are covariances: A unified model of risk and return. Journal of Financial Economics, 134(3):501- 524, 2019.","title":"Characteristics are covariances: A unified model of risk and return","authors":"Seth Bryan T Kelly, Yinan Pruitt, Su","year":"2019","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural infor- mation processing systems, 25, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Krizhevsky","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2021a] Hengxu Lin, Dong Zhou, Weiqing Liu, and Jiang Bian. Deep risk model: a deep learning solu- tion for mining latent risk factors to improve covariance matrix estimation. In Proceedings of the Second ACM In- ternational Conference on AI in Finance, pages 1-8, 2021.","title":"Deep risk model: a deep learning solution for mining latent risk factors to improve covariance matrix estimation","authors":"Lin","year":"2021","venue":"Proceedings of the Second ACM International Conference on AI in Finance","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2021b] Hengxu Lin, Dong Zhou, Weiqing Liu, and Jiang Bian. Learning multiple stock trading patterns with temporal routing adaptor and optimal transport. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1017-1026, 2021.","title":"Learning multiple stock trading patterns with temporal routing adaptor and optimal transport","authors":"Lin","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mari and Mari, 2023] Carlo Mari and Emiliano Mari. Deep learning based regime-switching models of energy com- modity prices. Energy Systems, 14(4):913-934, 2023.","title":"Deep learning based regime-switching models of energy commodity prices","authors":"Mari, Mari ;, Carlo Mari, Emiliano Mari","year":"2023","venue":"Energy Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nelson et al., 2017] David MQ Nelson, Adriano CM Pereira, and Renato A De Oliveira. Stock market's price movement prediction with lstm neural networks. In 2017 International joint conference on neural networks (IJCNN), pages 1419-1426. Ieee, 2017.","title":"Stock market's price movement prediction with lstm neural networks","authors":"Adriano Cm David Mq Nelson, Renato A De Pereira, Oliveira","year":"2017","venue":"International joint conference on neural networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In IJCAI, pages 2627-2633, 2017.","title":"A dual-stage attention-based recurrent neural network for time series prediction","authors":"Dongjin Yao Qin, Haifeng Song, Wei Chen, Guofei Cheng, Garrison W Jiang, Cottrell","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/366"},{"position":19,"raw":"[Ross, 2013] Stephen A Ross. The arbitrage theory of capital asset pricing. In Handbook of the fundamentals of finan- cial decision making: Part I, pages 11-30. World Scien- tific, 2013.","title":"The arbitrage theory of capital asset pricing","authors":"Ross Stephen","year":"2013","venue":"Handbook of the fundamentals of financial decision making: Part I","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rumelhart et al., 1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533-536, 1986.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1986","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sawhney et al., 2021] Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, Tyler Derr, and Rajiv Ratn Shah. Stock selection via spatiotemporal hypergraph attention net- work: A learning to rank approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 497-504, 2021.","title":"Stock selection via spatiotemporal hypergraph attention network: A learning to rank approach","authors":"Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, Tyler Derr, Rajiv Ratn Shah","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Selvin et al., 2017] Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan, Vijay Krishna Menon, and KP So- man. Stock price prediction using lstm, rnn and cnn- sliding window model. In 2017 international conference on advances in computing, communications and informat- ics (icacci), pages 1643-1647. IEEE, 2017.","title":"Stock price prediction using lstm, rnn and cnnsliding window model","authors":"Selvin","year":"2017","venue":"2017 international conference on advances in computing, communications and informatics (icacci)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharpe, 1964] William F Sharpe. Capital asset prices: A theory of market equilibrium under conditions of risk. The journal of finance, 19(3):425-442, 1964.","title":"Capital asset prices: A theory of market equilibrium under conditions of risk","authors":"F William, Sharpe","year":"1964","venue":"The journal of finance","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"James H Stock and Mark W Wat- son. Dynamic factor models. 2011. [Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing sys- tems, 27, 2014.","title":"Dynamic factor models","authors":"H James, Mark W Stock, Watson ; Oriol, Quoc V Vinyals, Le","year":"2011","venue":"Sequence to sequence learning with neural networks","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Uddin and Yu, 2020] Ajim Uddin and Dantong Yu. Latent factor model for asset pricing. Journal of Behavioral and Experimental Finance, 27:100353, 2020.","title":"Latent factor model for asset pricing","authors":"Yu ; Uddin, Ajim Uddin, Dantong Yu","year":"2020","venue":"Journal of Behavioral and Experimental Finance","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Velickovic et al., 2017] Petar Velickovic, Guillem Cucu- rull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat, 1050(20):10-48550, 2017.","title":"Graph attention networks","authors":"Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022] Heyuan Wang, Tengjiao Wang, Shun Li, Jiayi Zheng, Shijie Guan, and Wei Chen. Adaptive long- short pattern transformer for stock investment selection. In IJCAI, pages 3970-3977, 2022.","title":"Adaptive longshort pattern transformer for stock investment selection","authors":"Heyuan Wang, Tengjiao Wang, Shun Li, Jiayi Zheng, Shijie Guan, Wei Chen","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wei et al., 2022] Zikai Wei, Bo Dai, and Dahua Lin. Factor investing with a deep multi-factor model. arXiv preprint arXiv:2210.12462, 2022.","title":"Factor investing with a deep multi-factor model","authors":"Wei","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wei et al., 2023] Zikai Wei, Anyi Rao, Bo Dai, and Dahua Lin. Hirevae: An online and adaptive factor model based on hierarchical and regime-switch vae. In IJCAI, page 4903-4911, 2023.","title":"Hirevae: An online and adaptive factor model based on hierarchical and regime-switch vae","authors":"Wei","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/545"},{"position":30,"raw":"[Wu et al., 2021] Xueqing Wu, Lewen Wang, Yingce Xia, Weiqing Liu, Lijun Wu, Shufang Xie, Tao Qin, and Tie- Yan Liu. Temporally correlated task scheduling for se- quence learning. In International Conference on Machine Learning, pages 11274-11284. PMLR, 2021.","title":"Temporally correlated task scheduling for sequence learning","authors":"Xueqing Wu, Lewen Wang, Yingce Xia, Weiqing Liu, Lijun Wu, Shufang Xie, Tao Qin, Tie-Yan Liu","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2021a] Wentao Xu, Weiqing Liu, Jiang Bian, Jian Yin, and Tie-Yan Liu. Instance-wise graph-based frame- work for multivariate time series forecasting. arXiv preprint arXiv:2109.06489, 2021.","title":"Instance-wise graph-based framework for multivariate time series forecasting","authors":"Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2021b] Wentao Xu, Weiqing Liu, Lewen Wang, Yingce Xia, Jiang Bian, Jian Yin, and Tie-Yan Liu. Hist: A graph-based framework for stock trend forecast- ing via mining concept-oriented shared information. arXiv preprint arXiv:2110.13716, 2021.","title":"Hist: A graph-based framework for stock trend forecasting via mining concept-oriented shared information","authors":"Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2020] Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. Qlib: An ai- oriented quantitative investment platform. arXiv preprint arXiv:2009.11189, 2020.","title":"Qlib: An aioriented quantitative investment platform","authors":"Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, Tie-Yan Liu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 2141-2149, 2017.","title":"Stock price prediction via discovering multifrequency trading patterns","authors":"Zhang","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}