{"type":"ScholarlyArticle","title":"MacMic: Executing Iceberg Orders via Hierarchical Reinforcement Learning","authors":[{"name":"Hui Niu","slug":"hui-niu","given":"Hui","family":"Niu","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Siyuan Li","slug":"siyuan-li","given":"Siyuan","family":"Li","affiliation":"Faculty of Computing , Harbin Institute of Technology , Harbin , China"},{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"}],"abstract":"In recent years, there has been a growing interest in applying reinforcement learning (RL) techniques to order execution owing to RL’s strong sequential decision-making ability. However, realistic order execution tasks usually involve a large fine-grained action space and a long trading duration. The former hinders the RL agents from efficient exploration. The latter increases the task complexity, since the agent must capture price advantages throughout the day as well as micro changes within a few seconds on the limited order books. In addressing these challenges, we propose MacMic, a novel Hierarchical RL-based order execution approach that captures market patterns and executes orders from different temporal scales. MacMic employs a high-level agent to split the parent order into smaller slices at coarse-grained time steps. Then a low-level agent is adopted to execute these slices by placing fixed-size sub-orders at a continuous time. Besides, to balance the multifaceted objectives of the two tasks, MacMic pretrains a causal stacking hidden Markov model (SHMM) to obtain both effective macro-level and micro-level market states. Comprehensive experimental results on 200 stocks across the US and China A-share markets validate the effectiveness of the proposed method.","doi":"10.24963/ijcai.2024/664","url":"https://ijcai.org/proceedings/2024/664","pdf":"https://ijcai.org/proceedings/2024/0664.pdf","markdown":"https://ijcai.org/proceedings/2024/664.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/664","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":664,"pages":{"first":6008,"last":6016},"keywords":["Multidisciplinary Topics and Applications: MTA: Finance","Machine Learning: ML: Representation learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Almgren and Chriss, 2001] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5-40, 2001.","title":"Optimal execution of portfolio transactions","authors":"Chriss ; Almgren, Robert Almgren, Neil Chriss","year":"2001","venue":"Journal of Risk","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2017] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on rein- forcement learning. In International Conference on Ma- chine Learning, 2017.","title":"A distributional perspective on reinforcement learning","authors":"G Marc, Will Bellemare, Rémi Dabney, Munos","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsimas and Lo, 1998] Dimitris Bertsimas and An- drew W Lo. Optimal control of execution costs. Journal of financial markets, 1(1):1-50, 1998.","title":"Dimitris Bertsimas and Andrew W Lo. Optimal control of execution costs","authors":"Lo Bertsimas","year":"1998","venue":"Journal of financial markets","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Di Chen, Yada Zhu, Miao Liu, and Jianbo Li. Cost-efficient reinforcement learning for op- timal trade execution on dynamic market environment. In Proceedings of the Third ACM International Conference on AI in Finance, pages 386-393, 2022.","title":"Cost-efficient reinforcement learning for optimal trade execution on dynamic market environment","authors":"Di Chen, Yada Zhu, Miao Liu, Jianbo Li","year":"2022","venue":"Proceedings of the Third ACM International Conference on AI in Finance","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2014] Junyoung Chung, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. ArXiv, abs/1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2018] Yi Ding, Weiqing Liu, Jiang Bian, Dao- qiang Zhang, and Tie-Yan Liu. Investor-imitator: A frame- work for trading knowledge extraction. Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, 2018.","title":"Investor-imitator: A framework for trading knowledge extraction","authors":"Yi Ding, Weiqing Liu, Jiang Bian, Daoqiang Zhang, Tie-Yan Liu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Esser and Mönch, 2007] Angelika Esser and Burkart Mönch. The navigation of an iceberg: The optimal use of hidden orders. Finance Research Letters, 4(2):68-81, 2007.","title":"The navigation of an iceberg: The optimal use of hidden orders","authors":"Mönch Esser, Angelika Esser, Burkart Mönch","year":"2007","venue":"Finance Research Letters","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2021] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In AAAI Conference on Artificial Intel- ligence, 2021.","title":"Universal trading for order execution with oracle policy distillation","authors":"Yuchen Fang, Weiqing Kan Ren, Dong Liu, Weinan Zhou, Jiang Zhang, Yong Bian, Tie-Yan Yu, Liu","year":"2021","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on ma- chine learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Hoof, David Meger","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hendricks and Wilcox, 2014] Dieter Hendricks and Diane Wilcox. A reinforcement learning extension to the almgren-chriss framework for optimal trade execution. 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr), pages 457- 464, 2014.","title":"A reinforcement learning extension to the almgren-chriss framework for optimal trade execution","authors":"Wilcox Hendricks, Dieter Hendricks, Diane Wilcox","year":"2014","venue":"IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Higgins et al., 2016] Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In International Confer- ence on Learning Representations, 2016.","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loïc Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huberman and Stanzl, 2005] Gur Huberman and Werner Stanzl. Optimal liquidity trading. Review of finance, 9(2):165-200, 2005.","title":"Optimal liquidity trading","authors":"Gur Huberman, Werner Stanzl","year":"2005","venue":"Review of finance","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kakade et al., 2004] Sham M Kakade, Michael Kearns, Yishay Mansour, and Luis E Ortiz. Competitive algo- rithms for vwap and limit order trading. In Proceedings of the 5th ACM conference on Electronic commerce, pages 189-198, 2004.","title":"Competitive algorithms for vwap and limit order trading","authors":"Michael Sham M Kakade, Yishay Kearns, Luis E Mansour, Ortiz","year":"2004","venue":"Proceedings of the 5th ACM conference on Electronic commerce","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Konda and Tsitsiklis, 1999] Vijay Konda and John Tsitsik- lis. Actor-critic algorithms. Advances in neural informa- tion processing systems, 12, 1999.","title":"Actor-critic algorithms","authors":"Vijay Konda, John Tsitsiklis","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin and Beling, 2019] Siyu Lin and Peter A Beling. Opti- mal liquidation with deep reinforcement learning. In Pro- ceedings of the 33rd Conference on Neural Information Processing Systems, Deep Reinforcement Learning Work- shop, Vancouver, Canada, 2019.","title":"Optimal liquidation with deep reinforcement learning","authors":"Lin, Beling ; ] Siyu Lin, Peter A Beling","year":"2019","venue":"Proceedings of the 33rd Conference on Neural Information Processing Systems, Deep Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin and Beling, 2020] Siyu Lin and Peter A. Beling. An end-to-end optimal trade execution framework based on proximal policy optimization. In International Joint Con- ference on Artificial Intelligence, 2020.","title":"An end-to-end optimal trade execution framework based on proximal policy optimization","authors":"Siyu Lin, Peter A Beling","year":"2020","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/627"},{"position":17,"raw":"[Neuberg, 2003] Leland Gerson Neuberg. Causality: Mod- els, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19:675 -685, 2003.","title":"Causality: Models, reasoning, and inference, by judea pearl, cambridge university press","authors":"Leland Gerson, Neuberg","year":"2000","venue":"Econometric Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nevmyvaka et al., 2006] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. Proceedings of the 23rd international conference on Machine learning, 2006.","title":"Reinforcement learning for optimized trade execution","authors":"Yuriy Nevmyvaka, Yi Feng, Michael Kearns","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ning et al., 2018] Brian Ning, Franco Ho Ting Ling, and Sebastian Jaimungal. Double deep q-learning for optimal execution. Applied Mathematical Finance, 28:361 -380, 2018.","title":"Double deep q-learning for optimal execution","authors":"Brian Ning, Franco Ho Ting, Ling, Sebastian Jaimungal","year":"2018","venue":"Applied Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan et al., 2022] Feiyang Pan, Tongzhe Zhang, Ling Luo, Jia He, and Shuoli Liu. Learn continuously, act discretely: Hybrid action-space reinforcement learning for optimal execution. In International Joint Conference on Artificial Intelligence, 2022.","title":"Learn continuously, act discretely: Hybrid action-space reinforcement learning for optimal execution","authors":"Pan","year":"2022","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/543"},{"position":21,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srinivas et al., 2020] A. Srinivas, Michael Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. ArXiv, abs/2004.04136, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"A Srinivas, Michael Laskin, P Abbeel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stooke et al., 2020] Adam Stooke, Kimin Lee, P. Abbeel, and Michael Laskin. Decoupling representation learn- ing from reinforcement learning. ArXiv, abs/2009.08319, 2020.","title":"Decoupling representation learning from reinforcement learning","authors":"Adam Stooke, Kimin Lee, P Abbeel, Michael Laskin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2018] Wen Sun, J. Andrew Bagnell, and By- ron Boots. Truncated horizon policy search: Combin- ing reinforcement learning & imitation learning. ArXiv, abs/1805.11240, 2018.","title":"Truncated horizon policy search: Combining reinforcement learning & imitation learning","authors":"Wen Sun, J Andrew Bagnell, Byron Boots","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2015] Ziyun Wang, Tom Schaul, Matteo Hes- sel, H. V. Hasselt, Marc Lanctot, and Nando de Fre- itas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learn- ing, 2015.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyun Wang, Tom Schaul, Matteo Hessel, H V Hasselt, Marc Lanctot, Nando De Freitas","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Rundong Wang, Hongxin Wei, Bo An, Zhouyan Feng, and Jun Yao. Commission fee is not enough: A hierarchical reinforced framework for portfolio management. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 626-633, 2021.","title":"Commission fee is not enough: A hierarchical reinforced framework for portfolio management","authors":"Wang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watkins and Dayan, 1992] Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.","title":"Q-learning","authors":"Christopher Watkins, Peter Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}