{"type":"ScholarlyArticle","title":"Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution","authors":[{"name":"Feiyang Pan","slug":"feiyang-pan","given":"Feiyang","family":"Pan","affiliation":"Huawei EI Innovation Lab"},{"name":"Tongzhe Zhang","slug":"tongzhe-zhang","given":"Tongzhe","family":"Zhang","affiliation":"E Fund Innovation Lab"},{"name":"Ling Luo","slug":"ling-luo","given":"Ling","family":"Luo","affiliation":"Huawei EI Innovation Lab"},{"name":"Jia He","slug":"jia-he","given":"Jia","family":"He","affiliation":"Huawei EI Innovation Lab"},{"name":"Shuoling Liu","slug":"shuoling-liu","given":"Shuoling","family":"Liu","affiliation":"E Fund Innovation Lab"}],"abstract":"Optimal execution is a sequential decision-making problem for cost-saving in algorithmic trading. Studies have found that reinforcement learning (RL) can help decide the order-splitting sizes. However, a problem remains unsolved: how to place limit orders at appropriate limit prices?\r\nThe key challenge lies in the ``continuous-discrete duality'' of the action space. On the one hand, the continuous action space using percentage changes in prices is preferred for generalization. On the other hand, the trader eventually needs to choose limit prices discretely due to the existence of the tick size, which requires specialization for every single stock with different characteristics (e.g., the liquidity and the price range). So we need continuous control for generalization and discrete control for specialization. To this end, we propose a hybrid RL method to combine the advantages of both of them. We first use a continuous control agent to scope an action subset, then deploy a fine-grained agent to choose a specific limit price. Extensive experiments show that our method has higher sample efficiency and better training stability than existing RL algorithms and significantly outperforms previous learning-based methods for order execution.","doi":"10.24963/ijcai.2022/543","url":"https://ijcai.org/proceedings/2022/543","pdf":"https://ijcai.org/proceedings/2022/0543.pdf","markdown":"https://ijcai.org/proceedings/2022/543.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/543","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":543,"pages":{"first":3912,"last":3918},"keywords":["Multidisciplinary Topics and Applications: Finance","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987036","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985179","length_sec":627,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Almgren and Chriss, 2001] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5-40, 2001.","title":"Optimal execution of portfolio transactions","authors":"Chriss ; Almgren, Robert Almgren, Neil Chriss","year":"2001","venue":"Journal of Risk","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsimas and Lo, 1998] Dimitris Bertsimas and An- drew W Lo. Optimal control of execution costs. Journal of Financial Markets, 1(1):1-50, 1998.","title":"Dimitris Bertsimas and Andrew W Lo. Optimal control of execution costs","authors":"Lo Bertsimas","year":"1998","venue":"Journal of Financial Markets","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cartea et al., 2015] Álvaro Cartea, Sebastian Jaimungal, and José Penalva. Algorithmic and high-frequency trad- ing. Cambridge University Press, 2015.","title":"Algorithmic and high-frequency trading","authors":"Álvaro Cartea, Sebastian Jaimungal, José Penalva","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2021] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 107-115, 2021.","title":"Universal trading for order execution with oracle policy distillation","authors":"Yuchen Fang, Weiqing Kan Ren, Dong Liu, Weinan Zhou, Jiang Zhang, Yong Bian, Tie-Yan Yu, Liu","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hendricks and Wilcox, 2014] Dieter Hendricks and Diane Wilcox. A reinforcement learning extension to the almgren-chriss framework for optimal trade execution. In 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr), pages 457- 464. IEEE, 2014.","title":"A reinforcement learning extension to the almgren-chriss framework for optimal trade execution","authors":"Wilcox Hendricks, Dieter Hendricks, Diane Wilcox","year":"2014","venue":"IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huberman and Stanzl, 2000] G. Huberman and W. Stanzl. Optimal liquidity trading. Yale School of Management Working Papers, 9(2):165-200, 2000.","title":"Optimal liquidity trading","authors":"G Huberman, W Stanzl","year":"2000","venue":"Yale School of Management Working Papers","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kakade et al., 2004] Sham M Kakade, Michael Kearns, Yishay Mansour, and Luis E Ortiz. Competitive algo- rithms for vwap and limit order trading. In Proceedings of the 5th ACM conference on Electronic commerce, pages 189-198, 2004.","title":"Competitive algorithms for vwap and limit order trading","authors":"Michael Sham M Kakade, Yishay Kearns, Luis E Mansour, Ortiz","year":"2004","venue":"Proceedings of the 5th ACM conference on Electronic commerce","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin and Beling, 2019] Siyu Lin and AP Beling. Optimal liquidation with deep reinforcement learning. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) Deep Reinforcement Learning Workshop, Vancouver, Canada, 2019.","title":"Optimal liquidation with deep reinforcement learning","authors":"Lin, Beling ; ] Siyu Lin, Beling","year":"2019","venue":"33rd Conference on Neural Information Processing Systems (NeurIPS 2019) Deep Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin and Beling, 2020] Siyu Lin and Peter A Beling. An end-to-end optimal trade execution framework based on proximal policy optimization. In IJCAI, pages 4548-4554, 2020.","title":"An end-to-end optimal trade execution framework based on proximal policy optimization","authors":"Lin, Beling ; ] Siyu Lin, Peter A Beling","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/627"},{"position":10,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 2013. cite arxiv:1312.5602Comment: NIPS Deep Learning Work- shop 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS Deep Learning Workshop 2013","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nevmyvaka et al., 2006] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. In Proceedings of the 23rd international conference on Machine learning, pages 673-680, 2006.","title":"Reinforcement learning for optimized trade execution","authors":"Yuriy Nevmyvaka, Yi Feng, Michael Kearns","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ning et al., 2018] Brian Ning, Franco Ho Ting Lin, and Se- bastian Jaimungal. Double deep q-learning for optimal ex- ecution. arXiv preprint arXiv:1812.06600, 2018.","title":"Double deep q-learning for optimal execution","authors":"Brian Ning, Franco Ho Ting Lin, Sebastian Jaimungal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2019] Feiyang Pan, Qingpeng Cai, An-Xiang Zeng, Chun-Xiang Pan, Qing Da, Hualin He, Qing He, and Pingzhong Tang. Policy optimization with model-based explorations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4675-4682, 2019.","title":"Policy optimization with model-based explorations","authors":"Pan","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Watkins and Dayan, 1992] Cjch Watkins and P. Dayan. Q- learning. In Machine Learning, 1992.","title":"Qlearning","authors":"Dayan ; Cjch Watkins, P Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}