{"type":"ScholarlyArticle","title":"Towards Generalizable Reinforcement Learning for Trade Execution","authors":[{"name":"Chuheng Zhang","slug":"chuheng-zhang","given":"Chuheng","family":"Zhang","affiliation":"Microsoft Research"},{"name":"Yitong Duan","slug":"yitong-duan","given":"Yitong","family":"Duan","affiliation":"IIIS , Tsinghua University"},{"name":"Xiaoyu Chen","slug":"xiaoyu-chen","given":"Xiaoyu","family":"Chen","affiliation":"IIIS , Tsinghua University"},{"name":"Jianyu Chen","slug":"jianyu-chen","given":"Jianyu","family":"Chen","affiliation":"IIIS , Tsinghua University"},{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"IIIS , Tsinghua University"},{"name":"Li Zhao","slug":"li-zhao","given":"Li","family":"Zhao","affiliation":"Microsoft Research"}],"abstract":"Optimized trade execution is to sell (or buy) a given amount of assets in a given time with the lowest possible trading cost. Recently, reinforcement learning (RL) has been applied to optimized trade execution to learn smarter policies from market data. However, we find that many existing RL methods exhibit considerable overfitting which prevents them from real deployment. In this paper, we provide an extensive study on the overfitting problem in optimized trade execution. First, we model the optimized trade execution as offline RL with dynamic context (ORDC), where the context represents market variables that cannot be influenced by the trading policy and are collected in an offline manner.  Under this framework, we derive the generalization bound and find that the overfitting issue is caused by large context space and limited context samples in the offline setting. Accordingly, we propose to learn compact representations for context to address the overfitting problem, either by leveraging prior knowledge or in an end-to-end manner. To evaluate our algorithms, we also implement a carefully designed simulator based on historical limit order book (LOB) data to provide a high-fidelity benchmark for different algorithms. Our experiments on the high-fidelity simulator demonstrate that our algorithms can effectively alleviate overfitting and achieve better performance.","doi":"10.24963/ijcai.2023/553","url":"https://ijcai.org/proceedings/2023/553","pdf":"https://ijcai.org/proceedings/2023/0553.pdf","markdown":"https://ijcai.org/proceedings/2023/553.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/553","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":553,"pages":{"first":4975,"last":4983},"keywords":["Machine Learning: ML: Deep reinforcement learning","Multidisciplinary Topics and Applications: MDA: Finance"],"videos":[],"references":[{"position":1,"raw":"[Almgren and Chriss, 1999] Robert Almgren and Neil Chriss. Value under liquidation. Risk, 12(12):61-63, 1999.","title":"Robert Almgren and Neil Chriss","authors":"Chriss Almgren","year":"1999","venue":"Risk","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Almgren and Chriss, 2001] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5-40, 2001.","title":"Optimal execution of portfolio transactions","authors":"Chriss ; Almgren, Robert Almgren, Neil Chriss","year":"2001","venue":"Journal of Risk","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arjovsky et al., 2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.","title":"Invariant risk minimization","authors":"Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bulthuis et al., 2017] Brian Bulthuis, Julio Concha, Tim Leung, and Brian Ward. Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty. International Journal of Financial Engineer- ing, 4(1):175-200, 2017.","title":"Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty","authors":"Brian Bulthuis, Julio Concha, Tim Leung, Brian Ward","year":"2017","venue":"International Journal of Financial Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cobbe et al., 2019] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying general- ization in reinforcement learning. In International Con- ference on Machine Learning, pages 1282-1289. PMLR, 2019.","title":"Quantifying generalization in reinforcement learning","authors":"Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cobbe et al., 2020] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International con- ference on machine learning, pages 2048-2056. PMLR, 2020.","title":"Leveraging procedural generation to benchmark reinforcement learning","authors":"Karl Cobbe, Chris Hesse, Jacob Hilton, John Schulman","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dabérius et al., 2019] Kevin Dabérius, Elvin Granat, and Patrik Karlsson. Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks. Available at SSRN 3374766, 2019.","title":"Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks","authors":"Kevin Dabérius, Elvin Granat, Patrik Karlsson","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dietterich et al., 2018] Thomas Dietterich, George Trim- ponias, and Zhitang Chen. Discovering and removing exogenous state variables and rewards for reinforcement learning. In International Conference on Machine Learn- ing, pages 1262-1270. PMLR, 2018.","title":"Discovering and removing exogenous state variables and rewards for reinforcement learning","authors":"Thomas Dietterich, George Trimponias, Zhitang Chen","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2019] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Lang- ford. Provably efficient RL with rich observations via latent state decoding. In Proceedings of the 36th Inter- national Conference on Machine Learning, pages 1665- 1674. PMLR, 2019.","title":"Provably efficient RL with rich observations via latent state decoding","authors":"Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, John Langford","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fang et al., 2021] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 107-115, 2021.","title":"Universal trading for order execution with oracle policy distillation","authors":"Yuchen Fang, Weiqing Kan Ren, Dong Liu, Weinan Zhou, Jiang Zhang, Yong Bian, Tie-Yan Yu, Liu","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guéant et al., 2012] Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia. Optimal portfolio liquidation with limit orders. SIAM Journal on Financial Mathematics, 3(1):740-764, 2012.","title":"Optimal portfolio liquidation with limit orders","authors":"Olivier Guéant, Charles-Albert Lehalle, Joaquin Fernandez-Tapia","year":"2012","venue":"SIAM Journal on Financial Mathematics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirk et al., 2021] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of gener- alisation in deep reinforcement learning. arXiv preprint arXiv:2111.09794, 2021.","title":"A survey of generalisation in deep reinforcement learning","authors":"Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Beling, 2020] Siyu Lin and Peter A Beling. A deep reinforcement learning framework for optimal trade exe- cution. In Joint European Conference on Machine Learn- ing and Knowledge Discovery in Databases, pages 223- 240. Springer, 2020.","title":"A deep reinforcement learning framework for optimal trade execution","authors":"Lin, Beling ; ] Siyu Lin, Peter A Beling","year":"2020","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin and Beling, 2021] Siyu Lin and Peter A Beling. An end-to-end optimal trade execution framework based on proximal policy optimization. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 4548- 4554, 2021.","title":"An end-to-end optimal trade execution framework based on proximal policy optimization","authors":"Lin, Beling ; ] Siyu Lin, Peter A Beling","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/627"},{"position":15,"raw":"[Mao et al., 2017] Hongzi Mao, Ravi Netravali, and Mo- hammad Alizadeh. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pages 197-210, 2017.","title":"Neural adaptive video streaming with pensieve","authors":"Mao","year":"2017","venue":"Proceedings of the Conference of the ACM Special Interest Group on Data Communication","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mao et al., 2018] Hongzi Mao, Shaileshh Bojja Venkatakr- ishnan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input- driven environments. In International Conference on Learning Representations, 2018.","title":"Variance reduction for reinforcement learning in inputdriven environments","authors":"Mao","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nevmyvaka et al., 2006] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimized trade execution. In Proceedings of the 23rd international conference on Machine Learning, pages 673-680. PMLR, 2006.","title":"Reinforcement learning for optimized trade execution","authors":"Yuriy Nevmyvaka, Yi Feng, Michael Kearns","year":"2006","venue":"Proceedings of the 23rd international conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ning et al., 2018] Brian Ning, Franco Ho Ting Lin, and Se- bastian Jaimungal. Double deep Q-learning for optimal execution. arXiv preprint arXiv:1812.06600, 2018.","title":"Double deep Q-learning for optimal execution","authors":"Brian Ning, Franco Ho Ting Lin, Sebastian Jaimungal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oroojlooyjadid et al., 2022] Afshin Oroojlooyjadid, Mo- hammadReza Nazari, Lawrence V Snyder, and Martin Takáč. A deep Q-network for the beer game: Deep re- inforcement learning for inventory optimization. Manu- facturing & Service Operations Management, 24(1):285- 304, 2022.","title":"A deep Q-network for the beer game: Deep reinforcement learning for inventory optimization","authors":"Afshin Oroojlooyjadid, Lawrence V Mo-Hammadreza Nazari, Martin Snyder, Takáč","year":"2022","venue":"Manufacturing & Service Operations Management","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Packer et al., 2018] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282, 2018.","title":"Assessing generalization in deep reinforcement learning","authors":"Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shahamiri, 2008] Masoud Shahamiri. Reinforcement learn- ing in environments with independent delayed-sense dy- namics. Master Thesis, University of Alberta, 2008.","title":"Reinforcement learning in environments with independent delayed-sense dynamics","authors":"Masoud Shahamiri","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shen et al., 2020] Wei Shen, Xiaonan He, Chuheng Zhang, Qiang Ni, Wanchun Dou, and Yan Wang. Auxiliary-task based deep reinforcement learning for participant selec- tion problem in mobile crowdsourcing. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1355-1364, 2020.","title":"Auxiliary-task based deep reinforcement learning for participant selection problem in mobile crowdsourcing","authors":"Wei Shen, Xiaonan He, Chuheng Zhang, Qiang Ni, Wanchun Dou, Yan Wang","year":"2020","venue":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learn- ing, pages 387-395. PMLR, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2019] Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975, 2019.","title":"Xingyou Song","authors":"Song","year":"2019","venue":"Observational overfitting in reinforcement learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2020] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems, 33:7968-7978, 2020.","title":"Improving generalization in reinforcement learning with mixture regularization","authors":"Kaixin Wang, Bingyi Kang, Jie Shao, Jiashi Feng","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018a] Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018.","title":"A dissection of overfitting and generalization in continuous reinforcement learning","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018b] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018.","title":"A study on overfitting in deep reinforcement learning","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}