{"type":"ScholarlyArticle","title":"An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization","authors":[{"name":"Siyu Lin","slug":"siyu-lin","given":"Siyu","family":"Lin","affiliation":"University of Virginia , Charlottesville , VA , USA"},{"name":"Peter A. Beling","slug":"peter-a-beling","given":"Peter A.","family":"Beling","affiliation":"University of Virginia , Charlottesville , VA , USA"}],"abstract":"In this article, we propose an end-to-end adaptive framework for optimal trade execution based on Proximal Policy Optimization (PPO). We use two methods to account for the time dependencies in the market data based on two different neural network architecture: 1) Long short-term memory (LSTM) networks, 2) Fully-connected networks (FCN) by stacking the most recent limit orderbook (LOB) information as model inputs. The proposed framework can make trade execution decisions based on level-2 limit order book (LOB) information such as bid/ask prices and volumes directly without manually designed attributes as in previous research. Furthermore, we use a sparse reward function, which gives the agent reward signals at the end of each episode as an indicator of its relative performances against the baseline model, rather than implementation shortfall (IS) or a shaped reward function. The experimental results have demonstrated advantages over IS and the shaped reward function in terms of performance and simplicity. The proposed framework has outperformed the industry commonly used baseline models such as TWAP, VWAP, and AC as well as several Deep Reinforcement Learning (DRL) models on most of the 14 US equities in our experiments.","doi":"10.24963/ijcai.2020/627","url":"https://ijcai.org/proceedings/2020/627","pdf":"https://ijcai.org/proceedings/2020/0627.pdf","markdown":"https://ijcai.org/proceedings/2020/627.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/627","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":627,"pages":{"first":4548,"last":4554},"keywords":["AI for trading: AI for algorithmic trading","Foundation for AI in FinTech: Reinforcement learning for FinTech"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26849/file_26849.m3u8","length_sec":576,"provider":"hls"}],"references":[{"position":1,"raw":"Standard Deviation Model Name BA CRM FB GOOG GS MCD MSCI NVDA PEP PYPL QCOM TSLA TWLO WMT TWAP 335.05 885.62 4,154.47 707.03 92.74 221.72 159.14 478.18 1,058.16 681.91 2,913.72 203.00 196.88 1,442.92 VWAP 503.64 1,366.18 14,936.29 1,394.18 190.89 420.70 220.25 726.23 2,389.84 2,587.93 27,131.03 462.24 341.32 6,699.03 AC 335.80 894.99 4,517.32 707.40 93.19 222.10 159.59 479.65 1,058.48 776.66 4,724.56 210.49 215.42 1,456.07 DQN (Ning2018) 506.22 1,086.15 5,298.78 962.25 64.98 183.17 641.79 629.95 5,082.65 1,242.17 6,890.66 239.12 204.75 1,758.85 DQN (Lin2019) 326.62 830.79 4,117.55 696.35 91.07 214.50 642.64 454.73 1,033.27 631.29 2,821.33 226.80 196.34 1,386.26 DQN Sparse (Lin2019) 343.27 842.26 4,245.49 719.14 96.03 223.89 161.44 503.98 1,071.55 753.93 3,091.98 226.49 196.58 1,479.68 PPO Dense 345.79 836.57 4,379.35 708.64 92.68 221.73 159.47 478.18 1,153.07 672.30 3,795.95 244.20 204.27 1,599.52 PPO Stack 322.24 812.22 3,929.08 630.45 93.08 344.90 149.68 573.88 1,013.80 610.21 2,551.04 200.54 206.20 1,405.03 PPO LSTM 294.09 859.73 4,462.66 690.63 113.71 576.29 145.73 431.11 997.11 624.24 2,654.01 183.42 186.11 1,355.22","title":"Standard Deviation Model Name BA CRM FB GOOG GS MCD MSCI NVDA PEP PYPL QCOM","authors":null,"year":null,"venue":"TSLA TWLO WMT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Almgren and Chriss, 2000] Robert Almgren and Neil Chriss. Optimal execution of portfolio transactions. Journal of Risk, 3:5-40, 2000.","title":"Optimal execution of portfolio transactions","authors":"Chriss ; Almgren, Robert Almgren, Neil Chriss","year":"2000","venue":"Journal of Risk","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berkowitz et al., 1988] Stephen A. Berkowitz, Dennis E. Logue, and Eugene A. Noser Jr. The total cost of transac- tions on the nyse. Journal of Finance, 43(1):97-112, March 1988.","title":"The total cost of transactions on the nyse","authors":"Stephen A Berkowitz, Dennis E Logue, Eugene A Noser","year":"1988","venue":"Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bertsimas and Lo, 1998] Dimitris Bertsimas and Andrew W. Lo. Optimal control of execution costs. Journal of Finan- cial Markets, 1(1):1-50, April 1998.","title":"Optimal control of execution costs","authors":"Dimitris Bertsimas, Andrew W Lo","year":"1998","venue":"Journal of Financial Markets","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hendricks and Wilcox, 2014] Dieter Hendricks and Diane Wilcox. A reinforcement learning extension to the almgren- chriss framework for optimal trade execution. In Proceed- ings from IEEE Conference on Computational Intelligence for Financial Economics and Engineering, pages 57-464, London, UK, March 2014. IEEE.","title":"A reinforcement learning extension to the almgrenchriss framework for optimal trade execution","authors":"Wilcox Hendricks, Dieter Hendricks, Diane Wilcox","year":"2014","venue":"Proceedings from IEEE Conference on Computational Intelligence for Financial Economics and Engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huberman and Stanzl, 2005] Gur Huberman and Werner Stanzl. Optimal liquidity trading. Review of Finance, 9(2):165-200, 2005.","title":"Optimal liquidity trading","authors":"Gur Huberman, Werner Stanzl","year":"2005","venue":"Review of Finance","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kakade et al., 2004] Sham M. Kakade, Michael Kearns, Yishay Mansour, and Luis E. Ortiz. Competitive algo- rithms for vwap and limit order trading. In Proceedings of the ACM Conference on Electronic Commerce, New York, NY, 2004.","title":"Competitive algorithms for vwap and limit order trading","authors":"M Sham, Michael Kakade, Yishay Kearns, Luis E Mansour, Ortiz","year":"2004","venue":"Proceedings of the ACM Conference on Electronic Commerce","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liaw et al., 2018] Richard Liaw, Eric Liang, Robert Nishi- hara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118, 2018.","title":"Tune: A research platform for distributed model selection and training","authors":"Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, Ion Stoica","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin and Beling, 2019] Siyu Lin and Peter A. Beling. Op- timal liquidation with deep reinforcement learning. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019) Deep Reinforcement Learning Workshop, Vancouver, Canada, 2019.","title":"Optimal liquidation with deep reinforcement learning","authors":"Siyu Lin, Peter A Beling","year":"2019","venue":"33rd Conference on Neural Information Processing Systems (NeurIPS 2019) Deep Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep rein- forcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, February 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nevmyvaka et al., 2006] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement learning for optimal trade execution. In Proceedings of the 23rd International Confer- ence on Machine Learning, pages 673-68, Pittsburgh, PA, June 2006. Association for Computing Machinery. Ning et al., 2018]","title":"Reinforcement learning for optimal trade execution","authors":"Yuriy Nevmyvaka, Yi Feng, Michael Kearns","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Brian Ning, Franco Ho Ting Ling, and Sebastian Jaimungal. Double deep q-learning for optimal execution. arXiv:1812.06600, 2018.","title":"Double deep q-learning for optimal execution","authors":"Brian Ning, Franco Ho Ting, Ling, Sebastian Jaimungal","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1811.08540v2, 2017.","title":"Proximal policy optimization algorithms","authors":"Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}