{"type":"ScholarlyArticle","title":"Data-Driven Market-Making via Model-Free Learning","authors":[{"name":"Yueyang Zhong","slug":"yueyang-zhong","given":"Yueyang","family":"Zhong","affiliation":""},{"name":"YeeMan Bergstrom","slug":"yeeman-bergstrom","given":"YeeMan","family":"Bergstrom","affiliation":"Proprietary Trading , Chicago"},{"name":"Amy Ward","slug":"amy-ward","given":"Amy","family":"Ward","affiliation":"Booth School of Business , University of Chicago"}],"abstract":"This paper studies when a market-making firm should place orders to maximize their expected net profit, while also constraining risk, assuming orders are maintained on an electronic limit order book (LOB). To do this, we use a model-free and off-policy method, Q-learning, coupled with state aggregation, to develop a proposed trading strategy that can be implemented using a simple lookup table. Our main training dataset is derived from event-by-event data recording the state of the LOB. Our proposed trading strategy has passed both in-sample and out-of-sample testing in the backtester of the market-making firm with whom we are collaborating, and it also outperforms other benchmark strategies. As a result, the firm desires to put the strategy into production.","doi":"10.24963/ijcai.2020/615","url":"https://ijcai.org/proceedings/2020/615","pdf":"https://ijcai.org/proceedings/2020/0615.pdf","markdown":"https://ijcai.org/proceedings/2020/615.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/615","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":615,"pages":{"first":4461,"last":4468},"keywords":["AI for trading: AI for strategic trading and strategy design","AI for trading: AI for algorithmic trading","AI for trading: AI for trading incentive and strategy optimization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24893/file_24893.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Avellaneda and Stoikov, 2008] Marco Avellaneda and Sasha Stoikov. High-frequency trading in a limit order book. Quantitative Finance, 8(3):217-224, 2008.","title":"Marco Avellaneda and Sasha Stoikov. High-frequency trading in a limit order book","authors":"Stoikov Avellaneda","year":"2008","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas and Tsitsiklis, 1996] Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming, vol- ume 5. Athena Scientific Belmont, MA, 1996.","title":"Neuro-dynamic programming","authors":"P Dimitri, John N Bertsekas, Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas et al., 1995] Dimitri P Bertsekas, Dimitri P Bert- sekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA, 1995.","title":"Dynamic programming and optimal control","authors":"Dimitri P Dimitri P Bertsekas, Dimitri P Bertsekas, Dimitri P Bertsekas, Bertsekas","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cartea and Jaimungal, 2016] Álvaro Cartea and Sebastian Jaimungal. Incorporating order-flow into optimal execu- tion. Mathematics and Financial Economics, 10(3):339- 364, 2016.","title":"Álvaro Cartea and Sebastian Jaimungal. Incorporating order-flow into optimal execution","authors":"Jaimungal Cartea","year":"2016","venue":"Mathematics and Financial Economics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cartea et al., 2018] Álvaro Cartea, Ryan Donnelly, and Se- bastian Jaimungal. Enhancing trading strategies with order book signals. Applied Mathematical Finance, 25(1):1-35, 2018.","title":"Álvaro Cartea, Ryan Donnelly, and Sebastian Jaimungal. Enhancing trading strategies with order book signals","authors":"Cartea","year":"2018","venue":"Applied Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cont and De Larrard, 2012] Rama Cont and Adrien De Lar- rard. Order book dynamics in liquid markets: limit the- orems and diffusion approximations. Available at SSRN 1757861, 2012.","title":"Rama Cont and Adrien De Larrard. Order book dynamics in liquid markets: limit theorems and diffusion approximations","authors":"De Cont, Larrard","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Doloc, 2019] Cris Doloc. Applications of Computational Intelligence in Data-Driven Trading. John Wiley & Sons, 2019.","title":"Cris Doloc. Applications of Computational Intelligence in Data-Driven Trading","authors":"Doloc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dudík et al., 2014] Miroslav Dudík, Dumitru Erhan, John Langford, Lihong Li, et al. Doubly robust policy evalua- tion and optimization. Statistical Science, 29(4):485-511, 2014.","title":"Doubly robust policy evaluation and optimization","authors":"Dudík","year":"2014","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guilbaud and Pham, 2013] Fabien Guilbaud and Huyen Pham. Optimal high-frequency trading with limit and mar- ket orders. Quantitative Finance, 13(1):79-94, 2013.","title":"Optimal high-frequency trading with limit and market orders","authors":"Pham Guilbaud, Fabien Guilbaud, Huyen Pham","year":"2013","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaakkola et al., 1994] Tommi Jaakkola, Michael I Jordan, and Satinder P Singh. Convergence of stochastic iterative dynamic programming algorithms. In Advances in neural information processing systems, pages 703-710, 1994.","title":"Convergence of stochastic iterative dynamic programming algorithms","authors":"Tommi Jaakkola, Michael I Jordan, P Satinder, Singh","year":"1994","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiang and Li, 2015] Nan Jiang and Lihong Li. Doubly ro- bust off-policy value evaluation for reinforcement learn- ing. arXiv preprint arXiv:1511.03722, 2015.","title":"Doubly robust off-policy value evaluation for reinforcement learning","authors":"Nan Jiang, Lihong Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lagoudakis and Parr, 2003] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of machine learning research, 4(Dec):1107-1149, 2003.","title":"Least-squares policy iteration","authors":"G Michail, Ronald Lagoudakis, Parr","year":"2003","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lim and Gorse, 2018] Ye-Sheen Lim and Denise Gorse. Reinforcement learning for high-frequency market mak- ing. In ESANN, 2018.","title":"Reinforcement learning for high-frequency market making","authors":"Gorse ; Ye-Sheen Lim, Denise Lim, Gorse","year":"2018","venue":"ESANN","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Menkveld, 2013] Albert J Menkveld. High frequency trad- ing and the new market makers. Journal of financial Mar- kets, 16(4):712-740, 2013.","title":"High frequency trading and the new market makers","authors":"J Albert, Menkveld","year":"2013","venue":"Journal of financial Markets","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pepyne et al., 1996] David L Pepyne, Douglas P Looze, Christos G Cassandras, and Theodore E Djaferis. Applica- tion of q-learning to elevator dispatcidng. IFAC Proceed- ings Volumes, 29(1):4742-4747, 1996.","title":"Application of q-learning to elevator dispatcidng","authors":"David L Pepyne, Christos G Douglas P Looze, Theodore E Cassandras, Djaferis","year":"1996","venue":"IFAC Proceedings Volumes","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Powell, 2007] Warren B Powell. Approximate Dynamic Programming: Solving the curses of dimensionality, vol- ume 703.","title":"Approximate Dynamic Programming: Solving the curses of dimensionality","authors":"B Warren, Powell","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Robins et al., 1994] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):846-866, 1994.","title":"Estimation of regression coefficients when some regressors are not always observed","authors":"Andrea James M Robins, Lue Ping Rotnitzky, Zhao","year":"1994","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Spooner et al., 2018] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement learning. In Proceedings of the 17th In- ternational Conference on Autonomous Agents and Mul- tiAgent Systems, pages 434-442. International Foundation for Autonomous Agents and Multiagent Systems, 2018.","title":"Market making via reinforcement learning","authors":"Thomas Spooner, John Fearnley, Rahul Savani, Andreas Koukorinis","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and Mul-tiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Swaminathan and Joachims, 2015] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In advances in neural informa- tion processing systems, pages 3231-3239, 2015.","title":"The self-normalized estimator for counterfactual learning","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":"advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsitsiklis, 1994] John N Tsitsiklis. Asynchronous stochas- tic approximation and q-learning. Machine learning, 16(3):185-202, 1994.","title":"Asynchronous stochastic approximation and q-learning","authors":"John N Tsitsiklis","year":"1994","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}