{"type":"ScholarlyArticle","title":"Robust Market Making via Adversarial Reinforcement Learning","authors":[{"name":"Thomas Spooner","slug":"thomas-spooner","given":"Thomas","family":"Spooner","affiliation":"Department of Computer Science , University of Liverpool"},{"name":"Rahul Savani","slug":"rahul-savani","given":"Rahul","family":"Savani","affiliation":"Department of Computer Science , University of Liverpool"}],"abstract":"We show that adversarial reinforcement learning (ARL) can be used to produce market marking agents that are robust to adversarial and adaptively-chosen market conditions. To apply ARL, we turn the well-studied single-agent model of Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a market maker and adversary.  The adversary acts as a proxy for other market participants that would like to profit at the market maker's expense.  We empirically compare two conventional single-agent RL agents with ARL, and show that our ARL approach leads to: 1) the emergence of risk-averse behaviour without constraints or domain-specific penalties; 2) significant improvements in performance across a set of standard metrics, evaluated with or without an adversary in the test environment, and; 3) improved robustness to model uncertainty. We empirically demonstrate that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.","doi":"10.24963/ijcai.2020/633","url":"https://ijcai.org/proceedings/2020/633","pdf":"https://ijcai.org/proceedings/2020/0633.pdf","markdown":"https://ijcai.org/proceedings/2020/633.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/633","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Special Track on AI in FinTech","track":"Special Track on AI in FinTech","number":633,"pages":{"first":4590,"last":4596},"keywords":["AI for trading: AI for algorithmic trading","AI for trading: AI for high frequency (cross-market) trading","AI for trading: AI for trading incentive and strategy optimization","Foundation for AI in FinTech: Reinforcement learning for FinTech","Foundation for AI in FinTech: Computational intelligence for FinTech","AI for trading: AI for strategic trading and strategy design"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24860/file_24860.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Abergel et al., 2016] F. Abergel, M. Anane, A. Chakraborti, A. Je- didi, and I. M. Toke. Limit Order Books. Cambridge University Press, 2016.","title":"Limit Order Books","authors":"F Abergel, M Anane, A Chakraborti, A Jedidi, I M Toke","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abernethy and Kale, 2013] J. D. Abernethy and S. Kale. Adaptive Market Making via Online Learning. In Proc. of NIPS, 2013. [Avellaneda and Stoikov, 2008]","title":"Adaptive Market Making via Online Learning","authors":"J D Abernethy, S Kale","year":"2008","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"M. Avellaneda and S. Stoikov. High-frequency trading in a limit order book. Quantitative Fi- nance, 8(3):217-224, 2008.","title":"High-frequency trading in a limit order book","authors":"M Avellaneda, S Stoikov","year":"2008","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellemare et al., 2017] M. G. Bellemare, W. Dabney, and R. Munos. A Distributional Perspective on Reinforcement Learn- ing. In Proc. of ICML, 2017.","title":"A Distributional Perspective on Reinforcement Learning","authors":"M G Bellemare, W Dabney, R Munos","year":"2017","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cartea and Jaimungal, 2015] Á. Cartea and S. Jaimungal. Order- Flow and Liquidity Provision. Available at SSRN 2553154, 2015.","title":"Order-Flow and Liquidity Provision","authors":"Á Cartea, S","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cartea et al., 2015] Á. Cartea, S. Jaimungal, and J. Penalva. Al- gorithmic and High-Frequency Trading. Cambridge University Press, 2015.","title":"Algorithmic and High-Frequency Trading","authors":"Á Cartea, S Jaimungal, J Penalva","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cartea et al., 2017] Á. Cartea, R. Donnelly, and S. Jaimungal. Al- gorithmic Trading with Model Uncertainty. SIAM Journal on Financial Mathematics, 8(1):635-671, 2017.","title":"Algorithmic Trading with Model Uncertainty","authors":"Cartea","year":"2017","venue":"SIAM Journal on Financial Mathematics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chan and Shelton, 2001] N. T. Chan and C. Shelton. An Electronic Market-Maker. 2001.","title":"An Electronic Market-Maker","authors":"N T Chan, C Shelton","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chou et al., 2017] P.-W. Chou, D. Maturana, and S. Scherer. Im- proving Stochastic Policy Gradients in Continuous Control with Deep Reinforcement Learning Using the Beta Distribution. In Proc. of ICML, 2017.","title":"Improving Stochastic Policy Gradients in Continuous Control with Deep Reinforcement Learning Using the Beta Distribution","authors":"P.-W Chou, D Maturana, S Scherer","year":"2017","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fodra and Labadie, 2012] P. Fodra and M. Labadie. High- frequency market-making with inventory constraints and direc- tional bets. arXiv preprint arXiv:1206.4810, 2012.","title":"Highfrequency market-making with inventory constraints and directional bets","authors":"P Fodra, M Labadie","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garcıa and Fernández, 2015] J. Garcıa and F. Fernández. A com- prehensive survey on safe reinforcement learning. JMLR, 16(1):1437-1480, 2015.","title":"A comprehensive survey on safe reinforcement learning","authors":"J Garcıa, F Fernández","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Glosten and Milgrom, 1985] L. R. Glosten and P. R. Milgrom. Bid, Ask and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders. Journal of Financial Eco- nomics, 14(1):71-100, 1985.","title":"Bid, Ask and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders","authors":"L R Glosten, P R Milgrom","year":"1985","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goodfellow et al., 2014] I.","title":null,"authors":"Goodfellow","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. In Proc. of NIPS, 2014.","title":"Generative Adversarial Nets","authors":"J Goodfellow, M Pouget-Abadie, B Mirza, D Xu, S Warde-Farley, A Ozair, Y Courville, Bengio","year":"2014","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gould et al., 2013] M. D. Gould, M. A. Porter, S. Williams, M. McDonald, D. J. Fenn, and S. D. Howison. Limit Order Books. Quantitative Finance, 13(11):1709-1742, 2013.","title":"Limit Order Books","authors":"M D Gould, M A Porter, S Williams, M Mcdonald, D J Fenn, S D Howison","year":"2013","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Grossman and Miller, 1988] S. J. Grossman and M. H. Miller. Liq- uidity and Market Structure. The Journal of Finance, 43(3):617- 633, 1988.","title":"Liquidity and Market Structure","authors":"S J Grossman, M H Miller","year":"1988","venue":"The Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Guéant and Manziuk, 2019] O. Guéant and I. Manziuk. Deep rein- forcement learning for market making in corporate bonds: beat- ing the curse of dimensionality. arXiv:1910.13205, 2019.","title":"Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality","authors":"O Guéant, I Manziuk","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Guéant et al., 2011] O. Guéant, C.-A. Lehalle, and J. Fernandez- Tapia. Dealing with the Inventory Risk: A solution to the mar- ket making problem. Mathematics and Financial Economics, 7(4):477-507, 2011.","title":"Dealing with the Inventory Risk: A solution to the market making problem","authors":"O Guéant, C.-A Lehalle, J Fernandez-Tapia","year":"2011","venue":"Mathematics and Financial Economics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Guéant, 2017] O. Guéant. Optimal market making. Applied Math- ematical Finance, 24(2):112-154, 2017.","title":"Optimal market making","authors":"O Guéant","year":"2017","venue":"Applied Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ho and Stoll, 1981] T. Ho and H. R. Stoll. Optimal Dealer Pricing Under Transactions and Return Uncertainty. Journal of Financial Economics, 9(1):47-73, 1981.","title":"Optimal Dealer Pricing Under Transactions and Return Uncertainty","authors":"T Ho, H R Stoll","year":"1981","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lagoudakis and Parr, 2003] M. G. Lagoudakis and R. Parr. Least- Squares Policy Iteration. JMLR, 4:1107-1149, 2003.","title":"Least-Squares Policy Iteration","authors":"M G Lagoudakis, R Parr","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Littman, 1994] M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proc. of ICML. 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"M L Littman","year":"1994","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Maddison et al., 2017] C. J. Maddison, D. Lawson, G. Tucker, N. Heess, M. Norouzi, A. Mnih, A. Doucet, and Y. W. Teh. Particle Value Functions. In ICLR 2017 Workshop Proceedings, 2017.","title":"Particle Value Functions","authors":"C J Maddison, D Lawson, G Tucker, N Heess, M Norouzi, A Mnih, A Doucet, Y W Teh","year":"2017","venue":"ICLR 2017 Workshop Proceedings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nevmyvaka et al., 2006] Y. Nevmyvaka, Y. Feng, and M. Kearns. Reinforcement Learning for Optimized Trade Execution. In Proc. of ICML, 2006.","title":"Reinforcement Learning for Optimized Trade Execution","authors":"Y Nevmyvaka, Y Feng, M Kearns","year":"2006","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Othman, 2012] A. Othman. Automated Market Making: Theory and Practice. PhD thesis, CMU, 2012.","title":"Automated Market Making: Theory and Practice","authors":"A Othman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pérolat et al., 2018] J. Pérolat, B. Piot, and O. Pietquin. Actor- Critic Fictitious Play in Simultaneous Move Multistage Games. In Proc. of AISTATS, 2018.","title":"Actor-Critic Fictitious Play in Simultaneous Move Multistage Games","authors":"J Pérolat, B Piot, O Pietquin","year":"2018","venue":"Proc. of AISTATS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peters and Schaal, 2008] J. Peters and S. Schaal. Natural Actor- Critic. Neurocomputing, 71(7-9):1180-1190, 2008.","title":"Natural Actor-Critic","authors":"J Peters, S Schaal","year":"2008","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pinto et al., 2017] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust Adversarial Reinforcement Learning. In Proc. of ICML, 2017.","title":"Robust Adversarial Reinforcement Learning","authors":"L Pinto, J Davidson, R Sukthankar, A Gupta","year":"2017","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rajeswaran et al., 2017] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine. EPOpt: Learning Robust Neural Network Policies using Model Ensembles. In Proc. of ICLR, 2017.","title":"EPOpt: Learning Robust Neural Network Policies using Model Ensembles","authors":"A Rajeswaran, S Ghotra, B Ravindran, S Levine","year":"2017","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rummery and Niranjan, 1994] G. A. Rummery and M. Niranjan. On-line Q-learning Using Connectionist Systems. Technical re- port, Department of Engineering, University of Cambridge, 1994.","title":"On-line Q-learning Using Connectionist Systems","authors":"G A Rummery, M Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sion, 1958] M. Sion. On General Minimax Theorems. Pacific Journal of Mathematics, 8(1):171-176, 1958.","title":"On General Minimax Theorems","authors":"M Sion","year":"1958","venue":"Pacific Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Spooner and Savani, 2020] T. Spooner and R. Savani. Robust Mar- ket Making via Adversarial Reinforcement Learning. arXiv preprint arXiv:2003.01820, 2020.","title":"Robust Market Making via Adversarial Reinforcement Learning","authors":"T Spooner, R Savani","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Spooner et al., 2018] T. Spooner, J. Fearnley, R. Savani, and A. Koukorinis. Market Making via Reinforcement Learning. In Proc. of AAMAS, pages 434-442, 2018.","title":"Market Making via Reinforcement Learning","authors":"T Spooner, J Fearnley, R Savani, A Koukorinis","year":"2018","venue":"Proc. of AAMAS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sutton and Barto, 2018] R. S. Sutton and A. G. Barto. Reinforce- ment Learning: An Introduction. MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"R S Sutton, A G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tamar et al., 2012] A. Tamar, D. Di Castro, and S. Mannor. Policy Gradients with Variance Related Risk Criteria. In Proc. of ICML, 2012.","title":"Policy Gradients with Variance Related Risk Criteria","authors":"A Tamar, D Di Castro, S Mannor","year":"2012","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Thomas, 2014] P. Thomas. Bias in Natural Actor-Critic Algo- rithms. In Proc. of ICML, 2014.","title":"Bias in Natural Actor-Critic Algorithms","authors":"P Thomas","year":"2014","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vyetrenko and Xu, 2019] S. Vyetrenko and S. Xu. Risk-Sensitive Compact Decision Trees for Autonomous Execution in Presence of Simulated Market Response. arXiv:1906.02312, 2019.","title":"Risk-Sensitive Compact Decision Trees for Autonomous Execution in Presence of Simulated Market Response","authors":"S Vyetrenko, S Xu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}