{"type":"ScholarlyArticle","title":"IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making","authors":[{"name":"Hui Niu","slug":"hui-niu","given":"Hui","family":"Niu","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Siyuan Li","slug":"siyuan-li","given":"Siyuan","family":"Li","affiliation":"Faculty of Computing , Harbin Institute of Technology , Harbin , China"},{"name":"Jiahao Zheng","slug":"jiahao-zheng","given":"Jiahao","family":"Zheng","affiliation":"International Digital Economy Academy , Shenzhen , China"},{"name":"Zhouchi Lin","slug":"zhouchi-lin","given":"Zhouchi","family":"Lin","affiliation":"International Digital Economy Academy , Shenzhen , China"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"Skywork AI , Singapore; School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute for Interdisciplinary Information Sciences , Tsinghua University , Beijing , China"},{"name":"Jian Guo","slug":"jian-guo","given":"Jian","family":"Guo","affiliation":"International Digital Economy Academy , Shenzhen , China"}],"abstract":"Market making (MM) via Reinforcement Learning (RL) has attracted significant attention in financial trading. Most existing RL-based MM methods focus on optimizing single-price level strategies which fail at frequent order cancellations and loss of queue priority. By comparison, strategies involving multiple price levels align better with actual trading scenarios. However, given the complexity that multi-price level RL strategies involve a comprehensive trading action space, the challenge of effectively training RL persists. Inspired by the effective workflow of professional human market makers, we propose Imitative Market Maker (IMM), a novel RL framework leveraging knowledge from both suboptimal signal-based experts and direct policy interactions. Our framework starts with introducing effective state and action formulations that well encode information about multiprice level orders. Furthermore, IMM integrates a representation learning unit capable of capturing both short- and long-term market trends to mitigate adverse selection risk. Subsequently, IMM designs an expert strategy based on predictive signals, and trains the agent through the integration of RL and imitation learning techniques to achieve efficient learning. Extensive experimental results on four real-world market datasets demonstrate the superiority of IMM against current RL-based MM strategies.","doi":"10.24963/ijcai.2024/663","url":"https://ijcai.org/proceedings/2024/663","pdf":"https://ijcai.org/proceedings/2024/0663.pdf","markdown":"https://ijcai.org/proceedings/2024/663.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/663","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":663,"pages":{"first":5999,"last":6007},"keywords":["Multidisciplinary Topics and Applications: MTA: Finance","Machine Learning: ML: Representation learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abernethy and Kale, 2013] Jacob D. Abernethy and Satyen Kale. Adaptive market making via online learning. In NIPS, 2013.","title":"Adaptive market making via online learning","authors":"Jacob D Abernethy, Satyen Kale","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amihud and Mendelson, 1980] Yakov Amihud and Haim Mendelson. Dealership market: Market-making with in- ventory. Journal of Financial Economics, 8:31-53, 1980.","title":"Dealership market: Market-making with inventory","authors":"Mendelson Amihud, Yakov Amihud, Haim Mendelson","year":"1980","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Avellaneda and Stoikov, 2008] Marco Avellaneda and Sasha Stoikov. High frequency trading in a limit order book. Quantitative Finance, 8:217-224, 04 2008.","title":"Marco Avellaneda and Sasha Stoikov. High frequency trading in a limit order book","authors":"Stoikov Avellaneda","year":"2008","venue":"Quantitative Finance","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cartea et al., 2015a] Álvaro Cartea, Ryan Francis Donnelly, and Sebastian Jaimungal. Enhancing trading strategies with order book signals. Applied Mathematical Finance, 25:1 -35, 2015.","title":"Enhancing trading strategies with order book signals","authors":"Cartea","year":"2015","venue":"Applied Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cartea et al., 2015b] Álvaro Cartea, Sebastian Jaimungal, and Jose S. Penalva. Algorithmic and high-frequency trad- ing. 2015.","title":"Algorithmic and high-frequency trading","authors":"Cartea","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chakraborty and Kearns, 2011] Tanmoy Chakraborty and Michael Kearns. Market making and mean reversion. In ACM Conference on Economics and Computation, 2011.","title":"Market making and mean reversion","authors":"Tanmoy Chakraborty, Michael Kearns","year":"2011","venue":"ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chan and Shelton, 2001] Nicholas Tung Chan and Chris- tian R. Shelton. An electronic market-maker. 2001.","title":"An electronic market-maker","authors":"Nicholas Tung, Chan, Christian R Shelton","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chung et al., 2022] Gu Yoon Chung, Munki Chung, Yong- jae Lee, and Woo Chang Kim. Market making under order stacking framework: A deep reinforcement learning ap- proach. Proceedings of the Third ACM International Con- ference on AI in Finance, 2022.","title":"Market making under order stacking framework: A deep reinforcement learning approach","authors":"Chung Gu Yoon, Munki Chung, Yongjae Lee, Woo Chang, Kim","year":"2022","venue":"Proceedings of the Third ACM International Conference on AI in Finance","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fang et al., 2021] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with oracle policy distillation. In AAAI, 2021.","title":"Universal trading for order execution with oracle policy distillation","authors":"Yuchen Fang, Weiqing Kan Ren, Dong Liu, Weinan Zhou, Jiang Zhang, Yong Bian, Tie-Yan Yu, Liu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fernandez-Tapia, 2015] Joaquin Fernandez-Tapia. Model- ing, optimization and estimation for the on-line control of trading algorithms in limit-order markets. PhD thesis, 09 2015.","title":"Joaquin Fernandez-Tapia. Modeling, optimization and estimation for the on-line control of trading algorithms in limit-order markets","authors":"Fernandez-Tapia","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on ma- chine learning, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Fujimoto","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gašperov and Kostanjčar, 2021] Bruno Gašperov and Zvonko Kostanjčar. Market making with signals through deep reinforcement learning. IEEE Access, 9:61611- 61622, 2021.","title":"Market making with signals through deep reinforcement learning","authors":"Kostanjčar Gašperov, Bruno Gašperov, Zvonko Kostanjčar","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Glosten and Milgrom, 1985] Lawrence R. Glosten and Paul R. Milgrom. Bid, ask and transaction prices in a specialist market with heterogeneously informed traders. Journal of Financial Economics, 14:71-100, 1985.","title":"Bid, ask and transaction prices in a specialist market with heterogeneously informed traders","authors":"Lawrence R Glosten, Paul R Milgrom","year":"1985","venue":"Journal of Financial Economics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guéant and Manziuk, 2019] Olivier Guéant and Iuliia Manziuk. Deep reinforcement learning for market making in corporate bonds: Beating the curse of dimensionality. Applied Mathematical Finance, 26:387 -452, 2019.","title":"Deep reinforcement learning for market making in corporate bonds: Beating the curse of dimensionality","authors":"Manziuk Guéant, Olivier Guéant, Iuliia Manziuk","year":"2019","venue":"Applied Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Guéant et al., 2012] Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia. Dealing with the inventory risk: a solution to the market making problem. Mathematics and Financial Economics, 7(4):477-507, sep 2012.","title":"Dealing with the inventory risk: a solution to the market making problem","authors":"Olivier Guéant, Charles-Albert Lehalle, Joaquin Fernandez-Tapia","year":"2012","venue":"Mathematics and Financial Economics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2013] Weibing Huang, Charles-Albert Lehalle, and Mathieu Rosenbaum. Simulating and ana- lyzing order book data: The queue-reactive model. Papers 1312.0563, arXiv.org, December 2013.","title":"Simulating and analyzing order book data: The queue-reactive model","authors":"Huang","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jerome et al., 2022] Joseph Jerome, Gregory Palmer, and Rahul Savani. Market making with scaled beta policies. Proceedings of the Third ACM International Conference on AI in Finance, 2022.","title":"Market making with scaled beta policies","authors":"Joseph Jerome, Gregory Palmer, Rahul Savani","year":"2022","venue":"Proceedings of the Third ACM International Conference on AI in Finance","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jumadinova and Dasgupta, 2010] Janyl Jumadinova and Prithviraj Dasgupta. A comparison of different automated market-maker strategies. 2010.","title":"Janyl Jumadinova and Prithviraj Dasgupta. A comparison of different automated market-maker strategies","authors":"Dasgupta Jumadinova","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ke et al., 2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boost- ing decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 3149-3157, Red Hook, NY, USA, 2017.","title":"Lightgbm: A highly efficient gradient boosting decision tree","authors":"Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Curran Associates Inc. [Konda and Tsitsiklis, 1999]","title":null,"authors":"Curran Associates, Inc [ Konda, Tsitsiklis","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Vijay Konda and John Tsitsik- lis. Actor-critic algorithms. Advances in neural informa- tion processing systems, 12, 1999.","title":"Actor-critic algorithms","authors":"Vijay Konda, John Tsitsiklis","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lim and Gorse, 2018] Ye-Sheen Lim and Denise Gorse. Reinforcement learning for high-frequency market mak- ing. In ESANN, 2018.","title":"Reinforcement learning for high-frequency market making","authors":"Gorse ; Ye-Sheen Lim, Denise Lim, Gorse","year":"2018","venue":"ESANN","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Niu et al., 2022] Hui Niu, Siyuan K Li, and Jian Li. Meta- trader: An reinforcement learning approach integrating di- verse policies for portfolio optimization. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 2022.","title":"Metatrader: An reinforcement learning approach integrating diverse policies for portfolio optimization","authors":"Hui Niu, Siyuan K Li, Jian Li","year":"2022","venue":"Proceedings of the 31st ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Patel, 2018] Yagna Patel. Optimizing market mak- ing using multi-agent reinforcement learning. ArXiv, abs/1812.10252, 2018.","title":"Yagna Patel. Optimizing market making using multi-agent reinforcement learning","authors":"Patel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sadighian, 2019] Jonathan Sadighian. Deep reinforcement learning in cryptocurrency market making. arXiv: Trading and Market Microstructure, 2019.","title":"Deep reinforcement learning in cryptocurrency market making","authors":"Jonathan Sadighian","year":"2019","venue":"arXiv: Trading and Market Microstructure","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Spooner et al., 2018] Thomas Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis. Market making via reinforcement learning. In AAMAS, 2018.","title":"Market making via reinforcement learning","authors":"Thomas Spooner, John Fearnley, Rahul Savani, Andreas Koukorinis","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sun et al., 2023] Shuo Sun, Xinrun Wang, Wanqi Xue, Xi- aoxuan Lou, and Bo An. Mastering stock markets with efficient mixture of diversified trading experts. In Proceed- ings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 2109-2119, New York, NY, USA, 2023. Association for Computing Machinery.","title":"Mastering stock markets with efficient mixture of diversified trading experts","authors":"Shuo Sun, Xinrun Wang, Wanqi Xue, Xiaoxuan Lou, Bo An","year":"2023","venue":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Confer- ence on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2022] Ziyi Xu, Xue Cheng, and Yangbo He. Per- formance of deep reinforcement learning for high fre- quency market making on actual tick data. In AAMAS, 2022.","title":"Performance of deep reinforcement learning for high frequency market making on actual tick data","authors":"Ziyi Xu, Xue Cheng, Yangbo He","year":"2022","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu and Koltun, 2015] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhong et al., 2020] Yueyang Zhong, YeeMan Bergstrom, and Amy R. Ward. Data-driven market-making via model- free learning. In IJCAI, 2020.","title":"Data-driven market-making via modelfree learning","authors":"Zhong","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}