Robust Market Making via Adversarial Reinforcement Learning

1 Introduction

Market making refers to the act of providing liquidity to a market by continuously quoting prices to buy and sell a given financial instrument. The difference between these prices is called a spread, and the goal of a market maker (MM) is to repeatedly earn a spread by transacting in both directions. However, this is not done without risk. Market makers expose themselves to adverse selection, where toxic agents exploit their technological or informational edge, transacting with and thereby changing the MM's inventory before an adverse price move causes them a loss. This is known as inventory risk, and has been the subject of a great deal of research in optimal control, artificial intelligence, and reinforcement learning (RL) literature.

A standard assumption in existing work has been that the MM has perfect knowledge of market conditions. Robustness of market making strategies to model ambiguity has only recently received attention - Cartea et al. [2017] extend optimal control approaches for the market making problem to address the risk of model misspecification. In this paper, we deal with this type of risk by taking an adversarial approach. We design market making agents that are robust to adversarial and adaptively chosen market conditions by applying adversarial RL. Our starting point is a well-known single-agent mathematical model of market making of Avellaneda and Stoikov [2008], which has been used extensively in the quantitative finance [Cartea et al., 2015; Cartea et al., 2017; Gu´eant et al., 2011; Gu´eant, 2017]. We convert this into a discrete-time game, with a new "market player", the adversary, that can be thought of as a proxy for other market participants that would like to profit at the expense of the market maker. The adversary controls dynamics of the market environment in a zero-sum game against the market maker.

We evaluate the sensitivity of RL-based strategies to three core parameters of the market model dynamics that affect prices and execution, where each of these parameters naturally varies over time in real markets. We thus go beyond the fixed parametrisation of existing models - henceforth called the FIXED setting - with two extended learning settings. The RANDOM setting initialises each instance of the model (that is, episode in RL terminology) with values for the three parameters that are chosen independently and uniformly at random from three appropriate intervals. The STRATEGIC setting features an adversary - an independent "market" learner whose objective is to choose the parameters from these same intervals so as to minimise the cumulative reward of the market maker in a zero-sum game. The RANDOM and STRATEGIC settings are, on the one hand, more realistic than the FIXED setting, but on the other hand, significantly more complex for the market making agent. We show that market making strategies trained in each of these settings yield significantly different behaviour, and we demonstrate striking benefits of our STRATEGIC setting.

1.1 Contributions

The key contributions of this paper are as follows:

• We introduce a game-theoretic adaptation of a standard mathematical model of market making. Our adaption is useful to train robust MMs, and evaluate their performance in the presence of epistemic risk (Sections 2 and 3).
• We propose an algorithm for adversarial reinforcement learning in the spirit of RARL [Pinto et al., 2017], and

• demonstrate its effectiveness in spite of the well known challenges associated with finding Nash equilibria of stochastic games [Littman, 1994] (Sections 4 and 5).
• We thoroughly investigate the impact of three environmental settings (one adversarial) on learning market making. We show that training against a STRATEGIC adversary strictly dominates the other two settings (FIXED and RANDOM) in terms of a set of standard desiderata, including the Sharpe ratio (Section 5).
• We prove that, in several key instances of the STRATEGIC setting, the single-stage instantiation of our game has a Nash equilibrium resembling that found by our ARL algorithm for the multi-stage game. We then confirm broader existence of (approximate) equilibria in the multi-stage game by empirical best response computations (Sections 3 and 5).

1.2 RelatedWork

Optimal control and market making. The theoretical study of market making originated from the pioneering work of Ho and Stoll [1981], Glosten and Milgrom [1985] and Grossman and Miller [1988], among others. Subsequent work focused on characterising optimal behaviour under different market dynamics and contexts. Most relevant is the work of Avellaneda and Stoikov [2008], who incorporated new insights into the dynamics of the limit order book to give a new market model, which is the one used in this paper. They derived closed-form expressions for the optimal strategy of an MM with an exponential utility function when the MM has perfect knowledge of the model and its parameters. This same problem was then studied for other utility functions [Fodra and Labadie, 2012; Gu´eant et al., 2011; Cartea et al., 2015; Cartea and Jaimungal, 2015; Gu´eant, 2017]. As mentioned above, Cartea et al. [2017] study the impact of uncertainty in the model of Avellaneda and Stoikov [2008]: they drop the assumption of perfect knowledge of market dynamics, and consider how an MM should optimally trade while being robust to possible misspecification. This type of epistemic risk is the primary focus of our paper.

Machine learning and market making. Several papers have applied AI techniques to design automated market makers for financial markets. 1 Chan and Shelton [2001] focussed on the impact of noise from uninformed traders on the quoting behaviour of a market maker trained with reinforcement learning. Abernethy and Kale [2013] used an online learning approach. Spooner et al. [2018] later explored the use of reinforcement learning to train inventory-sensitive market making agents in a fully data-driven limit order book model. Most recently, Gu´eant and Manziuk [2019] addressed scaling issues of finite difference approaches for high-dimensional, multi-asset market making using model-based RL. While the approach taken in this paper is also based on RL, unlike the majority of these works, our underlying market model is taken from the mathematical finance literature. There, models are typically analysed using methods from optimal control. We justify this choice in Section 2. To the best of our knowledge, we are the first to apply ARL to derive trading strategies that are robust to epistemic risk.

1 A separate strand of work in AI and Economics and Computation has studied automated market makers for prediction markets, see e.g., Othman [2012]. While some similarities to the financial market making problem pertain, the focus in that strand of work focusses much more on price discovery and information aggregation.

Risk-sensitive reinforcement learning. Risk-sensitivity and safety in RL has been a highly active topic for some time. This is especially true in robotics where exploration is very costly. For example, Tamar et al. [2012] studied policy search in the presence of variance-based risk criteria, and Bellemare et al. [2017] presented a technique for learning the full distribution of (discounted) returns; see also Garcıa and Fern´andez [2015]. These techniques are powerful, but can be complex to implement and can suffer from numerical instability. This is especially true when using exponential utility functions which, without careful consideration, may diverge early in training due to large negative rewards [Maddison et al., 2017]. An alternative approach is to train agents in an adversarial setting [Pinto et al., 2017; P´erolat et al., 2018] in the form of a zero-sum game. These methods tackle the problem of epistemic risk 2 by explicitly accounting for the misspecification between train- and testtime simulations. This robustness to test conditions and adversarial disturbances is especially relevant in financial problems and motivated the approach taken in this paper.

2 TradingModel

We consider a standard model of market making as introduced by Avellaneda and Stoikov [2008] and studied by many others including Cartea et al. [2017]. The MM trades a single asset for which the price, Zn, evolves stochastically. In discrete-time,

where bn and σn are the drift and volatility coefficients, respectively. The randomness comes from the sequence of independent, Normally-distributed random variables, Wn, each with mean zero and variance ∆t. The process begins with initial value Z0 = z and continues until step N is reached.

The market maker interacts with the environment at each step by placing limit orders around Zn to buy or sell a single unit of the asset. The prices at which the MM is willing to buy (bid) and sell (ask) are denoted by p+n and p-n , respectively, and may be expressed as offsets from Zn:

these may be updated at each timestep at no cost to the agent. Equivalently, we may define:

called the quoted spread and reservation price, respectively. These relate to the agent's need for immediacy and bias in execution, amongst other things.

2 The problem of robustness has also been studied outside of the use of adversarial learning; see, e.g., [Rajeswaran et al., 2017].

In a given time step, the probability that one or both of the agent's limit orders are executed depends on the liquidity in the market and the values δ± n . Transactions occur when market orders, which arrive at random times, have sufficient size to consume one of the agent's limit orders. These interactions are modelled by independent Poisson processes, denoted by N + n and N - n for the bid and ask sides, respectively, with intensities λ±n ; not to be confused with the terminal timestep N. The dynamics of the agent's inventory process, or holdings, Hn, are then captured by the difference between these two terms,

where H0 is known and the values of Hn are constrained so that trading stops on the opposing side of the book when either limit is reached. The order arrival intensities are given by:

where A±n , k± n > 0 describe the rate of arrival of market orders and distribution of volume in the book, respectively. This particular form derives from assumptions and observations on the structure and behaviour of limit order books which we omit here for simplicity; see Avellaneda and Stoikov [2008], Gould et al. [2013], and Abergel et al. [2016] for more details.

In this framework, the evolution of the market maker's cash is given by the difference relation,

where ∆N ± n ≡ N ± n+1 - N ± n . The cash flow is a combination of: the profit due to executing at the premium δ± n , and the change in value of the agent's holdings. The total value accumulated by the agent by timestep n may thus be expressed as the sum of the cash held and value invested: Π(Xn, Hn, Zn), where

and Πn ≡ Π(Xn,Hn,Zn). This is known as the mark-tomarket (MtM) value of the agent's portfolio.

Why not use a data driven approach? Previous research into the use of RL for market making - and trading more generally - has focussed on data-driven limit order book models; see Nevmyvaka et al. [2006], Spooner et al. [2018], and Vyetrenko and Xu [2019]. These methods, however, are not amenable to the type of analysis presented in Section 3. Using an analytical model allows us to examine the characteristics of adversarial training in isolation while minimising systematic error due to bias often present in historical data.

3 Game Formulation and Single-Stage Analysis

We use the market dynamics above to define a zero-sum stochastic game between a market maker and an adversary that acts as a proxy for all other market participants.

Definition 1. [Market Making Game] The game between MM and an adversary has N stages. At each stage, MM chooses δ± and the adversary {b,A±, k±}. The resulting stage payoff is given by expected change in MtM value of the MM's portfolio, i.e., E[∆Π], see (7). The total payoff paid by the adversary to MM is the sum of the stage payoffs.

In the remainder of this section, we study theoretically the game when N = 1, i.e., when there is a single stage. Later, in Section 5, we analyse empirically the game for N = 200.

Single-stage analysis. At each stage, theMM's payoff may be unrolled to give:

For certain parameter ranges, this equation is concave in δ±.

Lemma 1 (Payoff Concavity in δ±). The payoff (8) is a concave function of δ± on the intervals [0, 2/k-b], and [0, 2/k+b], respectively.

Proof. The first derivative of the payoff w.r.t. δ± is given by:

The Hessian matrix is thus given by,

which is negative semi-definite iff δ± ≤ 2 k± ± b.

Note that (8) is linear in both b and A±. From this, we next show that there exists a Nash equilibrium (NE) when δ± and b are controlled, and k± and A± are fixed.

Theorem 1 (NE for fixed A±, k±). There is a pure strategy Nash equilibrium (δ± ⋆ , b⋆) for (δ+, δ-) ∈ [0, 2/k - b] × [0, 2/k + b] and b ∈ [b, b] (with finite b, b),

which is unique for |H| > 0. When H = 0, there is an equilibrium for every value b⋆ ∈ [b, b].

Proof. Equating (9) to zero and solving gives (10). To prove that these correspond to a pure strategy Nash Equilibrium of the game we show that the payoff is quasi-concave (resp. quasi-convex) in the MM's (resp. adversary's) strategy and then apply Sion's minimax theorem [Sion, 1958]. This follows from Lemma 1 for MM, and by the linearity of the adversary's payoff w.r.t. b. For |H| > 0, there is a unique solution. When H = 0, there exists a continuum of solutions, all with equal payoff.

The solution (10) has a similar form to that of the optimal strategy for linear utility with terminal inventory penalty [Fodra and Labadie, 2012], or equivalently that of a myopic agent with running penalty [Cartea and Jaimungal, 2015]. Interestingly, the extension of Theorem 1 to an adversary with control over all three model parameters {b, A±, k±} yields a similar result. In this case we refer the reader the extended version of the paper [Spooner and Savani, 2020] for a proof, and leave uniqueness to future work.

Theorem 2 (NE for general case). There exists a pure strategy Nash equilibrium (δ± ⋆ , {b⋆,A±⋆ , k± ⋆ }) of the MM game for (10), A±⋆ = A and k± ⋆ = k.

4 Adversarial Training

The single-stage setting is informative but unrealistic. Next we investigate a range of multi-stage settings with various different restrictions on the adversary, and explore how adversarial training can improve the robustness of MM strategies. The following three types of adversary in turn increase the freedom of the adversary to control the market's dynamics:

FIXED. The simplest possible adversary always plays the same fixed strategy, for us: bn = 0, A±n = 140 and k± n = 1.5; these values match those originally used by Avellaneda and Stoikov [2008]. This amounts to a single-agent learning setting with stationary transition dynamics.

RANDOM. The second type of adversary instantiates each episode with parameters chosen independently and uniformly at random from the following ranges: bn = b ∈ [-5, 5], A±n = A ∈ [105, 175] and k± n = k ∈ [1.125, 1.875]. These are chosen at the start of each episode and remain fixed until the terminal timestep N. This is analogous to single-agent RL with non-stationary transition dynamics.

STRATEGIC. The final type of adversary chooses the model parameters bn, An, kn (bounded as in the previous setting) at each step of the game. This represents a fully-adversarial and adaptive learning environment, and unlike the models presented in related work [Cartea et al., 2017], the source of risk here is exogenous and reactive to the quotes of the MM.

The principle of adversarial learning here - as with other successful applications [Goodfellow et al., 2014] - is that if the MM plays a strategy that is not robust, then this can (and ideally will) be exploited by the adversary. If a Nash equilibrium (NE) strategy is played by MM then their strategy is robust and cannot be exploited. While there are no guarantees that a NE will be reached in our case, we show in Section 5 via empirical best response computations that our ARL method does consistently converge to reasonable approximate NE. Moreover, we show that we consistently outperform previous approaches in terms of absolute performance and robustness to model ambiguity.

Robustness through ARL was first introduced by Pinto et al. [2017] who demonstrated its effectiveness across a number of standard OpenAI gym domains. We adapt their RARL algorithm to support incremental actor-critic based methods and facilitate asynchronous training; though many of the features remain the same. The adversary is trained in parallel with the market maker, is afforded the same access to state - including MM's inventory Hn - and uses the same method for learning, which is described below.

4.1 Learning Configuration

Both agents use the NAC-S(λ) algorithm, a natural actorcritic method [Thomas, 2014] for stochastic policies (i.e., mixed strategies) using semi-gradient SARSA(λ) [Rummery and Niranjan, 1994] for policy evaluation. The value functions are represented by compatible [Peters and Schaal, 2008]

radial basis function networks of 100 Gaussian prototypes with accumulating eligibility traces [Sutton and Barto, 2018].

States. The state of the environment sn = (tn,Hn) contains only the current time tn = nT N = n∆t and the agent's inventory Hn, where the transition dynamics are governed by the definitions introduced in Section 2.

Policies. The market maker learns a bivariate Normal policy for ˜ pn ≡ pn - Sn and ψn with diagonal covariance matrix. The mean and variance vectors are modelled by linear function approximators using 3 rd -order polynomial bases [Lagoudakis and Parr, 2003]. Both variances and the mean of the spread term, ψn, are kept positive via a softplus transformation. The adversary learns a Beta policy [Chou et al., 2017], shifted and scaled to cover the market parameter intervals. The two shape parameters are learnt the same as for the variance of the Normal distribution above, with a translation of +1 to ensure unimodality and concavity.

Rewards. The reward function is an adaptation of the optimisation objective of Cartea et al. [2017] and others, for the RL setting of incremental, discrete-time feedback:

where Πn refers to the MtM value of the agent's holdings (Eq. 7). This formulation can be either risk-neutral (RN) or risk-adverse (RA). For example, if η > 0 and ζ = 0, then the agent is punished if the terminal inventory HN is non-zero, and the solution becomes time-dependent. We refer to this case as RA and the case of η = ζ = 0 as RN.

5 Experiments

In each of the experiments to follow, the value function was pre-trained for 1000 episodes (with a learning rate of 10-3 ) to reduce variance in early policy updates. Both the value function and policy were then trained for 106 episodes, with policy updates every 100 time steps, and a learning rate of 10-4 for both the critic and policy. The value function was configured to learn λ = 0.97 returns. The starting time was chosen uniformly at random from the interval t0 ∈ [0.0, 0.95], with starting price Z0 = 100 and inventory H0 ∈ [H = -50, H = 50]. Innovations in Zn occurred with fixed volatility σ = 2 between [t0, 1] with increment ∆t = 0.005.

5.1 Market Maker Desiderata

Market makers typically aim to maximise profits while controlling inventory and quoted spread. Higher terminal wealth with lower variance, less exposure to inventory risk and smaller spreads are all indicators of the risk aversion (or lack thereof) of a strategy. This is reflected in the majority of objective functions for market markers studied in the literature. We consider the following quantitative desiderata:

Profit and loss / Sharpe Ratio. The distribution of a trading strategy's profit and loss is the most fundamental object used to measure its performance. In our setup, we look at an agent's distribution of episodic terminal wealth. In particular, we look at the moments E[ΠN] and V[ΠN] of this distribution, where the agent would like to maximise the former

• (a) Reservation price offset, ˜ p(t, H).

(b) Quoted spread, ψ(t, H).

• (c) Market makers trained against a STRATEGIC adversary.

Table 1: MM performance in terms of the standard desiderata. Each value was computed by evaluating the policy over 105 episodes and computing the mean and sample standard deviation. Each test episode was generated with a FIXED adversary. Wealth and average spread are measured in units of "currency"; inventory in terms of units of the "asset"; and the Sharpe ratio is unitless.

while minimising the latter. To this end, a common metric used in financial literature and in industry is the Shape ratio: E[ΠN ] V[ΠN] , in essence, the reward per unit of risk. 3 While larger values are better, it is important to note that the Sharpe ratio is not a sufficient statistic.

Terminal inventory. The distribution of terminal inventory, HN, tells us about the robustness of the strategy to adverse price movements. In practice, it is desirable for a market maker to finish the trading day with small absolute values for HN; though this is not always a strict requirement.

Quoted spread. The "competitiveness" of a market maker is often discussed in terms of the average quoted spread: En[ψn]. Tighter spreads imply more efficient markets and exchanges often compensate (with rebates) for smaller spreads.

5.2 Results

FIXED setting. We first trainedMMs against the FIXED adversary; i.e., a standard single-agent learning environment. Both RN and RA formulations of Eq. 11 were used with risk parameters η ∈ {1, 0.5, 0.1, 0.01} and ζ ∈ {0.01, 0.001}. Table 1a summarises the performance for the resulting agents. To provide some further intuition, we illustrate one of the learnt policies in Figure 1. In this case, the agent learnt to offset its price asymmetrically as a function of inventory, with increasing intensity as we approach the terminal time.

RANDOM setting. Next MMs were trained in an environment with a RANDOM adversary; a simple extension to the training procedure that aims to develop robustness to epistemic risk. To compare with earlier results, the strategies were also tested against the FIXED adversary - a summary of which, for the same set of risk parameters, is given in Table 1b. The impact on test performance in the face of model ambiguity was then evaluated by comparing market makers trained on the FIXED adversary with those trained against the RANDOM adversary. Specifically, out-of-sample tests were carried out in an environment with a RANDOM adversary. This means that the model dynamics at test-time were different from those at training time. While not explicitly adversarial, this misspecification of the model represents a non-trivial challenge for robustness. Overall, we found that market makers trained against the FIXED adversary exhibited no change in average wealth creation, but an increase of 98.1% in the variance across all risk parametrisations. On the other hand, market makers originally trained against the RANDOM adversary yielded a lower average 86.0% increase in the variance. The RANDOM adversary helps, but not by much compared with the STRATEGIC adversary, as we will see next.

3 While the Sharpe ratio would normally use returns, for our relative comparison of agents we use terminal wealth.

STRATEGIC setting. In this setting, we first consider an adversary that controls the drift b only. With RN rewards, we found that the adversary learns a (time-independent) binary policy (Figure 2) that is identical to the strategy in the corresponding single-stage game; see Section 3. We also found that the strategy learnt by the MM in this setting generates profits and associated Sharpe ratios in excess of all other strategies seen thus far when tested against the FIXED adversary (see Table 1c). This is true also when comparing with tests run against the RANDOM or STRATEGIC adversaries, suggesting that the adversarially trained MM is indeed more robust to test-time model discrepancies.

This, however, does not extend to STRATEGIC adversaries with control over either only A± or only k±. In these cases, performance was found to be no better than the corresponding MMs trained in the FIXED setting with a conventional learning setup. The intuition for this again derives from the singlestage analysis. That is, the adversary almost surely chooses a strategy that minimises A± (equiv. maximises k±) in order to decrease the probability of execution, thus decreasing MM's profits derived from execution and MM's ability to manage inventory effectively. The control afforded to the adversary must correlate in some way with sources of variance - such as inventory speculation - in order for robustness to correspond to practicable forms of risk-aversion.

Then the natural question is then whether an adversary with simultaneous control over b,A±, and k± produces strategies outperforming those where the adversary controls b alone. This is plausible since combining the three model parameters can lead to more interesting strategies, e.g., driving inventory up/down only to increase drift at the peak (i.e., pump and dump). We investigated by training an adversary with control over all three parameters. The resulting performance can be found in Table 1c, which shows an improvement in the Sharpe ratio of 0.27 and lower variance on terminal wealth. Interestingly, these MMs also quote tighter spreads on average - the values even approaching that of the risk-neutral MM trained against a FIXED adversary. This indicates that the strategies are able to achieve epistemic risk aversion without charging more to counterparties.

Exploring the impact of varying risk parameters of the reward function, η and ζ (see (11)), we found that in all cases MM strategies trained against a STRATEGIC adversary with an RA reward outperformed their counterparts in Tables 1a and 1b. It is unclear, however, if changes to the reward function away from the RN variant actually improved the strategy in general. Excluding when ζ = 0.001, all values appear to do worse than for an adversarially trained MM with RN reward. It may well be that the addition of inventory penalty terms actually boosts the power of the adversary and results in strategies that try to avoid trading at all, a frequent problem in this domain.

Verification of approximate equilibria. Holding the strategy of one player fixed, we empirically computed the best response against it by training. We found consistently that neither the trader nor adversary deviated from their policy. This suggests that our ARL method were finding reasonable approximate Nash equilibria. While we do not provide a full theoretical analysis of the stochastic game, these findings are corroborated by those in Section 3, since, as seen in Figure 2, the learned policy in the multi-stage setting corresponds closely to the equilibrium strategy from the singlestage case that was presented in Section 3.

6 Conclusions

We introduced a new approach for learning trading strategies with ARL. The learned strategies are robust to the discrepancies between the market model in training and testing. We show that our approach leads to strategies that outperform previous methods in terms of PnL and Sharpe ratio, and have comparable spread efficiency. This is shown to be the case for out-of-sample tests in all three of the considered settings, FIXED, RANDOM, and STRATEGIC. In other words, our learned strategies are not only more robust to misspecification, but also dominate in overall performance.

In some special cases we show that the learned strategies correspond to Nash equilibria of the corresponding singlestage game. More widely, we empirically show that the learned strategies correspond to approximate equilibria in the multi-stage stochastic game.

In future we plan to:

1. Extend to oligopolies of market makers.
2. Apply to data-driven and multi-asset models.
3. Further explore existence of equilibria and design provably-convergent algorithms.

Finally, we remark that, while our paper focuses on market making, the approach can be applied to other trading scenarios such as optimal execution and statistical arbitrage, where we believe it is likely to offer similar benefits. Further, it is important to acknowledge that this methodology has significant implications for safety of RL in finance. Training strategies that are explicitly robust to model misspecification makes deployment in the real-world considerably more practicable.

Software. All our code is freely accessible on GitHub: https://github.com/tspooner/rmm.arl.