Objective-aware Traffic Simulation via Inverse Reinforcement Learning

1 Introduction

Traffic simulation has long been one of the significant topics in transportation. Microscopic traffic simulation plays an important role in planing, designing and operating of transportation systems. For instance, an elaborately designed traffic simulator allows the city operators and planers to test policies of urban road planning, traffic restrictions, and the optimization of traffic congestion, by accurately deducing possible effects of the applied policies to the urban traffic environment [Toledo et al., 2003]. The prior work [Wei et al., 2018] employs a traffic simulator to train and test policies of intelligent traffic signal control, as it can generate a large number of simulated data for the training of the signal controller.

∗Equal contribution. Part of the work was done when Guanjie Zheng and Hanyang Liu were interns at Tianrang Inc. and when Guanjie Zheng was completing his PhD at the Pennsylvania State University.

Current transportation approaches used in the state-of-theart traffic simulators such as [Yu and Fan, 2017; Osorio and Punzo, 2019] employ several physical and empirical equations to describe the kinetic movement of individual vehicles, referred to as the car-following model (CFM). The parameters of CFMs such as maximum acceleration and driver reaction time must be carefully calibrated using traffic data. A calibrated CFM can be exploited as a policy providing the vehicle optimal actions given the state of the environment, as shown in Figure 1. The optimality of this policy is enabled by the parameter calibration that obtains a close match between the observed and simulated traffic measurements.

An effective traffic simulator should produce accurate simulations despite variant dynamics in different traffic environments. This can be factorized into two specific objectives. The first objective is to accurately imitate expert vehicle behaviors given a certain environment with stationary dynamics. The movement of real-world vehicles depends on many factors including speed, distance to neighbors, road networks, traffic lights, and also driver's psychological factors. A CFM usually aims to imitate the car-following behavior by applying physical laws and human knowledge in the prediction of vehicle movement. Faced with sophisticated environment, models with emphasis on fitting different factors are continuously added to the CFM family. For example, Krauss model [Krauß, 1998] focuses on safety distance and speed, while Fritzsche model sets thresholds for vehicle's movement according to driver's psychological tendency. However, there is no universal model that can fully uncover the truth of vehicle-behavior patterns under comprehensive situations. Relying on inaccurate prior knowledge, despite calibrated, CFMs often fail to exhibit realistic simulations.

The second objective is to make the model robust to variant dynamics in different traffic environments. However, it is challenging due to the non-stationary nature of real-world traffic dynamics. For instance, weather shifts and variances of road conditions may change a vehicle's mechanical property and friction coefficient against the road surface, and eventually lead to variances of its acceleration and braking performance. In a real-world scenario, a vehicle would accordingly adjust its driving policy and behave differently (e.g., use different acceleration or speed given the same observation) under these dynamics changes. However, given a fixed policy (i.e., CFM), current simulators in general, fail to adapt policies to different dynamics. To simulate a different traffic environment with significantly different dynamics, the CFMmust be re-calibrated using new trajectory data with respect to that environment, which is inefficient and sometimes unpractical. This makes these simulation models less effective when generalized to changed dynamics.

We aim to achieve both objectives. For the first objective, a natural consideration is to learn patterns of vehicles' behaviors directly from real-world observations, instead of relying on sometimes unreliable prior knowledge. Recently, imitation learning (IL) has shown promise for learning from demonstrations [Ho and Ermon, 2016; Fu et al., 2018]. However, direct IL methods such as behavioral cloning [Michie et al., 1990] that aim to directly extract an expert policy from data, would still fail in the second objective, as the learned policy may lose optimality when traffic dynamics change. A variant of IL, inverse reinforcement learning (IRL), different from direct IL, not only learns an expert's policy, but also infers the reward function (i.e., cost function or objective) from demonstrations. The learned reward function can help explain an expert's behavior and give the agent an objective to take actions imitating the expert, which enables IRL to be more interpretable than direct IL. To this end, we propose to use an IRL-based method that build off the adversarial IRL (AIRL) [Fu et al., 2018], to train traffic simulating agents that generate accurate trajectories while simultaneously recovering the invariant underlying reward of the agents, which facilitates the robustness to variant dynamics.

With disentangled reward of an agent recovered from realworld demonstrations, we can infer the vehicle's true objective. Similar to the human driver's intention (e.g., drive efficiently under safe conditions), the estimated objective can be invariant to changing dynamics (e.g., maximum acceleration or deceleration). Considering the intricate real-world traffic with multiple vehicles interacting with each other, we extend AIRL to the multi-agent context of traffic simulation. We incorporate a scalable decentralized parameter sharing mechanism in [Gupta et al., 2017] with AIRL, yielding a new algorithm called Parameter Sharing AIRL (PS-AIRL), as a dynamics-robust traffic simulation model. In addition, we propose an online updating procedure, using the learned reward to optimize new policies to adapt to different dynamics in the complex environment, without any new trajectory data. Specifically, our contributions are as threefold:

• We propose an IRL-based model that is able to infer realworld vehicle's true objective. It enables us to optimize policies adaptive and robust to different traffic dynamics.
• We extend the proposed model with the parameter sharing mechanism to a multi-agent context, enabling our model good scalable capacity in large traffic.
• Extensive experiments on both synthetic and real-world datasets show the superior performance in trajectory simulation, reward recovery and dynamics-robustness of PSAIRL over state-of-the-art methods.

2 RelatedWork

Conventional CFM Calibration. Traditional traffic simulation methods such as SUMO [Krajzewicz et al., 2012], AIMSUN [Barcel ´o and Casas, 2005] and MITSIM [Yang and Koutsopoulos, 1996], are based on calibrated CFM models, which are used to simulate the interactions between vehicles. In general, CFM considers a set of features to determine the speed or acceleration, such as safety distance, and velocity difference between two adjacent vehicles in the same lane.

Learning to Simulate. Recently, several approaches proposed to apply reinforcement learning in simulation learning problems such as autonomous driving [Wang et al., 2018]. For these works, the reward design is necessary but complicated because it is difficult to mathematically interpret the vehicle's true objective. The prior work [Bhattacharyya et al., 2018] proposed to frame the driving simulation problem in multi-agent imitation learning, and followed the framework of [Ho and Ermon, 2016] to learn optimal driving policy from demonstrations, which has a similar formulation to our paper. But different from [Bhattacharyya et al., 2018], our traffic simulation problem has totally different concentrations. Our work mainly focus on the fidelity of simulated traffic flow and car-following behavior, as well as correct reactions to traffic light switchover, while [Bhattacharyya et al., 2018] ignores traffic signals, and underlines safe driving when interacting with neighboring vehicles. The work [Zheng et al., 2020] uses a similar framework for traffic simulation but is unable to achieve dynamic-robustness like our proposed model.

3 ProblemDefinition

We frame traffic simulation as a multi-agent control problem, regarding the complicated interaction between vehicles in traffic. Formally, we formulate our model in a decentralized partially observable Markov decision process (DPOMDP) defined by the tuple (M, {Sm}, {Am}, T , r, γ, ρ0). Here M denotes a finite set of agents, {Sm} and {Am} the sets of states and actions for each agent m, respectively. We have Sm ∈ RS and Am ∈ RA for all agents m ∈ M. ρ0 denotes the initial state distribution. r(s, a) is the reward function, and γ denotes the long-term reward discount factor. We assume for a set of given expert demonstrations, the environment dynamics remain unchanged. The deterministic state transition function is defined by T (s′ |s, a). We formulate the microscopic traffic simulation problem in a inverse reinforcement learning (IRL) fashion. Given trajectories of expert vehicles, our goal is to learn the reward function of the vehicle agent. Formally, the problem is defined as follows:

Problem 1 Given a set of expert trajectories as demonstrations D = {τ1, τ2, ..., τM} generated by an optimal policy π∗ (a|s) where τ = {s1, a1, ..., sT , aT }, the goal is to recover the reward function rθ(s, a), so that the log likelihood of the expert trajectories are maximized, i.e.,

Here, pθ(τ) ∝ exp{∑Tt=0 γ t rθ (st, at)} is the distribution of trajectories parameterized by θ in the reward rθ(s, a).

4 Method

4.1 Adversarial Inverse Reinforcement Learning

Along the line of IRL, [Finn et al., 2016b] proposed to treat the optimization problem in Eq. (1) as a GAN-like [Goodfellow et al., 2014] optimization problem. This adversarial training framework alternates between training the discriminator to classify each expert trajectory and updating the policy to adversarially confuse the discriminator, and optimize the loss:

where Dθ is the discriminator parameterized by θ, πE is the optimal policy for expert demonstration data D, and πψ is the learned policy parameterized by ψ in the generator. Specifically, the discriminator scores each sample with its likelihood of being from the expert, and use this score as the learning cost to train a policy that generates fake samples. Thus, the objective of the generator is to generate samples that confuse the discriminator, where logDθ(s, a) is used as a surrogate reward to help guide the generated policy into a region close to the expert policy. Its value grows larger as samples generated from πψ look similar to those in expert data.

We use the same strategy to train a discriminator-generator network. However, one limitation of [Finn et al., 2016b] is that, the use of full trajectories usually cause high variance. Instead, we follow [Finn et al., 2016a] to use state-action pairs as input with a more straightforward conversion for the

where f is a learned function, and policy π is trained to maximize reward r(s, a) = log(1 -D(s, a)) - logD(s, a). In each iteration, the extracted reward serves as the learning cost for training the generator to update the policy. Updating the discriminator is equivalent to updating the reward function, and in turn updating the policy can be viewed as improving the sampling distribution used to estimate the discriminator. The structure of the discriminator in Eq. (3) can be viewed as a sigmoid function with input f(s, a) - log π, as shown in Fig. 2. Similar to GAN, the discriminator is trained to reach the optimality when the expected D(s, a) over all state-action pairs is 1/2. At the optimality, f ∗ (s, a) = logπ∗ (a|s) and the reward can be extracted from the optimal discriminator by r(s, a) = f (s, a) + Const.

4.2 Dynamics-robust Reward Learning

We seek to learn the optimal policy for the vehicle agent from real-world demonstrations. So the learned policy should be close to, in the real world, the driver's policy to control the movement of vehicles given the state of traffic environment. Despite the non-stationary nature of traffic dynamics, the vehicle's objective during driving should constantly remain the same. This means that although the optimal policy under different system dynamics may vary, the vehicle constantly has the same reward function. Therefore, it is highly appealing to learn an underlying dynamic-robust reward function beneath the expert policy. By defining the traffic simulation learning as an IRL problem, we adapt the adversarial IRL (AIRL) as in Section 4.1 on the traffic simulation problem and propose a novel simulation model, which serves to not only imitate the expert trajectories (i.e., driving behaviors), but simultaneously recover the underlying reward function.

We aim to utilize the reward-invariant characteristics of AIRL to train an agent that simulates a vehicle's behavior in traffic while being robust to dynamics with changing traffic environment. The reward ambiguity is an essential issue in IRL approaches. [Ng et al., 1999] proposes a class of reward transformations that preserve the optimal policy, for any function Φ : S -→ R, we have

Algorithm 1: Training procedure of PS-AIRL

• Input: Expert trajectories τE ∼ πE Output: Policy πψ, reward rθ,φ 1 Initialize policy parameters ψ, and discriminator parameters θ and φ 2 for k←- 0, 1, . . . do 3 Rollout trajectories for all M agents ⃗τ = {τ1, τ2, ..., τM} ∼ πψk 4 Update θ, φ in discriminator given ⃗τ via minimizing Eπψ[log (Dθ,φ(s, a, s′ ))]+ EπE[log (1-Dθ,φ(s, a, s′ ))] 5 Update reward using discriminator output rθ,φ ←- log Dθ,φ - log(1 - Dθ,φ) 6 Update policy πψ with a TRPO step by solving the optimization in Eq. (7). Then πψk+1 ←- πψ.

If the true reward is solely a function of state, we are able to extract a reward that is fully disentangled from dynamics (i.e., invariant to changing dynamics) and recover the ground truth reward up to a constant [Fu et al., 2018].

To learn this dynamic-robust reward and ensure it remains in the set of rewards that correspond to the same optimal policy, AIRL follows the above reward transformation, and replaces the learned function f(s, a) in Eq. (3) with

where ˆ rθ(s) is the state-only reward approximator, hφ is a shaping term, and s′ = π(s, a) is the next state. The transformation in Eq. (5) ensures that the learned function fθ,φ corresponds to the same optimal policy as the reward approximator ˆ rθ does. Correspondingly, Dθ(s, a) in Eq. (3) becomes

As a result, the discriminator can be represented by the combination of the two functions ˆ rθ and hφ as shown in Fig. 2.

4.3 Parameter-sharing Agents

As illustrated in Section 3, we formulate the traffic simulation as multi-agent system problem by taking every vehicle in the traffic system as an agent interacting with the environment and each other. Inspired by parameter sharing trust region policy optimization (PS-TRPO) [Gupta et al., 2017], we incorporate its decentralized parameter sharing training protocol with AIRL in Section 4.1, and propose the parameter sharing AIRL (PS-AIRL) that learns policies capable of simultaneously controlling multiple vehicles in complex traffic environment. In our formulation, the control is decentralized while the learning is not. Accordingly, we make some simple assumptions for the decentralized parameter sharing agents. See Appendix for details 1 .

Under the decentralized parameter sharing training protocol as in [Gupta et al., 2017], our proposed PS-AIRL can be highly sample-efficient since it reduces the number of parameters by a factor M, and shares experience across all agents in the environment. We use TRPO [Schulman et al., 2015] as our policy optimizer, which allows precise control of the expected policy improvement during the optimization. For a policy πψ, at each iteration k, we perform an update to the policy parameters ψ by solving the following problem:

1 The appendix will be released on the authors' website and arxiv.

where πψk is the policy obtained in the previous iteration, Aψk (s,m, a) is an advantage function that can be estimated by the difference between the empirically predicted value of action and the baseline value. The training procedure of PSAIRL is shown in Algorithm 1. The training pipeline details can be found in the appendix.

5 Experiment

We generally aim to answer the following three questions.

• Q1: Trajectory simulation. Is PS-AIRL better at imitating the vehicle movement in expert trajectories?
• Q2: Reward recovery. Can PS-AIRL learn the vehicle's objective by recovering the reward function?
• Q3: Capability to generalize. Is the recovered reward function robust to dynamics changes in the traffic environment?

5.1 Data and Experiment Setting

We evaluate our proposed model on 3 different real-world traffic trajectory datasets with distinct road network structures collected from Hangzhou of China, and Los Angeles of US, including 3 typical 4-way intersections, a 4 × 4 network, and a 1 × 4 arterial network. See Appendix for details.

We compared PS-AIRL to two traditional CFM-based methods, CFM-RS and CRF-TS [Krauß, 1998], and three imitation learning-based models, BC [Michie et al., 1990], DeepIRL [Wulfmeier et al., 2015] and MA-GAIL [Zheng et al., 2020]. See Appendix for details.

We use root mean square error (RMSE) to evaluate the mismatch between ground truth traffic trajectories and the trajectories generated by the simulation model. In addition, to verify the capacity of PS-AIRL in recovering the underlying reward that can generalize to different dynamics, we compare the reward values each model reaches given a hand-crafted reward function. See Appendix for its definition.

5.2 Traffic Simulation

To answer Q1, we evaluate each model by using them to recover traffic trajectories, as shown in Table 1 and Figure 3.

We first compare our proposed algorithm with state-of-theart simulation algorithms CFM-RS and CFM-TS (as in Table 1). We can observe that our proposed PS-AIRL outperforms the baseline methods on all datasets. The two traditional methods CFM-RS and CFM-TS perform poorly due to the generally inferior fitting ability of physical CFM models against sophisticated traffic environment. We further compare PS-AIRL with state-of-art imitation learning methods. The results are shown in Figure 3. We can observe that PS-AIRL performs consistently better than other baselines. Specifically, as expected, PS-AIRL and MA-GAIL perform better than the other two, for they can not only model the sequential decision process (compared to BC) and utilize the adversarial process to improve the imitation (compared to DeepIRL). More importantly, PS-AIRL achieves superior performance over MA-GAIL (except in HZ-1 these two perform similarly in terms of RMSE of speed). It indicates that our proposed model benefits from the learned invariant reward. For instance, if one vehicle enter the system right after another, we may not expect it to accelerate a lot. In contrast, if one vehicle enter the system with no preceding vehicle, it may speed up freely within the speed limit. Behaviors may seem different under these two cases, but the reward function still remains the same. Therefore, armed with explicit reward learning, PS-AIRL is able to outperform.

Simulation Case Study. To have a closer look at the simulation of vehicle behavior, we extract the learned trajectory of 5 vehicles (vehicles with id 20, 40, 60, 80, 100) from HZ-3 as examples. Comparison between trajectories generated by different methods is shown in Fig. 4. The trajectories recovered by PS-AIRL are most similar to the expert ones.

5.3 Objective Recovery

To answer Q2, we leverage a hand-designed reward function as the ground truth and let the algorithms recover it. The reward increases when the vehicle approaches its desired speed and succeed in keeping safe distance from its preceding vehicles (see Appendix for details). Specifically, we use this hand-designed reward function to train an expert policy and use the generated expert trajectories to train each method. Note that, CFM-based methods are not included here for the following reasons. CFM-based methods can yield very unreal parameters (e.g., acceleration of 10 m/s2 ) to match the generated trajectory. These will cause collision of vehicles or even one vehicle directly pass through another. This is an unreal setting. Hence, these CFM-based methods can not recover the true policy that vehicles follow in the simulation.

Achieved Reward Value. We conduct experiments on HZ1, HZ-2, HZ-3 and GD, and the results are shown in Table 2. As expected, PS-AIRL achieves higher reward than other baselines and approach much closer to the upper bound reward value provided by the Expert. This indicates the better imitation (i.e. simulation) power of PS-AIRL.

Recovered Reward Function. We are also interested in whether PS-AIRL can recover the groundtruth reward function. Hence, we visualize the reward function learned by PSAIRL and the surrogate reward of MA-GAIL (the best baseline). Due to the space limit, results from other methods are omitted. They generally show even worse results than MAGAIL. We enumerate the reward values for different speed and gap values. (Other state feature values are set to default values.) As in Fig. 5, compared to MA-GAIL, PS-AIRL recovers a reward closer to the groundtruth yet with a smoother shape. Our designed reward function applies penalties on slow speed and unsafe small gap. So the expert policy trained with this groundtruth reward tends to drive the vehicle on a faster speed and keep it against the front vehicle out of the safety gap. It shows that PS-AIRL is able to learn this objective of the expert in choosing optimal policies from demonstrations, assigning higher reward values to states with faster speed and safe gap, while applying penalties on the opposite. In contrast, the surrogate reward learned by the MA-GAIL totally ignores the safe gap. In addition, when vehicles are out of the safe gap, PS-AIRL recovered a reward value closer to the groudtruth.

5.4 Robustness to Dynamics

To answer Q3, we transfer the model learned in Section 5.3 to a new system dynamics. Specifically, compared with Section 5.3, we change the maximum acceleration and deceleration of vehicles from initially 2 m/s2 and 4 m/s2 into 5 m/s2 and 5 m/s2 respectively. Other settings remain unchanged. Because the reward function remains unchanged (i.e., drivers still want to drive as fast as possible but keep a safe gap), reward-learning methods should be able to generalize well.

For DeepIRL, MA-GAIL and PS-AIRL, we use the learned reward function (surrogate reward function for MA-GAIL) to train a new policy. (BC policy is directly transferred because it do not learn reward function.) We also include a direct transfer version of PS-AIRL as comparison to demonstrate the necessity of re-training policy under new dynamics. Similar to the reward recovery experiment, CFMmodels also do not apply to this comparison.

Table 3 shows the achieved reward in the new environment.

PS-AIRL achieves highest reward, because with the invariant reward function learned from the training environment, PSAIRL can learn a new policy that adapts to the changed environment. DeepIRL perform poorly in transferred environment because of its incapability of dealing with large number of states. PS-AIRL (D) fails to generalize to the new environment, which indicates that the policy learned in the old dynamics are not applicable anymore. MA-GAIL, though retrained with the surrogate reward function, can not transfer well, due to the dependency on the system dynamics.

6 Conclusion

In this paper, we formulated the traffic simulation problem as a multi-agent inverse reinforcement learning problem, and proposed PS-AIRL that directly learns the policy and reward function from demonstrations. Different from traditional methods, PS-AIRL does not need any prior knowledge in advance. It infers the vehicle's true objective and a new policy under new traffic dynamics, which enables us to build a dynamics-robust traffic simulation framework with PS-AIRL. Extensive experiments on both synthetic and realworld datasets show the superior performances of PS-AIRL on imitation learning tasks over the baselines and its better generalization ability to variant system dynamics.

Acknowledgments

This work was supported in part by NSF awards #1652525, #1618448 and #1639150. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.