Robust Reinforcement Learning via Progressive Task Sequence

1 Introduction

Reinforcement learning (RL) has recently achieved beyondhuman performance on a wide range of decision-making tasks like games [Silver et al., 2017], continuous control [Lyu et al., 2022] and robotics [Dalal et al., 2021]. In order to reduce risk of costly failure, RL agents are typically trained in a high-precision simulator. However, due to the inevitable reality gap, these algorithms are often overfitted to simulation environments and fail to generalize to the real world. In some safety-sensitive scenarios such as autonomous driving and medical treatment, using a bad policy can be costly and dangerous [Thomas and Brunskill, 2016]. To this end, learning policies that are robust to environmental shifts or mismatched configurations are becoming increasingly important.

∗Corresponding author.

Recent efforts in robust RL focus on performing perturbations at training time to achieve robustness. There are three main branches of the prior works that diverse in how the perturbations are introduced: 1) the Robust Markov Decision Process (RMDP)-based methods [Nilim and El Ghaoui, 2005; Iyengar, 2005; Wiesemann et al., 2013], where the perturbations are modeled as an uncertainty set to be introduced over different elements of the MDP tuple; 2) the Domain Randomization (DR) [Jakobi, 1997; Peng et al., 2018], a method that manipulates environmental perturbations by randomizing a set of parameters, which requires a delicate definition of the aspects that the agent should be robust to; and 3) the adversarial methods [Pinto et al., 2017; Vinitsky et al., 2020; Tessler et al., 2019; Dennis et al., 2020; Jiang et al., 2021], which introduces adversarial agents to automatically generate perturbations by applying external forces to the protagonist agent, formulating the robust RL as a max-min game. More recently, intrinsic motivation methods are proposed to encourage agents to learn robust policies, using the intrinsically motivated reward signal from evolutionary techniques [Song and Schneider, 2022; Niekum, 2010], distance constrains [Abdullah et al., 2019] and so on.

Despite the impressive progress, existing studies typically tackle the robust RL problem under the max-min settings, i.e., to maximize the return of the agent under worst-case perturbations. However, the max-min formulation often leads to overly conservative solutions and unstable training process

[Zhang et al., 2020; Yu et al., 2021]. In order to prevent the performance under the worst cases from being too bad, the max-min methods scarify the reward under non-worst cases. Moreover, as existing robust RL framework lacks the effective perturbation organization and programming during training, the worst-case perturbations may break the training stability and even bring a failure of policy learning. The same occurs in the field of classification problem, where the optimization technique is used to find the optimal hyperplane with the largest margin, i.e., to maximize the margin of the closest pair of classes. However, Gao et. al [Gao and Zhou, 2013] disclosed that maximizing the minimum margin does not necessarily lead to better performance, and instead, it is more crucial to optimize the margin distribution. Thus, the optimal margin distribution machine [Zhang and Zhou, 2018] is proposed, which is proved to have better generalization performance than large margin-based methods.

Inspired by the above works, an important insight brought by this work is to formulate the robust RL as a maxexpectation problem, where the agent is trained to maximize the expected reward simultaneously under as many perturbations as possible including the worst cases and the non-worst cases. We argue that for the max-min methods, the conservative solutions are caused by the limited knowledge space for the agent, which is only composed of the environments under worst-case perturbations. By introducing the "expectation", we aim to extend the knowledge space for the agent to explore the optimal policy, and thus to avoid conservative policies. Figure 1 shows the difference between the two robust formulation. However, the max-expectation formulation inevitably makes finding of the robust policy a more complex optimization problem. This presents us with a new question: how to organize the learning process to effectively solve the max-expectation optimization problem?

To solve the question, in this paper, we propose a novel approach, i.e., Dynamic Robust RL named DRRL, where we model the robust RL problem as a progressive multi-task learning problem. The core of DRRL is an effective perturbation (i.e., task) programming. Through the genetic algorithm (GA) based task generation and task evaluation, a group of feasible tasks are generated iteratively, which satisfies the appropriate difficulty level for the current policy. With the progressive task sequence, DRRL improves the policy robustness and the training stability via progressive task-oriented learning. In summary, our contributions are highlighted as follows:

• We make the first attempt to formulate the robust RL as a max-expectation optimization problem, which extends the agent's knowledge space about the RL task, and thus to avoid conservative solutions.
• We propose a novel approach DRRL to effectively solve the max-expectation optimization problem, which introduces progressive multi-task learning to improve the policy robustness and training stability.
• We conduct extensive experiments on high-dimensional continuous environments (i.e., Hopper, HalfCheetah, Walker2d and CarRacing) to validate the effectiveness of DRRL compared with state-of-the-art methods.

2 ProblemFormulation

Reinforcement learning is typically formalized as Markov Decision Processes (MDPs): M = (S; A; P; r; γ; s0), where S is a set of states, and A is a set of actions, P: S ×A×S → [0, 1] is the state transition probability describing the system's dynamics, r: S ×A → R is the reward function measuring the agent's performance, γ ∈ [0, 1) is the discount factor, and s0 is the initial state distribution. The learning goal for a given MDP is to obtain an optimal policy π∗ : S ×A → [0, 1] that maximizes the expected cumulative discounted reward:

It is worth noting that the cumulative discounted reward is conditioned on the transition probability P, which is always fixed in ordinary RL [Chen et al., 2021]. Motivated by real-world applications, a good RL policy should generalize well across a range of different transition probabilities. That is, a good RL policy should be robust to variations in dynamics. Taking OpenAI gym's CarRacing scenario [Brockman et al., 2016] as an example, we aim to learn a policy for autonomous vehicles that can run not only on high-grade roads (corresponding to one transition function) but also on icy roads (corresponding to another transition function).

Thus, instead of learning an optimal policy on a fixed transition probability (as shown in Eq. 1), the goal of robust RL is to learn a policy that is robust to a group of feasible transition probabilities. Generally, robust RL is achieved by solving the following max-min optimization problem [Wiesemann et al., 2013; Kuang et al., 2022]

P(·) is the set of possible transition probabilities. In this case, the objective of robust RL is to learn best-case policies under worst-case transitions. However, in the worst-cases, the agent is trained to always take conservative actions to avoid performing catastrophic actions [Lecarpentier and Rachelson, 2019]. Thus, the expected rewards in some best cases are inevitably reduced, which is obviously not the optimal solution for real-world applications.

In our work, instead of the max-min game, we formulate the robust RL problem as an expectation optimization, where the learning goal is to maximize the expected reward among all the possible transition probabilities:

To solve the expectation optimization problem in Eq. 3, we tackle the robust RL issue as a multi-task learning problem via modeling transition probabilities as the tasks, and propose a progressive robust policy optimization framework DRRL, which is detailed in the following section.

which satisfy the appropriate difficulty level for the current policy πt-1. Based on KGt, policy πt-1 will be trained and updated to πt. In this way, the feasible tasks at different iterations dynamically form an easy-to-hard tasks sequence. Finally, through task (or perturbation) organization and programming, we can accelerate the policy convergence and improve the policy generalization performance.

3.2 Task Generation and SequencingMechanism

Existing perturbation-based robust RL approaches perform perturbations on state observations, actions, or transition probabilities in the learning process. Unfortunately, most methods lack guarantees of the perturbation extents of the transition functions. Hence, as a prerequisite, we adapt the Wasserstein constraint [Abdullah et al., 2019; He et al., 2022] to restrict the admissible perturbed transition probabilities within a Wasserstein ball centered at the original transition probability P0 of the RL simulator:

Here, ϵ ∈ R+ is a hyper-parameter used to specify the maximumWasserstein bound. In our method, only the transition probabilities that satisfy the Wasserstein constraint (i.e., P ∈ Wϵ(P0)) will be chosen for robust RL training.

Task Generation

Sampling tasks fromWϵ(P0) directly, and filtering out feasible tasks to get DCT may not be the most sample efficient way. Thus, we adopt genetic algorithms (GAs) to gradually generate appropriate tasks. GA is a kind of search algorithm inspired by the process of natural selection, where a population of size NP is evolved iteratively [Pan et al., 2019]. In our GA variant, the population is composed of candidate tasks k, represented by a group of simulation parameters k = {p1, p2, · · ·, pn}, where the simulation parameters are restricted to obtain the variations of transition dynamics satisfying the Wasserstein constraints in Eq. 5. For each task k, we calculate the fitness score as:

3 Methodology

3.1 DRRLOverview

For our defined robust RL, we have the following two assumptions:

• Modeling errors of transition probability can be viewed as a set of RL tasks, in which each task corresponds to a specific variant of P0. Here, P0 represents the original transition probability in simulation.
• If the trained optimal policy is robust to a sequence of well-designed RL tasks, then it will have high probability to be robust to modeling errors or mismatch between simulation and real world.

Based on the above assumptions, our proposed DRRL tackles robust RL as a dynamic multi-task learning process on progressive task sequence.

Definition 1 (Dynamic Progressive Tasks). Dynamic progressive tasks are a sequence of task groups over a maximum of T training steps: DCT = {KG1|π0,KG2|π1, ...,KGt|πt-1, ...,KGT |πT-1}, in which KGt|πt-1 = {k1 t , k2 t , ..., ki t , ...}. Each single task ki t corresponds to a specific variant of the original transition probability P0:

such that the following three conditions are satisfied.

• The training difficulty levels (DLs) of tasks in the same group are much the same, i.e., DL(k1 t ) ≈ DL(k2 t ) · ··.
• The tasks of KGt are generated dynamically according to the current policy πt-1, i.e., KGt ∼ tasks(πt-1).
• The training difficulty levels of task groups gradually increase, i.e., DL(KGt) < DL(KGt+1).

Thus, the progressive multi-task learning trains a robust policy with easier tasks first, and then gradually increases the difficulty level until all the target tasks are traversed.

As shown in Figure 2, we propose Dynamic Robust Reinforcement Learning (DRRL). At the iteration t of training, tasks are firstly generated by the GA-based task generation module, where the fitness function is defined as the increase of reward. Then, according to the tasks' expected rewards, the task evaluation module will filter out feasible tasks KGt, Here, R(πi|k) and R(πi-1|k) are the task's expected reward of the current policy and the last policy, respectively. The fitness function defines the difficulty of a task to be the increase of reward, taking into account the policy change between the previous iteration and the current iteration [Xu et al., 2020]. The task with high fitness indicates that the policy improves the task's reward significantly in the current iteration, and is more likely to learn better in the next iteration. On the contrary, the task with low fitness indicates that (1) the current policy does not have enough competence to handle it and may wait to be learned at a later iteration; or (2) the current policy performs enough well on this task having no space to further increase and do not need to be trained repeatedly. Therefore, the tasks with higher fitness are viewed with more appropriate difficulty for the current policy.

The goal is to find a group of tasks with higher fitness scores. At every generation, each task k is evaluated using Eq. 6, produces a fitness score, and is sorted according to the score. The top K candidates (called parents) are kept to the next generation. Then a two-point crossover of the selected parents is used to generate their child candidates (called offspring). Also, random mutation is applied to the child candidates with the mutation rate Pm. By running such a GA algorithm, we aim to gradually generate appropriate tasks for the current policy.

Task Evaluation

In task generation, a group of training tasks with high fitness scores are outputted. However, the increase-based fitness function neglects the current expected reward. For example, one task with a high fitness score may have an extremely low expected reward and is still a difficult task for the current policy. Thus, we consider the tasks with both high fitness scores and appropriate expected rewards to be feasible tasks, used for training the policy. Here, we give a definition on the feasible task:

Definition 2 (Feasible Task). For a policy π, we consider a task k to be a feasible task if π on task k is able to receive expected reward Rmin ≤ R(π|k) ≤ Rmax.

On the one hand, R(π|k) ≥ Rmin guarantees task k can obtain enough reward and the policy π can reach convergence more easily. On the other hand, R(π|k) ≤ Rmax ensures we do not generate feasible tasks repeatedly from the already mastered tasks, as these mastered tasks usually have high rewards. The expected reward constraint in Definition 2 further guarantees the feasible tasks are at the appropriate level of difficulty for the current policy. After the task evaluation, only the feasible tasks KG are filtered out and will be used

Algorithm 1 Iterative training of DRRL

Input: Simulator environment ε; Size of the population NP; Size of the parent population K; Training iterations limit N; Generation limit G; Wasserstein constraint Wϵ(P0) Output: Robust policy πN DCT ← {KG0 = ∅,KG1 = ∅, · · · ,KGN = ∅} KG0 ← {ε} π0 ← initialize policy(KG0) R(π0) ← E[∑T-1 t=0 γ t · r(st-1, at-1, st, at)|ε] R1min ← wmin · R(π0), R1max ← wmax · R(π0) 1: for i = 1 to N do // task generation 2: population ← sample task(Wϵ(P0),NP) 3: for m = 1 to M do 4: fitness ← calculate fitness(population) 5: parents ← select(f itness, K) 6: population ← crossover and mutate(parents, NP ) 7: end for // task evaluation 8: for k in population do 9: R(πi-1|k) ← E[∑T-1 t=0 γ t · r((st-1, at-1, st, at)|k] 10: if Rimin ≤ R(πi-1|k) ≤ Rimax then 11: KGi ← add task(k) 12: end if 13: end for // policy update 14: πi ← update policy(πi-1,KGi) 15: R(πi) ←Ek∼DCT [E[∑T-1 t=0 γ t · r(st-1, at-1, st, at)|k]] 16: Ri+1 min ← wmin · R(πi), Ri+1 max ← wmax · R(πi) 17: end for 18: return πN

for updating the policy. In each training iteration, the final feasible tasks which are firstly generated by task generation and then filtered out by task evaluation, meet the difficulty requirement and guarantee the training stability.

Task Sequencing

In progressive multi-task learning, the difficulty levels for the feasible tasks should be updated as the policy is constantly updated. In other words, the values of Rmin and Rmax should be updated dynamically. For instance, at iteration m, we generate a group of feasible tasks KGm for the current policy. Then, we update the policy to πm on these tasks and obtain the expected reward:

KGi represents the generated feasible tasks at iteration i, and DCT (m) = ⋃mi=0 KGi is the union of all generated feasible tasks in the first m iterations. In order to set appropriate difficulty level for the feasible tasks according to current expected reward R(πm), Rmin and Rmax will be updated as:

Here, wmin and wmax are hyper-parameters. We suggest that wmin ∈ (0.5, 1) and wmax ∈ (1, 1.5). With new Rmin and Rmax, new task evaluation and policy training will be performed iteratively.

3.3 Robust Policy Network

Traditional RL policy network takes the current state s as input and outputs a policy π(s, a) = Pr(at = a|st = s), that gives an agent the probability of taking action a in state s.

In real-world applications, there are always several latent parameters that fail to be modeled in state space S, such as friction in autonomous driving scenarios. Traditional policy networks just take the current state st as input and cannot perceive environmental setting variants. Consequently, an agent will tend to take conservative actions to avoid catastrophic events in some worst-cases. To solve this problem, we improve the policy network as follows:

The agent would take action at according to not only the current state st but also the previous state st-1 and previous action at-1. Taking the CarRacing scenario as an example, although the friction value of the testing scenario is difficult to obtain, we can infer the friction value indirectly. Generally, an autonomous vehicle may show different driving statuses given the same state and action due to different friction coefficients. Hence, in principle, if an autonomous vehicle takes action at-1 at state st-1, making the state transfer to a new state st, we can extrapolate the current friction correspondingly. By using st-1, at-1, and st as pre-conditions of policy π, the improved policy network can represent possible latent environmental parameters in the training/test scenario.

As shown in Figure 3, if an autonomous vehicle takes action turn right at state st-1, and transfer to new state st, we can infer that the current friction is suitable. Then, according to policy π∗ (at|st, st-1, at-1), the autonomous vehicle should take action accelerate. Accordingly, the final convergent policy will perform uniformly well in all cases.

The input space of the proposed policy network will be intuitively increased from |s| to |s|2 |a|. Indeed, there are high temporal correlations between st-1 and st. An agent takes action at-1 at state st-1 and transfers to state st, and meanwhile the number of possible actions is small. Hence, the input space can be reduced to |s||a|2 . In particular, for a given environmental parameter setting where the transition probability is fixed, π∗ (at|st, st-1, at-1) = π∗ (at|st).

In summary, the proposed policy network will not bring about a serious increase in input space. Moreover, it can generate a separate policy for each environment parameter setting, while improving the overall performance on different transition probability variations.

3.4 Progressive Robust Policy Optimization

As mentioned earlier, we attempt to train a robust policy π∗ (at|st, st-1, at-1) to maximize the expected reward in Eq. 10. Algorithm 1 outlines our approach in detail.

Initially, we train a policy based on the initial environment parameters of simulator. For the initial policy π0, we calculate the expected reward R(π0) used to specify the appropriate level of difficulty for the feasible tasks at the first iteration (i.e., R1min and R1max ). At each iteration i, we firstly use the GA to generate a group of appropriate tasks. The initial population of size NP is sampled from Wϵ(P0). At each generation m, the increase-based fitness scores are calculated based on Eq. 6, and the top K candidates sorted by the fitness scores are selected as parents. After the crossover and mutation, a new population is generated, where all the candidates satisfy the Wasserstein constraint Wϵ(P0). By iteratively running GA, a group of tasks with high fitness scores are outputted. Next in the task evaluation, we calculate each task's expected reward. The tasks satisfying the current difficulty level (see Definition 2) will be labeled as feasible tasks for the current policy and added to the KGi. Then, we use feasible tasks KGi to update policy πi-1, with the objective function given by Eq. 10. Noting that, in our DRRL, any RL algorithm can be used as the optimizer to train the policy. At last, we update Ri+1 min and Ri+1 max according to the new policy πi, making the entire training process from easy to hard. The above process is repeated until convergence or the maximum number of iterations N is reached, and the current policy will be outputted as the final robust policy.

4 Experimental Results and Analysis

In our experiments 1 , we validate the following hypotheses:

• H1. Based on the max-expectation formulation, DRRL is able to improve the training stability and policy robustness compared to state-of-the-art methods.

H2. The proposed robust policy network enables the agent to learn from the history, leading to improved generalisation across environments with varying dynamics.

H3. The task generation and sequencing mechanism in DRRL provides better control over the perturbation difficulty, with the progressively hard task sequence.

1 The code is available at https://github.com/li-yike/DRRL.

4.1 H1. Robustness and Training Stability Analysis

In this evaluation, we implement the Hopper, HalfCheetah, and Walker2d benchmarks using OpenAI gym [Brockman et al., 2016] with the MuJoCo simulator [Todorov et al., 2012].

We compare our method with four state-of-the-art baselines. (1) TRPO [Schulman et al., 2015], that realizes a strong RL without perturbations. (2) RARL [Pinto et al., 2017], that trains the protagonist agent in the presence of an adversary agent's perturbations. (3) RAP [Vinitsky et al., 2020], that uses a population of adversary agents for perturbations. The size of the adversary population is 3 (called RAP-3), in our implementation. (4) DR [Mehta et al., 2020], that trains the agent under a set of random perturbations.

In our experiments, we use TRPO as the policy optimizer implemented by a neural network consisting of three hidden layers with 100 neurons each. We set the learning rate as 0.01 and other hyper-parameters (e.g., batchsize, discount factor) are tuned by grid search. For the GA implementation, the population size NP, the parent population size K, and the mutation rate Pm are set as 250, 50 and 0.9, respectively.

Training Stability

We analyze the training stability by comparing the mean and standard deviation of rewards at each training iteration. The comparison is shown in Figure 5, from which we can observe that for all the three environments, DRRL leads to a more stable increase of reward and a smaller standard deviation. These empirically highlight that by controlling perturbation difficulty and implementing training in a scheduled way, our DRRL gains better training stability.

We then evaluate the learned policy in original environments without perturbations, and the results are shown in the first part of Table 1. We can observe that across the three environments, generally DRRL matches or outperforms all the baselines. The DRRL policy reaches the highest mean reward in HalfCheetah and Walker2d, with 8.05%, and 12.04% improvements compared to the second best results, respectively. This indicates that, although trained under perturbations, our DRRL still achieves high rewards under no perturbations.

Robustness under Adversarial Perturbations

To simulate the uncertain effects of the latent parameters in the real world, following the idea of RARL, we train another adversarial agent to add adversarial forces to destabilize the protagonist agent. Under the well-trained adversary, the performance of the protagonist agent is shown in the second part of Table 1, from which we can observe that our method consistently outperforms all the baselines. The average improvements of our DRRL to the second best results are 66.39% in Hopper, 7.65% in HalfCheetah, and 25.62% in Walker2d. It proves that our method improves the robustness in the presence of adversarial perturbations, which enables the agent to better adapt to the latent environmental effects.

Robustness under Test Conditions

Further, we measure the robustness under a variety of test conditions, where different torso mass and friction coefficients are set for the three environments. The agent's rewards across all combinations are visualized in Figure 4, where each square denotes the mean reward of five trials. We observe that for all three environments, DRRL shows a significant outperformance on robustness against various test conditions. Note that, for Walker2d, the robust policy learned by DRRL is generalized to nearly all environmental settings instead of just earning high rewards locally. In addition, the quantitative experimental results are shown in the third part of Table 1 which reports the mean reward with standard deviations under all test conditions. Clearly, the performance of our DRRL is significantly higher than all baselines in these benchmarks.

4.2 H2. Effectiveness of Robust Policy Network

We conduct the following ablation study to demonstrate the effectiveness of the robust policy network in DRRL. Here, we compare DRRL with a variant DRRLTPN, where the robust policy network (RPN) π∗ (at|st, st-1, at-1) is replaced by the traditional policy network (TPN) π∗ (at|st) while other parts are kept unchanged. The ablation study is implemented in a new benchmark CarRacing, a top-down car racing from pixels environment with a three-dimensional continuous action space (i.e., steer, gas, brake). We introduce perturbations by setting friction coefficient f ∈ [0.6, 1], gas error Egas ∈ [0, 0.4], steer error Esteer ∈ [0, 0.4], and brake error Ebrake ∈ [0, 0.4].

Figure 6(a) shows the performance of the racing car under the handcrafted perturbed environments, where Egas = 0.4, Ebrake = 0.4, Esteer = 0.4, and f ∈ {0.6, 0.65, 0.7}. Besides, the performance under the test conditions is shown in Figure 6(b). To generate the test conditions, we take Egas ∈ {0, 0.2, 0.4}, Esteer ∈ {0, 0.2, 0.4}, Ebrake ∈ {0, 0.2, 0.4} and f ∈ {1, 0.8, 0.6}, and apply them to different combinations (81 in total). As presented in both figures, overall, our DRRL is consistently superior to other methods. PPO is least competitive, illustrating that PPO lacks robustness when variations are introduced in the dynamics. DRRLTPN achieves better performance, which demonstrates the benefits of progressive multi-task learning with appropriate task difficulty. However, it still underperforms the DRRL, illustrating our proposed robust policy network is vital to improve generalisation across varying dynamics.

4.3 H3. Stepwise Analysis

Here, we examine the effectiveness of the GA-based progressive perturbation generation. Figure 7(a) shows the perturbations generated by DRRL in Walker2d, where each point denotes a two-dimensional adversarial force (D1,D2). We observe that during the training, the point distributions shift from the area around the zero point to the boundary. It indicates that along with the improvement of the policy, the generated tasks for the agent are increasingly difficult. This easyto-hard task sequence enables the agent to generalize increasingly well under the test conditions, as shown in Figure 7(b).

5 RelatedWork

Robust RL problem has been studied extensively from different perspectives. DR-based methods [Peng et al., 2018; Andrychowicz et al., 2020; Loquercio et al., 2019] introduce random perturbations by changing simulation parameters. Another line of works [Pinto et al., 2017; Zhang et al., 2020; Curi et al., 2021; Jiang et al., 2021] consider the adversarial multi-agent setting. Other works [Wang and Zou, 2021; Badrinath and Kalathil, 2021] inspired by Robust Markov Decision Process theory consider the worst-case perturbations from transition probabilities. More recently, there are some intrinsic motivation methods that train with generated curriculum [Sheng et al., 2022; Dennis et al., 2020], historical failures [Song and Schneider, 2022], and gradually changing bounds [Wu et al., 2021; Liang et al., 2022].

Different from existing works, we reformulate the robust RL as a max-expectation optimization, and solve it via dynamic multi-task learning under a progressive task sequence.

6 Conclusions

In this paper, we make the first attempt to formulate the robust RL as a max-expectation optimization problem to avoid conservative solutions. Then, we propose DRRL, which realizes dynamic multi-task learning and achieves better organization of robust RL training. In particular, an effective task generation and sequencing mechanism with increase-based task fitness is introduced to gradually generate appropriate tasks for iterative training. Experimental results including stability analysis, robustness analysis, stepwise analysis, and ablation analysis demonstrate the effectiveness of DRRL.

Acknowledgements

The work was supported by the Fundamental Research Funds for the Central Universities of China (2022YJS023), the National Natural Science Foundation of China (61972025, 61802389), the National Key Research and Development Program of China under Grant 2020YFB1005604 and Grant 2020YFB2103802.

Contribution Statement

Yike Li and Yunzhe Tian contributed equally to this work.