Constrained Intrinsic Motivation for Reinforcement Learning

1 Introduction

In the realm of Reinforcement Learning (RL), Intrinsic Motivation (IM) plays a vital role in the design of exploration strategies in both Reward-Free Pre-Training (RFPT) tasks and Exploration with Intrinsic Motivation (EIM) tasks [Barto, 2013]. It allows the RL agent to efficiently visit novel states by assigning higher intrinsic bonuses to unfamiliar states [Zhang et al., 2021]. Current IMmethods can be classified into three categories: knowledge-based, data-based, and competence-based IMmethods [Laskin et al., 2021].

∗Corresponding author.

Knowledge-based and data-based IM methods are employed in both RFPT and EIM tasks to encourage the agent to explore novel regions. Knowledge-based IM methods maximize the deviation of the agent's latest state visitation from the policy cover (i.e., the regions covered by all prior policies) [Zhang et al., 2021]. These methods commonly estimate the density of the policy cover via the pseudo-count of state visit frequency [Bellemare et al., 2016; Fu et al., 2017], prediction errors of a neural network [Pathak et al., 2017; Burda et al., 2019], or variances of outputs of an ensemble of neural networks [Pathak et al., 2019; Lee et al., 2021; Bai et al., 2021]. Data-based IMmethods, on the other hand, directly maximize the state coverage (i.e., the region visited by the latest policy) via maximizing the state entropy [Hazan et al., 2019; Mutti et al., 2021; Liu and Abbeel, 2021b; Seo et al., 2021]. However, knowledge-based and data-based IM methods are inefficient in RFPT tasks since they do not condition the latent skill variable, limiting the fine-tuning performance of the pre-trained policy in downstream tasks [Liu and Abbeel, 2021a]. Moreover, when utilized in EIM tasks, these IMmethods introduce non-negligible biases to the policy optimization, leading to suboptimal policies [Chen et al., 2022]. Specifically, intrinsic objectives may result in excessive exploration even when the task rewards are already accessible. This distraction induced by intrinsic objectives can deteriorate the performance of the RL agent and impede the wider application of these methods in EIM tasks.

Competence-based IM methods are designed for unsupervised skill discovery in RFPT tasks. They primarily maximize the mutual information between the state representation and the latent skill variable to learn a latentconditioned poilcy [Gregor et al., 2017; Sharma et al., 2020; Laskin et al., 2022]. The policy conditioned on the latent skill variable is required to change the state of the environment in a consistent and meaningful way, e.g., walking, flipping, pushing, to be finetuned efficiently in the downstream tasks. However, current competence-based IMmethods have shown poor performance in the Unsupervised Reinforcement Learning Benchmark (URLB) [Laskin et al., 2021], a benchmark of IM methods evaluated in RFPT tasks. Intuitively, directly maximizing the mutual information does not guarantee extensive state coverage or the discovery of dynamic skills and easily converges to simple and static skills due to the invariance of the mutual information to scaling and invertible transformation of the input variables [Park et al., 2022; Park et al., 2023]. Here, "dynamic" skills refer to skills that facilitate large state variations, e.g., running for locomotion tasks and moving for manipulation tasks. To address this limitation, Park et al. [2022] proposed Lipschitz-constrained Skill Discovery (LSD) to encourage dynamic skills. However, LSD suffers from severe sample inefficiency. The primary reason is that maximizing the intrinsic objective of LSD cannot guarantee maximum state entropy.

To overcome the limitations of existing knowledge-based, data-based, and competence-based IMmethods, in this paper, we propose Constrained Intrinsic Motivation (CIM) which is 1) a novel constrained intrinsic objective in RFPT tasks, i.e., a lower bound of the conditional state entropy subject to an alignment constraint on the state encoder network, to make the RL agent discover dynamic and diverse (distinguishable) skills more efficiently; 2) a Lagrangian-based adaptive coefficient for the intrinsic objective in EIM tasks to alleviate the performance decrease due to the bias introduced by the intrinsic rewards.

In summary, we make the following main contributions:

• We propose Constrained Intrinsic Motivation (CIM) to overcome the limitations of knowledge/data-based and competence-based IM methods by combining the best of both worlds. CIM outperforms state-of-the-art IM methods, improving performance and sample efficiency in multiple MuJoCo robotics environments.
• CIM for RFPT introduces a lower bound for the state entropy, conditioning the state entropy on the latent skill variable without compromising the power of maximum state entropy exploration. CIM for RFPT also introduces a novel alignment constraint on the state encoder network. Compared with LSD, our CIM reduces the number of required samples by 20x less (e.g., from 400M to 20M in the environment Ant). Besides skill diversity and state coverage, our CIM achieves the highest fine-tuning performance in the Walker domain of URLB.
• CIM for EIM derives an adaptive coefficient of the intrinsic objective leveraging the constrained policy optimization method. We empirically demonstrate that the adaptive coefficient can effectively diminish the bias introduced by intrinsic bonuses in various MuJoCo tasks and improve the average task rewards.

2 Preliminaries

2.1 Markov Decision Processes

The discounted Markov Decision Process (MDP) is defined as M = (S, A, P, R, γ, µ), where S and A stand for the state space and the action space separately, P : S ×A → ∆(S) is the transition function mapping the state s and the action a to the distribution P(s′ |s, a) in the space of probability distribution ∆(S) over S, R : S ×A× S → R is the reward function, γ ∈ [0, 1) is the discount factor, and µ ∈ ∆(S) is the initial state distribution. We focus on the episodic setting where the environment is reset once the agent reaches a final state sf , a terminated state within the goal subsets G or a truncated state sT . At the beginning of each episode, the agent samples a random initial state s0 ∼ µ; at each time t = 0, 1, 2, ..., T - 1, it takes an action at ∈ A computed by a stochastic policy π : S → ∆(A) or a deterministic one π : S → A according to the current state st and steps into the next state st+1 ∼ P(·|st, at) with an instant reward signal r = R(st, at, st+1) obtained.

2.2 Reward-Free Pre-Training and Exploration

RFPT and EIM are two types of intrinsically motivated RL tasks. To present the optimization objectives of RFPT and EIM, we first define the state distribution induced by the policy π as dπ(s) = (1 - γ) ∑∞t=0 γ tP(st = s|µ, π) ∈ K, where K is the collection of all induced distributions. The extrinsic objective (the expectation of the task reward) is then JE(dπ) = Es∼dπ [rE], where rE = RE(s, a, s′ ) is the extrinsic task reward function. The intrinsic objective JI : K → R is defined as a differentiable function of the induced state distribution dπ with L-Lipschitz gradients.

In RFPT tasks, the task reward re is not available, and the agent aims to maximize only the intrinsic objective

The agent can learn either a policy π(a|s) without conditioning the latent skill variable when maximizing a knowledge-based or data-based intrinsic objective, or a latent-conditioned policy π(a|s, z) when maximizing a competence-based intrinsic objective. Common evaluation metrics for RFPT tasks include state coverage, skill diversity, and fine-tuning performance in downstream tasks.

On the contrary, in EIM tasks, the goal of the agent is to complete a specific downstream task and maximize only the expected task rewards. The optimization objective of EIM is

where τk is the coefficient of the intrinsic objective. Since the agent does not need to discover diverse skills for a specific task, JI(dπ) in EIM tasks is commonly a knowledge-based or data-based intrinsic objective without conditioning the latent skill variable. The evaluation metric for EIM is only the expected task rewards.

2.3 Intrinsic MotivationMethods

We conduct a comprehensive comparison between our proposed CIM for RFPT and eighteen IM algorithms in regards of the intrinsic objective and the corresponding intrinsic reward function in Table 1, including Intrinic Curiosity Module (ICM) [Pathak et al., 2017], Random Network Distillation (RND) [Burda et al., 2019], Disagreement (Dis.) [Pathak et al., 2019], MAximizing the DEviation from explored regions (MADE) [Zhang et al., 2021], Adversarially Guided Actor-Critic (AGAC) [Flet-Berliac et al., 2021], Maximum Entropy exploration (MaxEnt) [Hazan et al., 2019], Active Pre-Training (APT) [Liu and Abbeel, 2021b], Random Encoders for Efficient Exploration (RE3) [Seo et al., 2021], Variational Intrinsic Control (VIC) [Gregor et al., 2017], Diversity Is All You Need (DIAYN) [Eysenbach et al., 2019], Variational Intrinsic Successor featuRes (VISR) [Hansen et al., 2020], Dynamics-Aware Discovery of Skills (DADS) [Sharma et al., 2020], Active Pretraining with Successor features (APS) [Liu and Abbeel, 2021a], Contrastive Intrinsic Control (CIC) [Laskin et al., 2022], Mixture Of SurpriseS (MOSS) [Zhao et al., 2022], Behavior Contrastive Learning (BeCL) [Yang et al., 2023], LSD, Controllability-Aware Skill Discovery (CSD) [Park et al., 2023]. Among these methods, ICM, RND, Dis., MADE, and AGAC belong to knowledge-based IM methods since their intrinsic objectives depend on the agent's all historical experiences. MaxEnt, APT, and RE3 fall under data-based IM methods, directly maximizing the state entropy. Meanwhile, VIC, DIAYN, VISR, DADS, APS, CIC, MOSS, BeCL, LSD, and CSD are classified as competence-based methods, all of which condition the latent skill variable.

3 Constrained Intrinsic Motivation

In this section, we first present CIM for RFPT, a novel competence-based IM method that can learn dynamic and diverse skills efficiently. Specifically, we propose a constrained intrinsic objective J CIM I for RFPT tasks, maximizing the conditional state entropy instead of the state entropy under a novel alignment constraint for the state representation. We then derive the corresponding intrinsic reward rCIM I based on the Frank-Wolfe algorithm. Secondly, we propose CIM for EIM to adaptively adjust the coefficient of the intrinsic objective in Equation (2) based on constrained policy optimization to mitigate the bias introduced by the intrinsic objective. We then derive the adaptive coefficient τ CIM k based on the Lagrangian duality theory.

3.1 Constrained Intrinsic Motivation for RFPT

In this section, we develop CIM for RFPT, a novel constrained intrinsic objective for unsupervised RL. To better clarify the motivation for the design of the constrained intrinsic objective, we first review current coverage- and mutual information-based methods and analyze their limitations.

Problems of Previous Intrinsic Motivation Methods

Though knowledge-based and data-based IM methods may perform well in terms of state coverage in certain RFPT tasks, these methods lack awareness of latent skill variables and suffer from poor fine-tuning efficiency. To improve the fine-tuning performance in RFPT tasks, learning a latent-conditioned policy is necessary. However, existing competence-based IM methods perform poorly regarding skill diversity, state coverage, and sample inefficiency. We conjecture that there are two main issues:

Intrinsic objective. Maximizing only the mutual information is not suitable for dynamic and diverse skill discovery in RFPT tasks. Recall the two types of decomposition for the mutual information, that is, I(s; z) = H(z) - H(z|s) = H(s) - H(s|z). Note that minimizing H(z|s) can be achieved with slight differences in states, and minimizing H(s|z) clearly impedes the maximization of H(s). Thus, neither can encourage the agent to cover large state space. Moreover, using H(s) directly as the intrinsic objective (e.g., CIC and MOSS) also leads to low state coverage, as shown in Figure 1. One of the key drawbacks of maximizing H(s) is that it is challenging to estimate state density in the highdimensional space.

Alignment constraint. Current alignment constraints for the state encoder network are not efficient enough. For instance, LSD can learn dynamic skills with Lipschitz constraint on the state encoder network but suffers from heavy sample inefficiency. On the other hand, CIC applies noise contrastive estimation to formulate the alignment constraint on the state encoder network but fails to learn sufficiently dynamic and diverse skills.

Design of Constrained Intrinsic Objective

To address the first issue, we propose choosing the conditional state entropy H(ϕ(s)|z) as the intrinsic objective. This is a key difference between our method and previous IM methods. Intuitively, bymaximizingH(ϕ(s)|z), distances between adjacent states within the trajectories sampled by one skill are enlarged, which indicates a more dynamic skill.

For the second issue, we propose maximizing a novel lower bound of the mutual information between the state representation and the latent skill variable to make the trajectories sampled by different skills distinguishable, that is,

where La(ϕ(s), z) is the alignment loss as follows:

N is the total number of samples for estimating the mutual information, τ = (s, s′ ) is the slice of a trajectory, and Sis a set of negative samples that contains trajectories sampled via skills other than zi. We derive this lower bound based on Contrastive Predictive Coding [Oord et al., 2018] by regarding the latent skill z as the context and ϕdiff (τ) as the predictive coding. Based on Equation (3), the alignment constraint on the state encoder network ϕ(s) is

where C is a constant. Theoretically, as indicated in Table 1, C should represent the minimum of La(ϕ(s), z). In practice, we do not need to know the exact value of C. Instead, at each policy iteration step, we take several stochastic gradient descent steps on the alignment loss La(ϕ(s), z) to maximize the mutual information between the state representation ϕ(s) and the latent skill z.

The complete constrained intrinsic objective of CIM for RFPT is thusmax

where H(ϕ(s)|z) is the conditional state entropy estimated in the state projection space, which depends on both the latentconditioned policy network π(·|s, z) and the state encoder network ϕ(s), dπ(ϕ(s)) is the distribution of the latent state ϕ(s) induced by the latent-conditioned policy π(·|s, z).

Interpreting La(ϕ(s), z) as an alignment loss provides us a novel insight to unify former competence-based IM methods. We can derive the alignment loss li of all previous competence-based IMmethods listed in Table 1, e.g.,

where d(s, s′ ) in lLSD i is the state distance function, Sc in lvMF i is the cosine similarity between two vectors, ϕz in lCIC i is a projection network for the latent skill vector, and ϕ(s+i ) in lBeCL i is a state representation from a certain skill as the positive sample while all others are negative samples.

Estimation of Conditional State Entropy

We now introduce how to estimate the conditional state entropy H(ϕ(s)|z) involved in Equation (6). Recall the definition of the conditional state entropy

To estimate the outer expectation, we randomly sample the latent skill variables z from a prior distribution pz(z). For discrete skills, pz(z) can be a categorical distribution Cat(K, p) that is parameterized by p over a size-K the sample space, where pi denotes the probability of the i-th skill. For continuous skills, we can select p(z) as a uniform distribution U nz (a, b) over the interval [a, b], where nz is the dimension of the skill.

To estimate the inner expectation, we roll out trajectories using the latent-conditioned policy π(·|s, z) with z fixed. During the sampling phase, z is randomly sampled at the beginning of each episode and remains fixed throughout the entire trajectory. We store the state-skill pair (s,z) in the replay buffer. During the training phase, for each pair (s,z), we concatenate them as [s, z] to be the input of the latent-conditioned policy π(·|s, z).

To estimate the state density dπ, instead of training a parameterized generative model, we leverage a more practical non-parametric ξ-nearest neighbor (ξ-NN) estimator

where λ is the Lebesgue measure on Rd , Bξ is the smallest ball centered on si containing its ξ-th nearest neighbour sξi .

Lower bound of conditional state entropy. Each skill can be stochastic if we directly maximize the conditional state entropy H(s|z). To address this, we propose maximizing the lower bound of H(s|z) to encourage the skill z to produce large state variations along the direction of z in the latent space instead of being fully stochastic. To derive the lower bound of H(s|z), we first define a projection function gz(ϕ(s)) = ϕ(s)T z for a fixed skill z. It is easy to verify that H(ϕ(s)|z) ≥ H(gz(ϕ(s))|z)) with equality iff Sc(ϕ(s), z) = 1, that is, H(gz(ϕ(s))|z)) is a lower bound of H(ϕ(s)|z). We thus can maximize H(gz(ϕ(s))|z)) to maximize H(ϕ(s)|z) and estimate the distribution of the onedimensional random variable gz(ϕ(s)) for each z.

Intrinsic reward. Based on the above design, we can derive the intrinsic reward of CIM for RFPT as rCIM I (s) = log ∥gz(ϕ(s)) - gz(ϕ(s))ξ ∥. Here, gz(ϕ(s))ξ means the ξth nearest neighbor of gz(ϕ(s)). We adopted an averagedistance version similar to APT to make training more stable:

Intuitively, rCIM I (s) measures how sparse the state s is in the projection subspace spanned by its corresponding latent skill z. This reward function can be justified based on the procedure of the Frank-Wolfe algorithm. Specifically, since LRFPT k is concave in dπ, maximizing LRFPT k involves solving dπk+1 ∈ argmax⟨∇dπ L(dπk ), dπk - dπ⟩ iteratively [Hazan et al., 2019]. This iterative step is equivalent to policy optimization using a reward function proportional to ∇dπ L(dπk ).

3.2 Constrained Intrinsic Motivation for EIM

In this section, we present our CIM for EIM, an adaptive coefficient for the intrinsic objective in Equation (2). Currently, IM methods for EIM tasks commonly use a constant coefficient or an exponentially decaying coefficient, which requires costly hyperparameter tunning. To avoid this, we propose reformulating Equation (2) by regarding the extrinsic objective as a constraint for the intrinsic objective, i.e.,

where Rk represents the expected reward at the k-th iteration of policy optimization. We approximate Rk via ˆRk = maxj∈{1,2,...,k-1} JE(dπj ). We then leverage the Lagrangian method to solve Equation (11). The corresponding Lagrangian dual problem is minλ≥0 maxdπ JI(dπ) + λk(JE(dπ) - ˆRk). The Lagrangian multiplier λ is updated by Stochastic Gradient Descent (SGD), that is, λk = λk-1 - η(JE(dπk ) - ˆRk-1) where η is the updating step size of λk. Observing that Lk(dπ, λk) ∝ JE(dπ)+λ-1 k JI(dπ), the adaptive coefficient τ CIM k is then derived as

where the outer minimization is to ensure numerical stability. As the Lagrangian multiplier λk grows, the penalty for the violation of τ CIM k gradually tends to zero; that is, the bias introduced by the intrinsic objective JI is adaptively reduced.

4 Experiments

4.1 Experimental Setup

Experimental setup for RFPT. We evaluate our intrinsic bonus rCIM I for RFPT tasks on four Gymnasium environments, including two locomotion environments (Ant and Humanoid) and two manipulation environments (FetchPush and FetchSlide). We compare CIM for RFPT with fifteen IM methods in Table 1, including 1) four knowledge-based IM methods: ICM, RND, Dis., MADE, and AGAC; 2) one databased IM method: APT; 3) and nine competence-based methods: DIAYN, VISR, DADS, APS, CIC, MOSS, BeCL, LSD, and CSD.

Experimental setup for EIM. We evaluate our adaptive coefficient τ CIM k for EIM in two navigation tasks (PointMaze UMaze and AntMaze UMaze) in D4RL [Fu et al., 2020], and four sparse-reward tasks (SparseHalfCheetah, SparseAnt, SparseHumanoidStandup, and SparseGridWorld). τ CIM k is orthogonal with any intrinsic bonuses rI. Unless otherwise mentioned, we adopt the state-of-the-art data-based intrinsic bonus rAPT I = log(1+1/k ∑kj=1 ∥ϕ(s)- ϕ(s)j ∥). The total instant reward is then r = rE + τ CIM k rAPT I . We compare CIM for EIM with three baseline coefficient schemes, i.e., the constant coefficient τ C k ≡ 1, the linearly decaying coefficient τ L k = (1 - k/T), and the exponentially decaying coefficient τ E k = 0.001k .

4.2 Results in RFPT Tasks

Visualization of skills. As previous works like LSD do, we train CIM for RFPT to learn diverse locomotion continuous skills in the Ant and Humanoid environment and diverse manipulation skills in FetchPush and FetchSlide. The learned skills are visualized as trajectories of the agent on the x - y plane in Figure 1 and Figure 2. Our CIM for RFPT outperforms all 15 baselines in terms of skill diversity and state coverage. The skills learned via CIM are interpretable because of our alignment loss; the direction of the trajectory on the x - y plane changes consistently with the change in the direction of the skill. Specifically, CIM excels at learning dynamic skills that move far from the initial location in almost all possible directions, while most baseline methods fail to discover such diverse and dynamic primitives. Their trajectories are non-directional or less dynamic than CIM, especially in two locomotion tasks. Competence-based approaches like DIAYN, VISR, and DADS directly maximize the mutual information objectives but learn to take static postures instead of dynamic skills; such a phenomenon is also reported in LSD and CIC. Although APS and CIC can learn dynamic skills by directly maximizing the state entropy, CIM discovers skills that reach farther and are more interpretable via maximizing the lower bound of the state entropy. As for the two variants of CIC, MOSS and BeCL, they perform even worse than CIC in all tasks, reflecting their limitation in skill discovery. Lastly, LSD and CSD cannot learn dynamic skills within limited environment steps in Ant and Humanoid due to their low sample efficiency. Though they perform better in manipulation tasks than locomotion tasks, their learned skills are rambling compared with our CIM.

State coverage. To make a quantitative comparison between various IM methods, we measure their state coverage. The state coverage in Ant and Humanoid is determined by calculating the number of 2.5 × 2.5 m2 bins occupied on the x-y plane, based on 1000 randomly sampled trajectories. This was then averaged over five runs. For FetchPush and FetchSlide, we use smaller bins. As shown in Table 2, CIM significantly outperforms all the baseline methods in two torqueas-input locomotion tasks and is comparable in two positionas-input manipulation tasks. Although the state coverage of CIM is slightly lower than APT and CIC in FetchPush and FetchSlide, the skills learned via CIM are more interpretable, as shown in Figure 2.

Fine-tuning efficiency in URLB. We also evaluate CIM for RFPT in URLB, a benchmark environment for RFPT in terms of fine-tuning efficiency. The results are presented in Table 3. The score (the last line of the table) is standardized by the performance of the expert DDPG, the same as in URLB and CIC. CIM performs better in Run and Walk tasks and achieves the highest average score. The dynamic skills learned through CIM for RFPT can be adapted quickly to diverse fine-tuning tasks, including flipping and standing. Our experiments also show that the skill dimension nz = 3 is better for CIM to discover flipping skills than nz = 2. The fixed skill selection mechanism for CIM is the same as CIC.

Ablation study. According to the results in Table 4, loss functions that follow the NCE style, such as lCIC i , lBeCL i , and lCIM i , perform better than other styles like MSE and vMF. Besides, lCIM i is the most effective. As shown in Figure 3a, our CIM can also be utilized to discover discrete diverse and dynamic skills, though it is mainly designed for continuous skills. Moreover, our CIM for RFPT is also robust to the number of skill dimensions, as shown in Table 5. Based on the ablation study, we can conclude that the two components of CIM, i.e., minimizing NCE-style alignment loss and maximizing conditional state entropy, are equally critical. Specifically, the results in Table 4 show that replacing the alignment loss of CIM with a trivial MSE loss reduces the state coverage in Ant from 1042 to 64. Moreover, Table 2 reveals that the state coverage achieved by CIM can reach 1135 in the challenging 378-dimensional Humanoid, while that achieved by CIC and BeCL, which use similar NCE-style alignment losses, is lower than 100.

4.3 Results in EIMTasks

In PointMaze, we directly train a policy to control the Point without learning low skills since the environment dynamics are simple. In AntMaze, we train a meta-controller on top of the latent-conditioned policy pre-trained via our CIM for RFPT method. The meta-controller observes the target goal concatenated to the state observation [s; sg] and outputs the skill latent variable z at each timestep. We visualize the trajectories of the Ant in the x - y plane as shown in Figure 3b, where the skills in a single trajectory gradually change to make the Ant turn a corner. Figure 3c shows that the Lagrangian-based adaptive coefficient τ CIM k outperforms three baseline coefficients, especially in PointMaze. Specifically, we can observe a small peak in the early stage of the training in PointMaze, which means the agent can reach the randomly generated target point with a small probability at the beginning. However, as the training processes, the agent is distracted by the intrinsic bonuses when using a trivial coefficient τ C k or τ L k . Moreover, other latent-conditioned policies are of poor quality, and we fail to train a mete-controller on top of those policies. We also conduct experiments to demonstrate the performance of CIM for EIM across four sparsereward locomotion tasks. The results in Table 6 indicate that τ CIM k can effectively reduce the bias introduced by intrinsic rewards, thereby enhancing test-time average episode rewards in EIM tasks.

5 Conclusion

In this paper, we proposed Constrained Intrinsic Motivation (CIM) for RFPT and EIM tasks, respectively. For RFPT tasks, we designed a novel constrained intrinsic objective to discover dynamic and diverse skills. For EIM tasks, we designed an adaptive coefficient τ CIM k for the intrinsic objective based on constrained policy optimization. Our experiments demonstrated that CIM for RFPT outperformed all fifteen baselines across various MuJoCo environments regarding diversity, state coverage, sample efficiency, and fine-tuning performance. The latent-conditioned policy learned via CIM for RFPT was successfully applied to solve complex EIM tasks via training a meta-controller on top of it. We also empirically verified the effectiveness of our adaptive coefficient τ CIM k in multiple EIM tasks.

Acknowledgments

We thank the anonymous reviewers for their valuable feedback. This work was partially supported by HK RGC under Grants (CityU 11218322, R6021-20F, R1012-21, RFS21221S04, C2004-21G, C1029-22G, and N CityU139/21) and the National Natural Science Foundation of China (U21B2018, 62161160337, 61822309, U20B2049, 61773310, U1736205, 61802166, 62276067).