---
title: "G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction"
authors:
  - "Zhanwei Zhang"
  - "Zishuo Hua"
  - "Minghao Chen"
  - "Wei Lu"
  - "Binbin Lin"
  - "Deng Cai"
  - "Wenxiao Wang"
doi: "10.24963/ijcai.2024/285"
url: "https://ijcai.org/proceedings/2024/285"
pdf: "https://ijcai.org/proceedings/2024/0285.pdf"
venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence"
year: 2024
pages: "2579-2587"
track: "Main Track"
keywords:
  - "Machine Learning: ML: Time series and data streams"
  - "Data Mining: DM: Mining spatial and/or temporal data"
  - "Computer Vision: CV: Motion and tracking"
---

# G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction

Zhanwei Zhang, Zishuo Hua, Minghao Chen, Wei Lu, Binbin Lin, Deng Cai, Wenxiao Wang

*Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 2579-2587. DOI: [10.24963/ijcai.2024/285](https://doi.org/10.24963/ijcai.2024/285)

## Abstract

Predicting future trajectories of traffic agents accurately holds substantial importance in various applications such as autonomous driving. Previous methods commonly infer all future steps of an agent either recursively or simultaneously. However, the recursive strategy suffers from the accumulated error, while the simultaneous strategy overlooks the constraints among future steps, resulting in kinematically infeasible predictions. To address these issues, in this paper, we propose G2LTraj, a plug-and-play global-to-local generation approach for trajectory prediction. Specifically, we generate a series of global key steps that uniformly cover the entire future time range. Subsequently, the local intermediate steps between the adjacent key steps are recursively filled in. In this way, we prevent the accumulated error from propagating beyond the adjacent key steps. Moreover, to boost the kinematical feasibility, we not only introduce the spatial constraints among key steps but also strengthen the temporal constraints among the intermediate steps. Finally, to ensure the optimal granularity of key steps, we design a selectable granularity strategy that caters to each predicted trajectory.  Our G2LTraj significantly improves the performance of seven existing trajectory predictors across the ETH, UCY and nuScenes datasets. Experimental results demonstrate its effectiveness. Code will be available at https://github.com/Zhanwei-Z/G2LTraj.

## Full text

## 1 Introduction

Accurate trajectory prediction is crucial in numerous applications like self-driving, enabling autonomous vehicles to make safe and reliable decisions. This task involves predicting the future trajectories of moving agents (e.g., vehicles and pedestrians) by learning their interactions within the scenarios. Recently, deep learning methods have come to dominate the field of trajectory prediction. These methods typically predict all future steps of an agent either simultaneously [Shi et al., 2021; Shi et al., 2022b; Jia et al., 2023b; Girgis et al., 2022; Li et al., 2021; Bae et al., 2023] or recursively [Yuan et al., 2021; Chen et al., 2021; Gu et al., 2022; Lee et al., 2022; Zhao andWildes, 2021]. Specifically, the simultaneous methods assume that the future steps of an agent are independent of each other and commonly generate all future steps at once. One branch of these methods [Aydemir et al., 2023; Bae and Jeon, 2023; Gu et al., 2021; Mangalam et al., 2021] leverages trajectory anchors or endpoints to generate the intermediate steps concurrently. While these methods have demonstrated promising performance enhancements, the constraints among future steps are ignored, potentially leading to kinematically infeasible predictions, as shown in Fig. 1(a).

†Corresponding author

Figure 1: (a) State-of-the-art simultaneous approaches like GraphTERN [Bae and Jeon, 2023] do not model the constraints among future steps well, resulting in kinematically infeasible predictions (i.e., the fluctuation in the predicted trajectories), as evaluated on the UCY [Lerner et al., 2007] dataset. (b) Our G2LTraj incorporates spatial-temporal constraints to generate consistent trajectories. (c) Overview of our G2LTraj for trajectory prediction in a global-tolocal generation process. For instance, initially, we simultaneously generate the global key steps t1, t9 and so on. Subsequently, t5 is generated by utilizing t1 and t9. This iterative process continues until all future steps are generated.

![Image](https://ijcai.org/proceedings/2024/fulltext/285/image_000000_d18211bf01b2fe71c1b6b4e29ea7a22ffef44a4bae688ca050e8afb45365d7cb.png)

On the other hand, to capture the temporal constraints among future steps, the recursive strategy utilizes previously predicted time steps to forecast the subsequent ones. However, this strategy suffers from the accumulated error throughout the recursive process [Bae and Jeon, 2023], with the error propagating from the initial step to the final step. To balance the accumulated error and the constraints among future steps, [Jia et al., 2023a] simultaneously generates all future steps and utilizes a recursive network for trajectory refinement. Nonetheless, during refinement, the accumulated error still propagates from the initial step to the final step.

To tackle the above issues, in this paper, we propose G2LTraj, a plug-and-play global-to-local generation approach for trajectory prediction. Specifically, we generate a series of global key steps that uniformly span the entire future time range, establishing the trajectory outline. We treat any two adjacent key steps as the head and the tail steps of a local section. By partitioning the entire trajectory into multiple sections, the accumulated error would not propagate beyond each section. Subsequently, the intermediate steps within each section are recursively generated to complete the whole trajectory. We illustrate the overview of our G2LTraj in Fig. 1(c) intuitively. During the global-to-local generation process, we identify three specific challenges requiring attention. First, during the global key step generation process, since all key steps are generated at once, there is a lack of constraints among them. To this end, we introduce the spatial constraints among key steps by regulating the positional difference. This enables G2LTraj to generate consistent trajectories that adhere to kinematics, as shown in Fig. 1(b). Second, during the local recursive generation process, if merely relying on the head and tail steps to generate the intermediate step, the model's access to past kinematic information is limited, resulting in fragile temporal constraints with past trajectories. This fragility possibly leads to plausible but kinematically infeasible intermediate steps. To maximize the temporal constraints, we integrate the agent features along with the position information of the head and tail steps when generating the intermediate step. By doing so, G2LTraj can enhance its understanding of local motion patterns. Third, trajectories from different agents can exhibit varying kinematics characteristics like distinct velocities. Thus, using a fixed granularity (i.e., the time interval between two adjacent key steps) might not be suitable for each trajectory, which aligns with previous findings [Bae and Jeon, 2023]. Distinguished from manually adjusting the granularity for all trajectories [Bae and Jeon, 2023], we design a selectable granularity strategy tailored for each trajectory. Specifically, we generate key steps at a fine granularity and then downsample these steps to form multiple groups of coarse-grained key steps. Next, by learning the confidence scores of these groups for each trajectory, we select the optimal group for them. Our contributions can be summarized as follows:

- we propose a global-to-local generation approach for trajectory prediction that mitigates the accumulated error and introduces the constraints among future steps.
- To enhance the kinematical feasibility, we propose the spatial constraints among key steps and reinforce the
- temporal constraints among the intermediate steps.
- To ensure the optimal granularity of key steps for each trajectory, we devise a selectable granularity strategy.
- Our G2LTraj demonstrates notable performance gains over seven existing trajectory predictors when evaluated on ETH [Pellegrini et al., 2009], UCY [Lerner et al., 2007] and nuScenes [Caesar et al., 2020]. Extensive experiment results validate its effectiveness.

## 2 RelatedWork

Accurate trajectory prediction plays a vital role in various applications such as self-driving. Recent state-of-the-art methods primarily rely on neural networks to forecast all future steps of an agent, either simultaneously or recursively.

Simultaneous Trajectory Predictors assume that the future steps of an agent are independent of each other and typically generate future steps concurrently [Jia et al., 2023b; Shi et al., 2021; Shi et al., 2022b; Girgis et al., 2022]. [Shi et al., 2022a] utilizes tree paths based on the observed trajectories to predict future trajectories in parallel. [Shi et al., 2021] and [Li et al., 2021] achieve simultaneous trajectory prediction by capturing both spatial and temporal clues. [Zhou et al., 2023] endeavors to achieve faster inference while maintaining effectiveness. Another branch [Bae and Jeon, 2023; Aydemir et al., 2023; Gu et al., 2021] initially learns the potential endpoints or anchor points of local trajectories and subsequently generates the intermediate steps simultaneously. For example, [Aydemir et al., 2023] employs the predicted endpoints to jointly predict the trajectories of all agents in the scene. [Ye et al., 2022] considers both spatial and temporal perturbations when predicting the endpoint of future trajectories. However, these methods commonly neglect the constraints among future steps, possibly resulting in predictions that are kinematically infeasible [Jia et al., 2023a].

Recursive Trajectory Predictors utilizes previously predicted time steps to generate the subsequent ones [Gu et al., 2022; Lee et al., 2022; Yuan et al., 2021; Chen et al., 2021; Zhao and Wildes, 2021; Salzmann et al., 2020; Xu et al., 2020]. [Marchetti et al., 2022] and [Zhao and Wildes, 2021] employ supplementary storage space to generate future trajectories sequentially. [Navarro and Oh, 2022] proposes guiding long-term time-series trajectory prediction in complex environments. [Gu et al., 2022] progressively discards area indeterminacy until reaching the desired trajectory. [Lee et al., 2022] leverages RNN-based Conditional Variational AutoEncoder (CVAE) to obtain the full trajectories. Moreover, [Jia et al., 2023a] exploits a recursive network for trajectory refinement while simultaneously generating all future steps. Despite capturing temporal constraints, these methods encounter the accumulated error throughout the recursive process, as the error propagates from the initial step to the final step.

To this end, we propose a global-to-local generation approach that mitigates the accumulated error and incorporates constraints among future steps.

Figure 2: The overview framework of our G2LTraj. We introduce a global-to-local generation approach for trajectory prediction that attenuates the accumulated error and proposes constraints among future steps. In this illustration, we exemplify the global-to-local generation process at a granularity of 8. we simultaneously generate the globakey steps t1, t9 and so on. Subsequently, t5 is generated by utilizing t1 and t9. This local recursion process continues until all future steps are generated.

![Image](https://ijcai.org/proceedings/2024/fulltext/285/image_000001_cf5b89a6f920cd401081f5f7a7d3b5071e3ba396f22fc0579e0b6b1acd77c80a.png)

## 3 Methodology

## 3.1 ProblemFormulation and Overview

In the field of trajectory prediction, we are provided with the observed trajectory coordinates of an interested agent a, denoted as P = {(xi, yi)}0i=-Tp+1 . Here, Tp represents the length of past steps. Naturally, in the given scenario, agent a interacts with various elements in the environment, including other agents and the map. For brevity, we denote these elements as S. Our work can generally build upon any encoderdecoder trajectory predictor, denoted as G. Following the encoder-decoder structure of G, the agent features of a can typically be derived byA = G (P, S) . (1)

Through A, the objective is to predict a's future trajectory coordinates F = {Fi}Tf i=1 , where Tf is the length of future steps. Fi = (xi, yi) denotes the coordinate of step ti.

The overview framework of our G2LTraj is depicted in Fig. 2. Specifically, we propose a global-to-local generation approach for trajectory prediction. In the example of Fig. 2, initially, we simultaneously generate the global key steps t1, t9 and so on with a granularity of 8. Subsequently, the intermediate step t5 is generated by utilizing t1 and t9. Next, t3 (t7) is generated by utilizing t1 and t5 (t5 and t9). This local recursion continues until all future steps are generated.

## 3.2 Global Key Step Generation

In this section, we generate key future time steps of agent a at once. Specifically, we set the granularity of key steps as L (i.e., the time interval between two adjacent key steps.) Subsequently, we determine the key steps necessary to cover the length of future steps uniformly. Let GL denote the group of key steps under the granularity L, which can be derived by

<!-- formula-not-decoded -->

Here, we will generate a total of N + 1 key steps. Any two adjacent key steps are considered as the head and the tail steps of a local section. Consequently, we partition the entire trajectory into n sections. In this manner, we ensure that the accumulated error among future steps would not propagate beyond each section. Next, we predict the position coordinates ˆ Z ∈ R(N+1)×2 of these N + 1 key steps by

<!-- formula-not-decoded -->

where ψL is a two-layer Multilayer-Perceptron (MLP). Let ˆ Z1+nL ∈ R2 denotes the corresponding predicted coordinate of step t1+nL, where n ∈ {0, 1, ..., N}.

Since the key steps are generated simultaneously, these steps currently lack interrelated constraints, potentially causing kinematically infeasible predictions, as shown in Fig. 1(a). To address this issue, we introduce spatial constraints among key steps by regulating their positional difference. The detailed spatial constraint loss LLs under the granularity L can be calculated by

<!-- formula-not-decoded -->

where Lr denotes the regression loss, such as Mean Square Error (MSE) loss [Gu et al., 2021] or the Negative LogLikelihood (NLL) loss [Varadarajan et al., 2022]. F1+nL denotes the ground truth coordinate of step t1+nL. Our G2LTraj adopts the MSE loss according to the empirical performance. Notably, if Tf &lt; 1+NL, the value F1+NL is not available. In such cases, we set the upper limit of n in Eq. (4) to N -2. As shown in Eq. (4), LLs endeavors to adjust inconsistent steps to a reasonable relative distance from adjacent steps. In this way, we ensure generate consistent trajectories that adhere to kinematic feasibility principles.

Figure 3: Merely relying on t5 and t9 limits the model's access to historical kinematic information, possibly generating plausible but kinematically infeasible t7, as shown in the red oval. By incorporating historical kinematics clues, the prediction range can be reduced to the kinematically feasible range, as shown in the black oval.

![Image](https://ijcai.org/proceedings/2024/fulltext/285/image_000002_fa2dc53ee0000059fbd1be4084741faa4534bb144a16bc6e8bf595840b827f0f.png)

## Algorithm 1 Global-to-local Generation Process

Input: The granularity of key steps L, the predicted key coordinates { ˆ Z1+iL}Ni=0 , two-layer MLPs {ϕl}l∈{L,L/2,...,2}, the position embedding {p∗}1+NL ∗=0 , and agent features A.

Output: A predicted future trajectory ˆ FL ∈ R(1+NL)×2 for agent a.

- 1: Let l ←L.
- 2: while l &gt; 1 do
- 3: Calculate { ˆ Z1+(i+ 12 )l} NL l -1 i=0 by Eq. (6) in parallel with ϕl, ˆ Z1+il, p1+il, ˆ Z1+(i+1)l, p1+(i+1)l and A as input;
- 4: l ← l/2.
- 5: ▷ l denotes the current interval of existing adjacent predicted steps. Notably, we set L ∈ {2, 4, 8, ...} to ensure l/2 is an integer value continuously. When l = 1, it indicates that the entire trajectory coordinates of 1 + NL steps have already been generated.
- 6: end while

## 3.3 Local Recursive Generation

After generating the global key steps of agent a, the next phase is to fill in the intermediate steps within each local section. If merely relying on the head and tail steps to generate the intermediate step, the temporal constraints among them are fragile. To illustrate this issue intuitively, we enumerate three cases, as shown in Fig. 3. In these cases, the ground truths display similar t5 and t9 positions in the local coordinate system while exhibiting distinct positions for the intermediate step t7. Solely depending on t5 and t9 limits the model's access to historical kinematic information, resulting in delicate temporal constraints with past trajectories. Consequently, this manner possibly generates plausible but kinematically infeasible t7 which is inconsistent with past trajectories, as shown in the red oval in Fig. 3.

To alleviate this issue, we propose incorporating agent features in addition to the positional information of the head and tail steps. Specifically, agent features A encompass historical kinematics clues. By incorporating A, we can improve the temporal constraints with past trajectories. The improved constraints aim to reduce the plausible prediction range to the kinematically feasible range, as shown in the black oval in the Fig. 3. Besides, we follow [Zhou et al., 2021; Zeng et al., 2023] to employ the position embedding of each step in the time sequence, as it can provide the local temporal information. Next, given the predicted coordinates of the head step ti and the tail step tj, the coordinate of the intermediate step t(i+j)/2 can be derived by a projection layer

<!-- formula-not-decoded -->

where pi ∈ RD and pj ∈ RD denote the position embeddings of steps ti and tj, respectively. Empirically, we project these elements in a simple yet effective manner, which can be calculated by

<!-- formula-not-decoded -->

where ϕl is a two-layer MLP, l is the interval between ti and tj. Due to the changing interval, we utilize multiple ϕl functions that vary with the subscript l, instead of relying on a single MLP. || denotes the concatenation operation. Since the values of ˆ Zi and ˆ Zj may differ significantly with a large l, we project their dimension to align with that of the position embedding using two different weight matrices (i.e., Wh ∈ R2×D and Wt ∈ R2×D ). After finishing the globalto-local recursive generation process, we obtain a predicted trajectory of agent a, denoted as ˆ FL ∈ R(1+NL)×2 , as shown in Algorithm 1.

## 3.4 Selectable Granularity Strategy

Future Trajectories of different agents can exhibit diverse kinematics characteristics. Thus, employing a fixed granularity may not be appropriate for each trajectory. For instance, unusual behaviors like abrupt direction changes involve rapid shifts in velocity and trajectory. In such cases, a finer granularity might be more adequate as it can better capture sudden motion changes. Conversely, for usual behaviors such as moving straight uniformly, a coarser granularity might suffice, as it can enhance the temporal constraints from more distant steps. To this end, we propose a selectable granularity strategy that aims to be as suitable as possible for each trajectory. In the following, for the sake of brevity, we focus on discussing the cases where Tf = 2M + 1, M is an integer.

Table 1: Performance comparison between six trajectory prediction predictors and our G2LTraj on ETH and UCY test set. We obtain the performance values of SGCN, STGCNN, PECNet and AgentFormer from the original paper [Bae et al., 2023], while the performance values of GraphTERN, and DMRGCN are derived based on their official codes. For a fair comparison with these state-of-the-art methods, all predictors are constructed based on the Eigentrajectory space [Bae et al., 2023]. Best performances are in bold. Gains: performance improvements, Unit for ADE and FDE: meter.

|       | SGCN - ADE↓   | SGCN - FDE↓   | SGCN(G2LTraj) - ADE↓ (Gains)   | SGCN(G2LTraj) - FDE↓ (Gains)   | STGCNN - ADE↓   | STGCNN - FDE↓   | STGCNN(G2LTraj) - ADE↓ (Gains)   | STGCNN(G2LTraj) - FDE↓ (Gains)   |
|-------|---------------|---------------|--------------------------------|--------------------------------|-----------------|-----------------|----------------------------------|----------------------------------|
| ETH   | 0.360         | 0.565         | 0.340(+5.56%)                  | 0.548(+3.01%)                  | 0.365           | 0.582           | 0.362(+0.82%)                    | 0.576(+1.03%)                    |
| HOTEL | 0.131         | 0.210         | 0.123(+6.11%)                  | 0.197(+6.19%)                  | 0.147           | 0.220           | 0.133(+9.52%)                    | 0.211(+4.09%)                    |
| UNIV  | 0.244         | 0.430         | 0.242(+0.82%)                  | 0.431(-0.70%)                  | 0.246           | 0.427           | 0.244(+0.81%)                    | 0.430(-0.70%)                    |
| ZARA1 | 0.200         | 0.347         | 0.193(+3.50%)                  | 0.349(-0.58%)                  | 0.217           | 0.393           | 0.214(+1.38%)                    | 0.380(+3.31%)                    |
| ZARA2 | 0.153         | 0.261         | 0.147(+3.92%)                  | 0.255(+2.30%)                  | 0.168           | 0.290           | 0.160(+4.76%)                    | 0.277(+4.48%)                    |
|       | PECNet        | PECNet        | PECNet(G2LTraj)                | PECNet(G2LTraj)                | AgentFormer     | AgentFormer     | AgentFormer(G2LTraj)             | AgentFormer(G2LTraj)             |
|       | ADE↓          | FDE↓          | ADE↓ (Gains)                   | FDE↓ (Gains)                   | ADE↓            | FDE↓            | ADE↓ (Gains)                     | FDE↓ (Gains)                     |
| ETH   | 0.365         | 0.572         | 0.348(+4.66%)                  | 0.536(+6.29%)                  | 0.362           | 0.568           | 0.354(+2.21%)                    | 0.592(-4.23%)                    |
| HOTEL | 0.132         | 0.211         | 0.116(+12.12%)                 | 0.183(+13.27%)                 | 0.147           | 0.222           | 0.136(+7.48%)                    | 0.228(-2.70%)                    |
| UNIV  | 0.244         | 0.432         | 0.241(+1.23%)                  | 0.424(+1.85%)                  | 0.244           | 0.430           | 0.234(+4.10%)                    | 0.416(+3.26%)                    |
| ZARA1 | 0.195         | 0.348         | 0.185(+5.13%)                  | 0.329(+5.46%)                  | 0.216           | 0.397           | 0.211(+2.31%)                    | 0.389(+2.02%)                    |
| ZARA2 | 0.143         | 0.250         | 0.138(+3.50%)                  | 0.241(+3.60%)                  | 0.166           | 0.290           | 0.163(+1.81%)                    | 0.282(+2.76%)                    |
|       | GraphTERN     | GraphTERN     | GraphTERN(G2LTraj)             | GraphTERN(G2LTraj)             | DMRGCN          | DMRGCN          | DMRGCN(G2LTraj)                  | DMRGCN(G2LTraj)                  |
|       | ADE↓          | FDE↓          | ADE↓ (Gains)                   | FDE↓ (Gains)                   | ADE↓            | FDE↓            | ADE↓ (Gains)                     | FDE↓ (Gains)                     |
| ETH   | 0.378         | 0.578         | 0.360(+4.76%)                  | 0.589(-1.90%)                  | 0.382           | 0.603           | 0.366(+4.19%)                    | 0.577(+4.31%)                    |
| HOTEL | 0.129         | 0.205         | 0.122(+5.43%)                  | 0.196(+4.39%)                  | 0.129           | 0.204           | 0.124(+3.88%)                    | 0.197(+3.43%)                    |
| UNIV  | 0.255         | 0.447         | 0.248(+2.75%)                  | 0.434(+2.91%)                  | 0.256           | 0.439           | 0.245(+4.30%)                    | 0.427(+2.73%)                    |
| ZARA1 | 0.209         | 0.373         | 0.204(+2.39%)                  | 0.366(+1.88%)                  | 0.210           | 0.377           | 0.207(+1.43%)                    | 0.369(+2.12%)                    |
| ZARA2 | 0.157         | 0.273         | 0.156(+0.64%)                  | 0.262(+4.03%)                  | 0.165           | 0.284           | 0.159(+3.64%)                    | 0.275(+3.17%)                    |

The selectable strategy begins by generating key steps at a fine granularity. In other words, we set L to 2 and obtain the key group GL=2 = {t1, t3, ..., tTf }, Subsequently, we downsample GL=2 to derive M - 1 groups of coarse-grained key steps, i.e., GL=4 = {t1, t5, ..., tTf }, ..., GL=2M = {t1, tTf }. Based on these groups, we follow the above procedure to generate M trajectories, denoted as { ˆ F2, ˆ F4, ..., ˆ F2M}. To select the optimal trajectory from these M trajectories, we learn their confidence scores ˆ C ∈ RM by

<!-- formula-not-decoded -->

where ψc is a two-layer MLP, σ denotes a softmax function. ˆ Z1 and ˆ ZTf are the shared head and tail coordinates of these M trajectories. By incorporating them with A, the model can capture the overall trajectory outlines, thereby enhancing the evaluation of the M trajectories. Subsequently, we establish the ground truth for the confidence scores of { ˆ F2m}Mm=1 by

<!-- formula-not-decoded -->

where ADE denotes the average L2 distance between each coordinate of the predicted trajectory and the ground truth. Let C denote {Cm}Mm=1 . The predicted confidence scores ˆ C can be optimized by

<!-- formula-not-decoded -->

where Lr denotes the regression loss. Notably, during the inference process, we only generate the predicted trajectory corresponding to the highest confidence score through our global-to-local approach. This way can significantly reduce the inference time. The final loss function can be denoted as

<!-- formula-not-decoded -->

where LG denotes the inherent loss of the existing predictor G, L2i s is the spatial constraint loss in Eq. (4), η1 and η2 are trade-off hyper-parameters.

## 4 Experiments

## 4.1 Experimental Setup

Datasets. We conducted experiments on three widely-used datasets: ETH [Pellegrini et al., 2009], UCY [Lerner et al., 2007] and nuScenes [Caesar et al., 2020]. ETH and UCY comprise five distinct real-world scenarios, (i.e., ETH and HOTEL from ETH; UNIV, ZARA1 and ZARA2 from UCY), where 1,536 pedestrians were recorded in the world coordinates. We employ the standard leave-one-out strategy [Bae et al., 2023; Alahi et al., 2016] for training and evaluation. The task in ETH and UCY is to predict the future 12 steps (4.8 seconds) based on the observed 8 steps (3.2 seconds). nuScenes is a large-scale trajectory dataset that includes both vehicles and pedestrians. The task in nuScenes is to predict the future 12 steps (6 seconds) given the observed 4 steps (2 seconds) and the High-Definition (HD) maps.

Baseline Predictors. We evaluate the effectiveness of our G2LTraj by incorporating it into seven state-of-the-art predictors: SGCN [Shi et al., 2021], STGCNN [Mohamed et al., 2020], PECNet [Mangalam et al., 2020], AgentFormer [Yuan et al., 2021], GraphTERN [Bae and Jeon, 2023], and DMRGCN [Bae and Jeon, 2021] for the ETH and UCY datasets, and AutoBot [Girgis et al., 2022] for the nuScenes datasets. Evaluation Metrics. Following the standard evaluation strategy [Bae et al., 2023; Zhao and Wildes, 2021], we assess the performance on ETH and UCY using the average displacement error(ADE) and the final displacement error(FDE). minADE5 and minADE10 represent the average pointwise L2 distances between the predicted trajectory and the ground truth for the 5 and 10 most likely predictions, respectively, minFDE1 denotes the L2 distance between the final points of the prediction and the ground truth of the most likely prediction. MR5 ( MR10 ) calculates the ratio of scenarios where none of the predicted 5 (10) trajectories are within 2 meters of the ground truth according to FDE. We use the best-ofK strategy to compute the final ADE and FDE with K=20, following prior works. We evaluate the results on nuScenes

validation set with K=10, following [Girgis et al., 2022].

Implementation Details. We set the position embedding dimension D as 64 in Sec. 3.3. For each trajectory, We select the optimal granularity from three groups, i.e., GL=2, GL=4 and GL=8. In Eq. (10), η1 and η2 are set as 0.1 and 1, respectively. For a fair comparison, we adopt the default configurations of the baseline predictors [Bae et al., 2023; Girgis et al., 2022]. Specifically, for ETH and UCY, we employ the AdamW optimizer [Loshchilov and Hutter, 2018], set the batch size to 128, use a learning rate of 0.001, and train for 256 epochs. For nuScenes, we utilize the Adam optimizer [Kingma and Ba, 2014], set the batch size to 64, use a learning rate of 7.5e-4 , and train for 80 epochs. The model is trained using a single 3090 GPU. Notably, in practice, the predictor G commonly generates K trajectories for each agent, aiming to model the multimodal distribution of future trajectories [Bae and Jeon, 2023; Girgis et al., 2022; Bae et al., 2023]. The global-to-local generation process of these trajectories is the same and in parallel.

## 4.2 Performance Comparison of Main Results

Performance Comparison on ETH and UCY. As shown in Table 1, our G2LTraj framework consistently outperforms all the baseline models in most cases. Notably, when compared to the simultaneous methods, our approach achieves substantial improvements in ADE and FDE, reaching up to 12.12% and 13.27%. Similarly, when compared to the recursive method AgentFormer, our approach demonstrates a significant enhancement in prediction accuracy. Our G2LTraj also has certain limitations, particularly when evaluating on the UNIV subset, using SGCN and STGCNN as the backbone predictors. Overall, these results highlight the effectiveness of our global-to-local generation approach.

Performance Comparison on nuScenes. As shown in Table 2, our G2LTraj framework continuously surpasses AutoBot in three benchmarks in most cases. For example, when utilizing the ensemble strategy, G2LTraj achieves a significant decrease of 3.65% and 12.6% in terms of minADE5

Table 2: Performance comparison between AutoBot and our G2LTraj on three benchmarks of nuScenes. Compared with AutoBot, AutoBotNM denotes that the model trained without incorporating the map information. AutoBotEN denotes the ensemble of three AutoBot models to enhance final predictions. We obtain the performance values of AutoBotNM, AutoBot and AutoBotEN from the original paper [Girgis et al., 2022]. Best performances are in bold.

| Model              |   minADE5↓minADE10↓MR5↓MR10↓minFDE1↓ |      |      |      |      |
|--------------------|--------------------------------------|------|------|------|------|
| AutoBotNM          |                                 1.88 | 1.35 | 0.72 | 0.58 | -    |
| AutoBotNM(G2LTraj) |                                 1.77 | 1.24 | 0.72 | 0.55 | -    |
| AutobBot           |                                 1.43 | 1.05 | 0.66 | 0.45 | 8.66 |
| AutoBot(G2LTraj)   |                                 1.40 | 0.96 | 0.63 | 0.39 | 8.30 |
| AutoBotEN          |                                 1.37 | 1.03 | 0.62 | 0.44 | 8.19 |
| AutoBotEN(G2LTraj) |                                 1.32 | 0.90 | 0.62 | 0.37 | 8.01 |

Table 3: Effectiveness analysis in mitigating the accumulated error. Instead of generating key steps, the G2LTraj-R variant simultaneously generates all future steps and uses an RNN-based network to refine them, following [Jia et al., 2023a].

| Model     |   minADE5↓ |   minADE10↓ |   MR5↓ |   MR10↓ |   minFDE1↓ |
|-----------|------------|-------------|--------|---------|------------|
| G2LTraj-R |       1.61 |        1.03 |   0.68 |    0.48 |       9.41 |
| G2LTraj   |       1.40 |        0.96 |   0.63 |    0.39 |       8.30 |

and minADE10. On the other hand, as for the MR5 metric, the performance of G2LTraj exhibits a slight improvement compared to the raw predictor. Notably, the ensemble strategy demonstrates the best performance across all metrics. whereas the model without map information exhibits the poorest performance.

## 4.3 Ablation Studies

We conduct several ablation studies based on the AutoBot predictor and evaluated on nuScenes, unless otherwise stated. Effectiveness for Mitigating the Accumulated Error. In this part, we validate the effectiveness for mitigating the accumulated error. Specifically, as shown in Table 3, compared to G2LTraj, G2LTraj-R exhibits inferior performance. We show the comparison results more intuitively in Fig. 4. Notably, G2LTraj-R outperforms our G2LTraj by a margin when the time step is less than 3. This presents the effectiveness of the RNN-based network in successfully incorporating temporal constraints for short-term trajectory prediction. As the time step increases, both approaches experience a growth in the accumulated error. However, G2LTraj exhibits a slower growth trend compared to G2LTraj-R. It demonstrates that G2LTraj effectively mitigates the accumulated error by partitioning the entire trajectory into multiple sections.

Effectiveness Analysis of Spatial Constraints. As shown in Table 4, without any spatial constraint, the model exhibits the lowest performance, resulting in reductions of 3.6% and 7.3% in terms of minADE5 and minADE10, respectively. Besides, the constraints applied at the fine granularity level outweigh those applied at the coarse granularity level. Both constraints at the fine and coarse granularity level contribute to performance gains.

Effectiveness Analysis of Temporal Constraints. As shown in Table 5, without the position embedding, the model performance slightly decreases. Without agent features providing the past kinematics information, the model suffers from a substantial performance reduction of approximately 3.13% in terms of minADE10. Only relying on the head and the tail steps, the model is limited to the fragile temporal constraints, resulting in the lowest performance.

Figure 4: The accumulated error comparison at each future step. Accumulated Error (5) and (10) are the average of pointwise L2 distance between each future step and ground truth over 5 and 10 most likely predictions respectively.

![Image](https://ijcai.org/proceedings/2024/fulltext/285/image_000003_248eb9b95adf1847a42ca2899b22dd53853d8bab84dec40190943ed23c5792c2.png)

Table 4: Effectiveness analysis of spatial constraints in Sec. 3.2. w/o SC denotes removing all spatial constraints. SC- CG and SC- FG denote imposing the spatial constraints only under coarse granularities (i.e., L &gt; 2) and the fine granularity (i.e., L = 2), respectively.

| Model   |   minADE5↓ |   minADE10↓ |   MR5↓ |   MR10↓ |   minFDE1↓ |
|---------|------------|-------------|--------|---------|------------|
| w/o SC  |       1.45 |        1.03 |   0.63 |    0.44 |       8.51 |
| SC- CG  |       1.42 |        0.98 |   0.65 |    0.43 |       8.49 |
| SC-FG   |       1.42 |        0.96 |   0.65 |    0.40 |       8.73 |
| G2LTraj |       1.40 |        0.96 |   0.63 |    0.39 |       8.30 |

Table 5: Ablation studies of temporal constraints in Sec. 3.3. w/o PE and w/o A denote removing position embeddings and agent features in Eq. (6), respectively. only-HT denotes relying on the head and tail steps to generate the intermediate step. G2LTraj-I denotes generating the intermediate steps within each section at once.

| Model     |   minADE5↓ |   minADE10↓ |   MR5↓ |   MR10↓ |   minFDE1↓ |
|-----------|------------|-------------|--------|---------|------------|
| w/o PE    |       1.41 |        0.98 |   0.63 |    0.41 |       8.35 |
| w/o A     |       1.43 |        0.99 |   0.65 |    0.43 |       8.53 |
| only-HT   |       1.43 |        1.02 |   0.64 |    0.42 |       8.57 |
| G2LTraj-I |       1.43 |        1.01 |   0.65 |    0.45 |       8.73 |
| G2LTraj   |       1.40 |        0.96 |   0.63 |    0.39 |       8.30 |

Table 6: Ablation studies of the selectable granularity strategy in Sec. 3.4. GL=2, GL=4, GL=8 denote using a fixed granularity (i.e., 2,4 and 8, respectively) when generating all future trajectories.

| Model   |   minADE5↓ |   minADE10↓ |   MR5↓ |   MR10↓ |   minFDE1↓ |
|---------|------------|-------------|--------|---------|------------|
| GL=2    |       1.42 |        1.03 |   0.65 |    0.41 |       8.27 |
| GL=4    |       1.43 |        1.01 |   0.64 |    0.40 |       8.48 |
| GL=8    |       1.43 |        1.03 |   0.65 |    0.42 |       8.60 |
| G2LTraj |       1.40 |        0.96 |   0.63 |    0.39 |       8.30 |

Effectiveness Analysis of the Selectable Granularity Strategy. As shown in Table 6, employing three distinct fixed granularities consistently results in a reduction in performance, while the performance of G2 slightly surpasses that of G2LTraj. Overall, the performance across these three granularities surpasses that of the raw predictor. This finding confirms the efficacy of the global-to-local generation approach. Qualitative Results. As depicted in Fig. 5, the results clearly illustrate that G2LTraj greatly enhances the prediction performance compared to GraphTERN [Bae and Jeon, 2023]. Specifically, when spatial-temporal constraints are not enforced among future steps, the predicted trajectories from GraphTERN exhibit more pronounced fluctuations compared to G2LTraj. In contrast, the predicted trajectories generated by G2LTraj demonstrate greater consistency. This confirms that our G2LTraj tends to produce kinematically feasible predictions. Additionally, the predicted trajectories from our G2LTraj align more closely with the ground truth. Notably, both methods face challenges in accurately modeling unexpected behaviors, such as abrupt changes in direction, as shown in the black oval in Fig. 5. These failure cases generally involve rapid shifts in velocity and trajectory.

Figure 5: Qualitative results on the UCY validation set. The black ovals denote a failure case.

![Image](https://ijcai.org/proceedings/2024/fulltext/285/image_000004_b9bbca54764306f1f960eb84707a3340ac7f74d79b778f299f632a6e37db2d62.png)

## 5 Conclusion

In this paper, we introduce a global-to-local generation approach for trajectory prediction that alleviates the accumulated error and introduces the constraints among future steps. To enhance the kinematical feasibility, we propose the spatial constraints among the global key steps and boost the temporal constraints among the local intermediate steps. To ensure the optimal granularity of key steps for each trajectory, we introduce a selectable granularity strategy. Our G2LTraj achieves noteworthy performance improvements over seven existing trajectory predictors when evaluated on three widely used datasets. Experiment results verify its effectiveness.

## Acknowledgements

This work was supported in part by The National Nature Science Foundation of China (Grant No: 62303406, 62273302, 62273303, 62036009, 61936006), in part by Yongjiang Talent Introduction Programme (Grant No: 2023A-194-G, 2022A240-G).

## Contribution Statement

Zhanwei Zhang and Zishuo Hua made equal contributions.

## References

1. [Alahi et al., 2016] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Sil- vio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
2. [Aydemir et al., 2023] Görkay Aydemir, Adil Kaan Akan, and Fatma Güney. Adapt: Efficient multi-agent trajec- tory prediction with adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8295-8305, 2023.
3. [Bae and Jeon, 2021] Inhwan Bae and Hae-Gon Jeon. Disen- tangled multi-relational graph convolutional network for pedestrian trajectory prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 911-919, 2021.
4. [Bae and Jeon, 2023] Inhwan Bae and Hae-Gon Jeon. A set of control points conditioned pedestrian trajectory predic- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6155-6165, 2023.
5. [Bae et al., 2023] Inhwan Bae, Jean Oh, and Hae-Gon Jeon. Eigentrajectory: Low-rank descriptors for multi-modal trajectory forecasting. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 10017- 10029, 2023.
6. [Caesar et al., 2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
7. [Chen et al., 2021] Guangyi Chen, Junlong Li, Nuoxing Zhou, Liangliang Ren, and Jiwen Lu. Personalized trajec- tory prediction via distribution discrimination. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 15580-15589, 2021.
8. [Girgis et al., 2022] Roger Girgis, Florian Golemo, Fe- lipe Codevilla, Martin Weiss, Jim Aldon D'Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set transformers for joint multi- agent motion prediction. In International Conference on Learning Representations, 2022.
9. [Gu et al., 2021] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303-15312, 2021.
10. [Gu et al., 2022] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113- 17122, 2022.
11. [Jia et al., 2023a] Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the temporal dynamics for trajectory prediction: a coarse-to-fine approach. In Conference on Robot Learn- ing, pages 910-920. PMLR, 2023.
12. [Jia et al., 2023b] Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Hetero- geneous driving graph transformer for multi-agent trajec- tory prediction via scene encoding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13860- 13875, 2023.
13. [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
14. [Lee et al., 2022] Mihee Lee, Samuel S Sohn, Seonghyeon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: multi-scale vae for environment- aware long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2221-2230, 2022.
15. [Lerner et al., 2007] Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer graph- ics forum, volume 26, pages 655-664. Wiley Online Li- brary, 2007.
16. [Li et al., 2021] Shijie Li, Yanying Zhou, Jinhui Yi, and Juergen Gall. Spatial-temporal consistency network for low-latency trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1940-1949, 2021.
17. [Loshchilov and Hutter, 2018] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Inter- national Conference on Learning Representations, 2018.
18. [Mangalam et al., 2020] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Ji- tendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory pre- diction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part II 16, pages 759-776. Springer, 2020.
19. [Mangalam et al., 2021] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, way- points & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15233-15242, 2021.
20. [Marchetti et al., 2022] Francesco Marchetti, Federico Be- cattini, Lorenzo Seidenari, and Alberto Del Bimbo. Smemo: social memory for trajectory forecasting. arXiv preprint arXiv:2203.12446, 2022.
21. [Mohamed et al., 2020] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social- stgcnn: A social spatio-temporal graph convolutional neu- ral network for human trajectory prediction. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14424-14432, 2020.
22. [Navarro and Oh, 2022] Ingrid Navarro and Jean Oh. Social- patternn: Socially-aware trajectory prediction guided by motion patterns. In 2022 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 9859-9864. IEEE, 2022.
23. [Pellegrini et al., 2009] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision, pages 261-268. IEEE, 2009.
24. [Salzmann et al., 2020] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heteroge- neous data. In Computer Vision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part XVIII 16, pages 683-700. Springer, 2020.
25. [Shi et al., 2021] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgcn: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8994-9003, 2021.
26. [Shi et al., 2022a] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Fang Zheng, Nanning Zheng, and Gang Hua. Social interpretable tree for pedestrian trajectory pre- diction. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 36, pages 2235-2243, 2022.
27. [Shi et al., 2022b] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. Advances in Neural Information Processing Systems, 35:6531-6543, 2022.
28. [Varadarajan et al., 2022] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douil- lard, Chi Pang Lam, Dragomir Anguelov, et al. Multi- path++: Efficient information fusion and trajectory aggre- gation for behavior prediction. In 2022 International Con- ference on Robotics and Automation (ICRA), pages 7814- 7821. IEEE, 2022.
29. [Xu et al., 2020] Yi Xu, Jing Yang, and Shaoyi Du. Cf-lstm: Cascaded feature-based long short-term networks for pre- dicting pedestrian trajectory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12541-12548, 2020.
30. [Ye et al., 2022] Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tengfei Wang, Tongyi Cao, and Qifeng Chen. Dcms: Mo- tion forecasting with dual consistency and multi-pseudo- target supervision. arXiv preprint arXiv:2204.05859, 2022.
31. [Yuan et al., 2021] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal multi-agent forecasting. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9813-9823, 2021.
32. [Zeng et al., 2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series fore- casting? In Proceedings of the AAAI conference on artifi- cial intelligence, volume 37, pages 11121-11128, 2023.
33. [Zhao and Wildes, 2021] He Zhao and Richard P Wildes. Where are you heading? dynamic trajectory predic- tion with expert goal examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7629-7638, 2021.
34. [Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106-11115. AAAI Press, 2021.
35. [Zhou et al., 2023] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory predic- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17863- 17873, 2023.