Pivot-Centric Trajectory Prediction: Bridging Long Horizons via Dynamical Guidance

1 Introduction

Spatio-temporal prediction is essential in many domains [Mei et al., 2025; Tian et al., 2026; Xu et al., 2026; Miao et al., 2025b]. In autonomous driving systems, accurately predicting the long-term future trajectory of surrounding agents is of crucial importance to capture others' intentions and make trustworthy decisions [Hagedorn et al., 2023; Wang et al., 2025; Miao et al., 2025a; Liu et al., 2026]. However, as the forecast horizon extends, the behavior uncertainty and ambiguity increase significantly, from an expanding search space, compounding errors, and the complex interaction among heterogeneous information, including dynamic agent history and static map features [Hu et al., 2023; Tolstaya et al., 2021; Salzmann et al., 2020; Ngiam et al., 2021; Zhou et al., 2022].

∗Corresponding author

To ease the long-term prediction uncertainty, the research communities have mainly focused on endpointcompletion [Gu et al., 2021; Shi et al., 2022; Cui et al., 2023; Ye et al., 2023; Aydemir et al., 2023] and iterative-refinement methods [Chai et al., 2019; Zhou et al., 2023; Jiang et al., 2023; Liu et al., 2024; Zhou et al., 2024], where the former focuses on utilizing prior-knowledge-based endpoint to guide the trajectory prediction for the purpose of reducing the search space, while the latter prefer to refine trajectory proposals iteratively to utilize detailed context information. (See Fig. 1 for more details).

However, the guidance provided by a simple anchor decreases significantly as the horizon length increases for priorknowledge-based methods. For example, in a scenario where an agent's goal is to bypass an obstacle in front over the next six seconds, using only one endpoint behind the obstacle can cause ambiguity between left and right bypass behaviors. In contrast, iterative-refinement-based methods can avoid this ambiguity because the proposed trajectory predicted in the first stage provides strong guidance-however, the quality of proposals matters [Zhou et al., 2023]. A wrong proposal would lead to further incorrect context for the fine-grained final predictions.

Based on the insight into how humans interact with other traffic participants, we propose a new trajectory decoding scheme called Pivot-Centric Trajectory Prediction (PCTP), trying to combine long-term intention prediction and shortterm fine-grained action learning. For this purpose, we propose the concept of pivots, which stands for the key points in a trajectory. PCTP consists of the process of Pivot prediction and pivot-guided trajectory prediction. In the first stage, the model predicts several pivots, which represent the longterm goal, considering agent-to-agent interaction and cause map information. In the second stage, the model infers the fine-grained waypoints based on the controlling pivot point to further capture the local kinematics. Moreover, the first stage can be divided into hierarchical pivot prediction processes, where each level focuses on different time scales.

Compared to the single endpoint-completion method, PCTP offers more comprehensive intermediate guidance. For instance, these intermediate pivots provide clearer indications on when and where to make a right turn at an intersection, and on when and where to interact with other agents. Compared to the trajectory-refinement method, PCTP offers a much less intermediate process such that the cumulated noise and compounding errors could be minimized. The key insight is that high-quality pivots are easier to learn, even over longer horizons, due to the reduced search space compared to an entire trajectory. In PCTP, the factorization of pivot prediction and pivot-based trajectory refinement not only preserves the diversity and quality of anchors but also ensures that timeadjacent waypoints attend to the same local information. It is also worth noting that PCTP only changes the decoding strategy and does not introduce additional input, making it easy to integrate into most state-of-the-art trajectory prediction models.

In summary, the key contributions of this work are as follows:

• We introduce Pivot-Centric Trajectory Evolution (PCTP), a method that infers trajectory based on predicted pivots, which factorizes global and local attention. Pivot prediction models intention through global interaction, while pivotguided trajectory prediction focuses on local interaction.
• We propose a trajectory decoding framework based on PCTP, which can be integrated into any trajectory predic-

tion model as a plugin.

• We evaluate the proposed method on the Argoverse I and Argoverse II datasets. Our method combined with QCNet outperforms all published ensemble-free works on the Argoverse II leaderboard.

2 RelatedWork

Trajectory prediction involves taking scene representations, including surrounding agents' histories and road maps, as inputs to forecast the future motion of agents. There are two major learning-based approaches: dense representation and sparse representation. Dense representation methods [Chai et al., 2019; Phan-Minh et al., 2020] use fixed-resolution grid structures to encode heterogeneous data and aggregate them using Convolutional Neural Networks, which focus on local information rather than global interactions. In contrast, sparse representation methods aggregate vector-based inputs using permutation-invariant set operators [Gao et al., 2020; Varadarajan et al., 2022], graph convolutions [Liang et al., 2020; Zeng et al., 2021; Da and Zhang, 2022; Liu et al., 2024; Liao et al., 2024], and attention mechanisms [Ngiam et al., 2021; Zhou et al., 2022; Nayakanti et al., 2023; Jiang et al., 2023; Philion et al., 2024], allowing for effective global interactions. To address the challenge of long-term prediction uncertainty, recent work has adopted a two-stage scheme that factorizes the task into intention prediction and intentionbased trajectory prediction.

Endpoint-based Completion. These methods assume that a trajectory is largely defined by its endpoint. Endpoint can be classified from a pre-defined set [Shi et al., 2022; Cui et al., 2023]. TNT. [Zhao et al., 2021] samples possible endpoints on centerlines in a dynamic scene and then predicts offsets from candidates to construct a discrete set for classification. DenseTNT. [Gu et al., 2021] learns an endpoint distribution heatmap during training. MTR. [Shi et al., 2022] clusters a set of endpoints in the dataset using K-Means before training, then uses these endpoints to query encoded features. Additionally, some methods [Gilles et al., 2021; Gilles et al., 2022] regress the endpoints using an anchor-free scheme.

Trajectory-based Refinement. These methods [Chai et al., 2019; Jiang et al., 2023; Liu et al., 2024] combine refinement networks [Carion et al., 2020] from computer vision with trajectory prediction. They take a proposed trajectory generated in the first stage as input and predict the offset of each waypoint in the proposed trajectory. DCMS [Ye et al., 2022] adds temporal and spatial constraints to the refinement network. QCNet. [Zhou et al., 2023] embeds the proposed trajectory into a Fourier feature to query refinement-related features. SmartRefine. [Zhou et al., 2024] uses an adaptive network to dynamically adjust refinement configurations and the number of iterations.

3 Methodology

PCTP is a two-stage trajectory decoding pipeline designed to alleviate the compounding error of long-term trajectory prediction, in which we follow a long-term pivot prediction and short-term trajectory refinement scheme. Fig.2 illustrates our framework. In Sec. 3.1, we provide a brief overview of the trajectory prediction problem. In Sec. 3.2, we delve into the intuition, design, and learning process underlying the pivot mechanism. Sec. 3.3 introduces pivot-guided trajectory prediction, while Sec. 3.4 presents the whole training details.

3.1 ProblemFormulation

Under the context of auto-driving, the goal of trajectory prediction is to predict future trajectories of surrounding agents given agent history and essential map context. Specifically, we have states of N agents in the historic Th steps A = {a1∼N -Th+1 , a1∼N -Th+2 , · · · , a1∼N 0 }, and M static local HD-map informationM= {m1,m2, · · ·mM}. It is a common practice to encode raw data of various types into high-dimensional features using individual encoders:

where Ea and Em denote the agent encoder and map encoder, respectively. H represents the features dimension after encoding.

Based on the encoder's output, we aim to predict K possible future modalities for N agents, where each modality is represented by the positions over Tf future steps. The output O is typically generated by a decoder D as:

where p represents the dimension of each prediction, e.g., p = 2 representing (x, y). Usually, a D fuses information from ea and em, and output each agent' possible trajectories directly.

PCTP, as a decoder plugin, follows the aforementioned overall framework. At the same time, we argue that directly decoding long trajectories from the raw encoded feature may lose many details and lead to significant compounding errors. We modify the decoding process of D by introducing pivots, which represent these important intermediate key points. Then, the decoding process would be decoupled as Pivot Prediction(Sec. 3.2), and Pivot-Guided Trajectory Prediction (Sec. 3.3).

3.2 Pivot Prediction

Pivot aims to bridge the gap between raw encoded features and the long-horizon predicted trajectories. Intuitively, we seek to identify the "pivot point" within trajectories that predominantly defines the trajectory temporal dynamics, thereby reducing distribution noise in the trajectory decoding process. Pivot Definition. Intuitively, there are several ways to define pivots. Ideal pivots should consider map geometry, temporal dynamics, and agent interactions. However, it would be quite complicated and error-prone to design such complex pivots. In this paper, we propose that pivots focusing solely on temporal dynamics are sufficient for most scenarios. By definition, temporal-dynamic pivots are sampled from the raw trajectory, which could be defined by sampling from the ground truth as:

where yt represents the ground truth waypoint at t time step. By introducing a time skip interval ∆t, pivots are defined by sampling from the raw ground truth. Fig 3 illustrates how we sample pivots from the initial trajectory. Using these pivots as the bridge between the raw feature and the final trajectory would significantly reduce unnecessary redundancy. Without loss of generality, we denote the sampled pivot points as P = {p0, p1, · · · , pTf/∆t }.

From End-Point To Pivot-Points. In previous works [Gu et al., 2021; Shi et al., 2022; Cui et al., 2023; Ye et al., 2023; Aydemir et al., 2023], to reduce the trajectory uncertainty, a common practice involves predicting the endpoint of an entire trajectory in the first stage, with the intention of dividing the whole trajectory action space into several endpoint-defined subspaces. This approach performs well for short-term trajectory predictions, where complex temporal dynamics are minimal. However, as the prediction horizon extends, the guidance provided by a single endpoint diminishes significantly, offering little improvement for long-term prediction tasks due to the lack of intermediate guidance. The endpoint acts like a distant destination in a fog: while the model knows where it needs to reach, there are no concrete steps to guide it along the way. We contend that, in the first stage, "pivot points" would aggregate more intermediate information and provide clear space-time transition paths.

From Trajectory-Refine To Pivot-Refine. In the long-term trajectory prediction task, there usually involves an extra stage as trajectory refinement that iteratively refines given trajectories to capture more fine-grained motion details. This extra refinement stage guidance significantly improves the model's ability to tackle long-term prediction tasks, which, at the same time, would suffer from more compounding errors when a longer time horizon is involved. In this paper, we argue that, in the second stage, a whole-size proposal trajectory is not necessary for the refinement, while pivots are all we need to conduct the refinement, as these pivots already contain all the necessary information for a target trajectory.

Hierarchical Multi-Scale Pivots. Humans typically begin with a high-level goal, then establish intermediate sub-goals, progressively decomposing these sub-goals into smaller, hierarchical objectives. Each level of goals focuses on different contextual aspects of the task. Drawing inspiration from this hierarchical cognitive process, PCTP further sets hierarchical multi-scale pivots as goals of different levels. We establish L levels of pivots, where each level uses progressive sampling intervals. At the highest level, the interval is set to Tf , which means only the endpoint is considered as a goal pivot. Following, the second highest level interval is set to Tf/2, which means the endpoint and middle-point are considered as pivots, and this pattern continues downward. Formally, the hierarchical multi-scale pivots are defined by a set of hierarchical sampling intervals {∆(l) t }:

where ∆(l) t represents skip interval at the l-th level, αl is interval growth factor from l-th level to l + 1-th level. Fig 3 illustrates how we sample pivots from different scales. Pivots sampled from different skip intervals certainly focus on different scale scene contexts. It is worth noting that pivots are predicted in agent-centric coordinates which can also be regarded as multi-scale relative spatial positional embedding. Pivot Learning. Pivot learning leverages information from ea and em to decode pivots. Consistent with previous work [Nayakanti et al., 2023; Varadarajan et al., 2022; Tang et al., 2024], we employ a cross-attention decoder to predict multiple groups of pivots with multiple learnable queries. Each group of pivots is decoded into a final single trajectory. Moreover, pivots are predicted using a hierarchical multi-scale schema that enables the decoder to progressively refine pivots based on high-level intentions and various scales of global context.

Learnable Pivot. Each query begins as a learnable embedding and corresponds to a final decoded trajectory in the interactive decoding process. In each level, queries cross-attend globally to the scene context, and output pivots at the corresponding scale, representing the current level of intention. This mode-to-scene cross-attention involves three types of queries: mode-to-history, mode-to-map, and mode-to-agent queries. Through these specialized queries, each query retrieves essential features from the corresponding scene context, enhancing the overall trajectory prediction accuracy. Formally, given the encoded agent feature ea, map feature em, and K initial queries q ∈ RK×H , pivots are learned as:

where DAttn, named Decoupled Attention, consists of mode-to-history attention, mode-to-agent attention, modeto-map attention and mode-to-mode attention with q as the queries, [ea, em] as the keys and values. P represents the predicted pivots generated by the pivot decoder Dpivot.

Multi-Scale Pivot Prediction. Instead of introducing additional modules and parameters, we reuse the mode-to-scene cross-attention module to gradually refine the mode query from high-level intention to low-level pivots in an iterative manner. In each iteration, mode queries perform mode-toscene cross-attention, outputting pivots at the current scale. Before entering the next iteration, the current output pivots are transformed to Fourier features, then embedded into the query feature space, and finally fused with the original query to achieve high-level intention embedding. Therefore, as iterations proceed, the pivot scale becomes finer, allowing for increasingly precise pivot refinement. Formally:

where we reuse the same pivot decoder Dpivot, and output multi-level pivots.

3.3 Pivot-Guided Trajectory Prediction

In the second stage, we employ a Pivot-Guided Trajectory Prediction scheme to transform pivots into a fine-grained trajectory. Intuitively, each pivot offers substantial information within its neighborhood. Consequently, the Pivot-Guided Trajectory Prediction process operates by predicting the offset of each trajectory point from its associated pivot. This approach facilitates a hierarchical learning process, where the pivot captures macro dynamics while the trajectory learns the fine-grained temporal structure.

The process begins by using predicted pivots to query the essential local context around each pivot by pivot-tocontext cross-attention. Given the predicted pivot P = {p0, p1, · · · , pTf/∆t }, along with the encoded feature ea and em, the pivot embeddings are obtained by DAttn as follows:

Next, ePi is used to predict the local offsets δpi:pi+1 :

where Dtraj is our Pivot-Guided Trajectory Decoding Module, pi denotes the i-th pivot and opi:pi+1 is final trajectory between pi and pi+1. Under this framework, we reframe the initial long-term prediction task as a series of short-term subtasks, where each subtask operates within a more manageable temporal window.

In order to facilitate the integration of our model into most SOTA models, we provide two solutions corresponding to the two current mainstream trajectory decoding solutions.

Pivot-Guided One-Shot Trajectory Decoding. For oneshot trajectory decoders[Gu et al., 2021; Cui et al., 2023; Shi et al., 2022; Tang et al., 2024] which don't need iterative operations, PCTP only learns top-level pivots without iterative operations and then replaces mode-to-context crossattention with pivot-to-context cross-attention, guiding the model to focus on relevant local context information. After generating the trajectory, each waypoint is added to its assigned pivot, effectively performing short-term trajectory prediction at each pivot and then producing the final trajectory.

Pivot-Guided k-Shot Trajectory Decoding. For decoders with refinement [Jiang et al., 2023; Zhou et al., 2023; Liu et al., 2024; Zhou et al., 2024], often referred to as k-shot trajectory decoding, PCTP also replaces mode-to-context crossattention with pivot-to-context cross-attention, while retaining the iterative refinement module to achieve a more detailed trajectory. However, experimental results show that, with PCTP, the improvement gained from additional refinement becomes minimal. Consequently, the number of refinement iterations can be reduced to as few as one or even eliminated altogether.

3.4 Training Loss

Following common practice [Zhou et al., 2022; Zhou et al., 2023], we parameterize the pivots as a mixture of Laplace distributions:

where {πk}Kk=1 are the mixing coefficients, and the Laplace density of the k-th mixture component for the i-th pivot is parameterized by location µi,k and scale bi,k. We optimize the mixing coefficients using a classification loss Lcls.

In addition, we apply the Winner-Takes-All[Lee et al., 2016] strategy to optimize PCTP. Specifically, our matching operation occurs during the pivot prediction stage to ensure intention consistency between pivot-level trajectory and point-level trajectory. This process is defined as follows:

where yt denotes the ground truth at time step t and Pi represents the i-th pivot. This formulation enables us to identify the most similar mode k∗ by minimizing the sum of L2 distances.

For stabilization, the pivot-guided trajectory decoding module stops the gradient flow through the pivots. The final loss function combines pivot-level trajectory loss Lpivot, pointlevel trajectory loss Ltraj and the classification loss Lcls for end-to-end training:

where Llpreg is the pivot's regression loss at l level, ∆lt is the skip interval of l level and β is a hyper-parameter that balances regression and classification. It is noted that the normalization factor ∆lt /Tf ensures that the learning rate for each pivot aligns with that for each point.

4 Experiments

We show our result on validation set and test set of Argoverse I and Argoverse II. As shown in Table 1, PCTP can improve the accuracy of all considered state-of-the-art methods.

4.1 Experimental Setting

Datasets. We evaluate our approach on two large-scale autonomous driving datasets: Argoverse I and Argoverse II. Both datasets capture real-world driving scenarios and include high-definition maps annotated with detailed motion data sampled at 10 Hz. Argoverse I contains 323,557 sequences collected fromMiami and Pittsburgh. The prediction task requires forecasting 3-second future trajectories based on 2 seconds of historical observations. Argoverse II, comprising 250,000 scenarios across six cities, offers enhanced data quality and presents a more challenging task: predicting 6second future trajectories given 5 seconds of observation history. We follow the official dataset guidelines for partitioning both datasets into training, validation, and test sets.

Metrics. Following the official evaluation protocols, we assess our approach using standard motion prediction metrics, including Brier minimum Final Displacement Error (b-minFDEK), minimum Average Displacement Error (minADEK), minimum Final Displacement Error

(minFDEK), and Miss Rate (MRK). The minADEK metric calculates the l2 distance between the ground truth trajectory and the best of K predicted trajectories, averaged over all future steps, while minFDEK computes only the best prediction error among the K predicted endpoints. For b-minFDEK, we add (1 - ˆπ)2 to minFDE6 to evaluate classifier performance, where ˆπ denotes the highest predicted probability score output by the classifier. Additionally, MRK represents the proportion of predictions where minFDEK exceeds 2 meters. Typically, K is set to 1 or 6; for cases where the number of trajectories exceeds K, we reduce the count by selecting those with the top-K probability scores.

Baselines. The modular design of PCTP enables integration with most existing trajectory prediction frameworks. We demonstrate this versatility by evaluating PCTP with four state-of-the-art prediction models as backbones: HPNet [Tang et al., 2024], LaFormer [Liu et al., 2024], DenseTNT [Gu et al., 2021], and QCNet [Zhou et al., 2023].

4.2 Quantitative Result

Performance on Val Set. We present qualitative results on the Argoverse I and Argoverse II validation set in Table. 1. We integrate PCTP into each state-of-the-art prediction model and achieve good improvement. For instance, PCTP can reduce b-minFDE of LaFormer, HPNet, DenseTNT, and QCNet by 3.5%, 1.2%, 1.9%, and 3.9% respectively. For the implementation of HPNet and QCNet, we provide the result of both the original version and the version without the refinement module, since the pivot-guided module also has a refinement function.

Performance on Test Set. We also present some qualitative results on the Argoverse II test set in Table. 2. It shows that PCTP improves the results of QCNet by 0.3% on bminFDE, 0.9% on minADE, 0.4% on minFDE, and 1.2% on MR. Specifically, PCTP based on QCNet outperforms all published ensemble-free works on the Argoverse 2 leaderboard.

4.3 Ablation Study

To thoroughly evaluate the impact of each component of PCTP, we conducted ablation studies across three dimensions: module effectiveness, prediction performance across varying future time steps, and the effect of different pivot sampling intervals. We only show the performance of QCNet with PCTP, while the other results of baseline with PCTP can be found in the supplement materials.

Component of PCTP. We analyze the contribution of each core module in PCTP: Pivot Prediction (PP), Multi-Scale Pivot prediction (MSP), and Pivot-guided Trajectory Prediction (PTP) by progressively removing them from the full model. PP provides foundational intermediate anchors for capturing motion dynamics, MSP enables adaptive multiscale intention interaction, and PTP maintains temporal consistency via local pivot context. As shown in Table

reftable3, each module contributes meaningfully to overall performance, with the full PCTP configuration achieving the best accuracy.

Statistical Robustness. To confirm that our improvements are not artifacts of training stochasticity, we report the mean and standard deviation over 5 independent runs in Table 4. The consistently low variance and the improvement in Negative Log-Likelihood (NLL) demonstrate that PCTP yields statistically robust gains in both point-wise accuracy and probabilistic calibration.

4.4 Case Illustration

We present a case illustration of PCTP integrated with QCNet in Fig 4. The PCTP framework generates 6 distinct trajectory modalities at the top level, each accompanied by a

6-second pivot prediction that explicitly specifies destination goals. As illustrated in Fig 4 (a), PCTP produces 4 straightpath modalities and 2 right-turn modalities. The straight modes include 1 deceleration mode (orange '×') and 3 differentiated acceleration modes (red, gray, and purple '×'), while the right-turn modes encode lane-specific intentions - firstlane (green '×') and second-lane (blue '×') targeting respectively. In subsequent stages, full trajectories are incrementally constructed through hierarchical refinement guided by these top-level pivot objectives in Fig 4 (b,c,d).

5 Conclusion

This paper introduces PCTP, a novel trajectory prediction framework, designed to address the limitations of traditional endpoint-based and iterative refinement methods, particularly for long-term prediction tasks. Unlike these two approaches, PCTP decomposes long-term trajectory prediction into a series of short-term sub-tasks. Considering hierarchical agent interactions and adaptive trajectory refinement, PCTP introduces pivot-level anchors that balance global intention with fine-grained local prediction. Ultimately, it significantly improves model accuracy and consistency over extended horizons. We believe that PCTP will serve as a foundation for future innovations in reliable trajectory prediction [Miao et al., 2025c].

Acknowledgements

This work is partially supported by SCRI, The Hong Kong Polytechnic University (No. Q-CDDG). Hao Miao is the corresponding author.