AHead-VLN: A Hierarchical Reasoning Framework Guided by Foresight for Human-Aware Navigation

1 Introduction

Vision-and-Language Navigation (VLN) requires the agent to reach a target in an unknown environment using only firstperson visual observations and natural language instructions [Anderson et al., 2018; Gu et al., 2022; Park and Kim, 2023]. Early VLN studies mainly focused on static environments where agents move between discrete viewpoints. However, real-world applications require intelligent agents to navigate in a human-centered dynamic environment and abide by social norms. This demand has driven the transition from traditional VLN to human-aware VLN, shifting the research focus from finding the shortest path to robust risk perception and decision-making in dynamic social environments [M¨oller et al., 2021; Singamaneni et al., 2024].

∗Corresponding author.

Navigating in a dynamic social environment presents three major challenges. Firstly, the agent needs to predict and avoid collisions with pedestrians and other moving objects. Secondly, many critical risks lie outside the current field of view or are partially occluded, creating perceptual occlusion issues. Thirdly, the agent must respect implicit human norms of yielding, interpersonal distance, and passing order, which introduces social compliance requirements [Biswas et al., 2022]. Thus, the problem is no longer "where to go" but "how and when to reach the goal", which exceeds the capacity of traditional geometric path planning [Singamaneni et al., 2024; Zhang et al., 2024a].

The existing obstacle avoidance methods can be mainly classified into two types, but both have limitations. As shown in Figure 1(top), the reactive strategy was more common in the early VLN research. It relies on immediate control strategies and only makes short-sighted responses to the current state. In the face of complex situations, it is prone to deadlock and severe jitter [Paez-Granados et al., 2022; Sathyamoorthy et al., 2020; Gao and Huang, 2022]. The numerical prediction strategy incorporates time-related occupancy prediction and generates cost fields that change over time on the map [Thomas et al., 2023; Poddar et al., 2023; Le et al., 2024]. Although it provides forward-looking predictions, it is only utilized by the underlying planning module and is difficult to support the intelligent agent in making a reasonable trade-off between efficiency and social norms, resulting in unnecessary detours and overly cautious actions [Xue et al., 2024]. Therefore, risk information is introduced into the reasoning and decision-making process to serve as the basis for strategy weighing.

Large language models (LLMs) applied in VLN can provide powerful semantic understanding and common sense reasoning capabilities [Song et al., 2023; Yao et al., 2022]. However, directly embedding LLMs into embodied navigation brings additional challenges. Firstly, at the input level, LLMs can't directly understand the urgency risk structure contained in continuous spatio-temporal signals; these signals must be transformed into semantic forms that the model can perform reasoning on [Driess et al., 2023; Liang et al., 2022]. Secondly, at the decision-making level, a single LLM simultaneously undertaking high-level decision-making and fine motor execution leads to hierarchical mixing and reasoning conflicts [Song et al., 2023]. Frequent rewriting of local plans by high-level semantic reasoning causes decision jitter and repeated re-planning, and the low-level actions are diluted by high-level language-oriented goals, resulting in outputs that don't satisfy the constraints of the action space. Therefore, separating semantic decision-making from tactical execution is the key to maintaining the stability and executability of LLM reasoning in dynamic social environments [Pateria et al., 2021].

To address the above challenges, we propose AHeadVLN, a human-aware navigation framework that incorporates prospective temporal risks at the language level for decisionmaking. This integration doesn't simply add additional signals, instead, it transforms short-term occupancy predictions into interpretable and comparable risk alerts, embedding an understanding of future risks into the core of the decisionmaking process. By doing so, we not only enhance the safety and success rate of intelligent agents in dynamic crowd environments, but also improve the stability and interpretability of the decision-making process. This approach is expected to make navigation behaviors in complex dynamic crowd scenarios more in line with the decision-making logic and social norms of humans [Dong et al., 2025].

The main contributions are summarized as follows: (1) We have designed a path-risk association module (PRA), which is a transition from neural to symbolic compiler. It converts the future spatial-temporal occupancy prediction into a risk description in terms of paths and time indices, providing an interpretable and repeatable interface for LLM planning. (2) We have designed a hierarchical language model planner (HLP), which divides the responsibilities of LLM at the cognitive level: the Brain LLM accomplishes route selection by balancing semantic goals and predicting risks, while the Cerebellum LLM simultaneously generates movement plans for multiple candidate paths. This division method enhances decision consistency by reducing cross-level interference and reduces LLM's latency through parallel processing, thereby improving response capabilities. (3) We conduct extensive experiments and ablation studies on the HA-VLN benchmark, showing consistent improvements over strong reactive and numerical anticipatory baselines in terms of success rate, safety, and decision stability.

2 RelatedWork

Vision-and-Language Navigation. VLN aims to enable embodied agents to understand natural language instructions and make navigation decisions in visual environments [Anderson et al., 2018; Zhang et al., 2024b]. Early work was mostly conducted in discrete simulation environments based on predefined navigation maps such as Matterport3D[Chang et al., 2017], enhancing the grounding ability of VLN by extending task settings and supervisory signals [Ku et al., 2020]. To bridge the gap between simulation and real-world deployment, [Krantz et al., 2020] introduced the VLN in Continuous Environments (VLN-CE) benchmark, extending VLN from discrete navigation graphs to continuous 3D spaces and thus better reflecting realistic execution processes. Subsequent studies inherit cross-modal alignment strategies from discrete VLN and commonly adopt a two-stage paradigm, in which language-conditioned waypoints are predicted and then executed by a local navigator, connecting traditional VLN with continuous navigation settings [Hong et al., 2022; An et al., 2024]. However, most VLN-CE methods still rely on static assumptions in modeling and evaluation, making them insufficient to handle anticipatory collision risks and personal-space constraints introduced by dynamic pedestrians. HA-VLN [Dong et al., 2025] incorporates safety metrics such as collision rate in addition to the indicators, and examines the prediction of dynamic population changes and the timing strategies of waiting and detouring under occlusion and sensor noise. Thus, the task shifts from static spatial path planning to spatio-temporal risk decision-making for semantic targets.

Social Navigation and Collision Avoidance. Social navigation aims to generate motions that are both safe and socially acceptable in human-robot coexisting environments. Beyond the hard requirement of collision avoidance, it also needs to respect soft social norms such as proxemics, yielding behaviors, and human comfort [Singamaneni et al., 2024; Gao and Huang, 2022]. To address the short-sightedness of classical reactive strategies in complex interactions, existing studies mainly advance along three directions. Firstly, they combine prediction with planning by mapping the trajectories or occupancy predictions onto time-varying cost maps or risk areas, in order to achieve forward reasoning [Poddar et al., 2023; Liu et al., 2022; Le et al., 2024; Salzmann et al., 2020;

Yuan et al., 2021]. Secondly, they leverage deep reinforcement learning to incorporate social costs, risk regions, or action constraints, balancing efficiency and compliance [Xue et al., 2024; Feng et al., 2024; Cheng et al., 2024; Fl¨ogel et al., 2024]. Thirdly, they integrate uncertainty awareness and online risk adaptation, adjusting risk sensitivity based on confidence to mitigate the robustness issues caused by prediction errors [Sun et al., 2025]. Although significant progress has been made, most methods still rely on numerical costs, which makes it difficult to match spatial-temporal risks with high-level intentions in language-influenced environments. This has prompted us to adopt a more structured risk representation approach for decision-making.

Large Language Models for VLN. In recent years, large language models (LLMs) have been incorporated into the advanced decision-making of visual and language navigation (VLN), achieving route selection and action planning through language-based reasoning [Zhang et al., 2024b]. Existing methods typically connect text reasoning with executable navigation in different ways. For instance, SayCan selects skills or actions based on feasibility scores to ensure executability [Brohan et al., 2023]. LLM-Planner generates structured intermediate plans and implements them through low-level modules, thereby reducing illusions and unfeasible outputs [Song et al., 2023]. ReAct improves the robustness of online decision making via a cyclic interaction of reasoning, action, and feedback [Yao et al., 2022]. In contrast, NavGPT [Zhou et al., 2024], DiscussNav [Sun et al., 2025], and MapGPT [Chen et al., 2024] place greater emphasis on organizing observations into prompts, performing explicit reasoning, and outputting decisions at the action or path level. However, under the social awareness constraints defined by HA-VLN, navigation needs to predict human actions while meeting the requirements for collision avoidance and personal space. In this case, relying solely on language reasoning is insufficient to support stable decisions. Without structured input that clearly indicates the dynamics of the crowd and personal space, language models usually struggle to reliably generate time-sensitive strategies such as waiting and detouring. Therefore, we have designed the HLP, which separates high-level language reasoning from continuous control and risk handling to enhance the stability and reliability of navigation decisions in dynamic and populated environments.

3 Method

Task Setup. In this work, we address Vision-and-Language Navigation in Continuous Environments (VLN-CE). The agent navigates within a continuous 3D space E ∈ R3 , where its pose at time step t is denoted by Xt = (xt, yt, zt) ∈ E, representing its absolute coordinate. To perceive the environment, the agent captures panoramic RGB-D observations from a set of discrete, uniformly distributed viewpoints Θ = 0◦ , 30◦ , . . . , 330◦ . This yields a set of 12 RGB and 12 depth images, denoted as It = (I rgb t,i , I depth t,i ) | i = 1, . . . , 12, where I rgb t,i ∈ RH×W×3 and I depth t,i ∈ RH×W . The agent is provided with a natural language instruction composed of a sequence of tokens L = l1, l2, . . . , lM, describing the intended path from a starting pose Xstart ∈ E to a target location Xgoal ∈ E. Navigation is performed within a continuous 3D mesh environment using a discrete low-level action space A (e.g., move forward, turn left, turn right). The objective of this agent is to reach the target position Xgoal while adhering to the instruction L, thereby minimizing the final navigation error.

Overview of our Approach. We propose AHead-VLN to address a key challenge in human-crowded, dynamic environments: VLN decision-making lacks anticipatory risk understanding and therefore struggles with semantic reasoning and real-time planning. As illustrated in Figure 2, our method takes the current panoramic RGB-D observation It and the language instruction L as input, and produces an executable low-level action sequence. Overall, the pipeline of AHead-VLN goes from perception to compilation to planning. First, a waypoint predictor f waypoint θ generates a set of candidate paths P = {Pk}Kk=1 , while a spatio-temporal occupancy forecasting module fϕpred produces a sequence of high-dimensional future spatio-temporal occupancy grid maps MSOGM [Agro et al., 2023]. The former provides geometrically feasible options consistent with the instruction, and the latter captures the anticipated evolution of static, movable, and dynamic entities over the next few seconds. Additionally, we introduce PRA module f PRA ψ , aligning the continuous spatio-temporal risks with the discrete path key points, and compiling the high-dimensional risk information into a tactical summary Sk for global strategy selection, as well as a precise risk representation of conflict event time and location Rk. This enables the planner to carry out time-aware actions such as waiting or yielding. Finally, decision-making is carried out by the HLP. The high-level Brain reasons over {Sk}Kk=1 and selects the best path index k∗ , while the lowlevel Cerebellum generates an executable action script Πk for each candidate path in parallel. Once the Brain makes its choice, the system directly retrieves Πk∗ as the final plan, which reduces decision latency via parallel generation and fast retrieval, and improves behavioral stability while maintaining semantic reasoning capability.

3.1 Dual-StreamPerception

Waypoint Prediction. The main objective of this approach is to generate geometrically feasible candidate paths. We use the waypoint predictor f waypoint θ to generate candidate waypoints based on the current observations, bridging the gap between the discrete environment and the continuous space [Hong et al., 2022]. The predictor consists of three components. First, Visual Encoders (ResNet-50) extract features from the current panoramic RGB-D observation It. Second, a spatial relation transformer models the global scene layout by integrating the features from 12 discrete perspectives. Finally, the waypoint heat map predictor outputs the probability distribution of navigable points and uses nonmaximum suppression (NMS) to extract K discrete candidate waypoints {wk | k = 1, . . . ,K }. For each waypoint wk, the local path planner generates a kinematically feasible trajectory from the current position and downsamples it to a sequence of key points with a length of L, denoted as

{P = Pk | k = 1, . . . ,K }. These paths P serve as the geometric queries for the subsequent risk association module.

Proactive Spatio-temporal Forecasting. Parallel to the generation of waypoints, the proactive spatio-temporal prediction also operates independently. To capture comprehensive scene dynamics, we synthesize the panoramic depth observations into an approximately 360° point cloud Ct, extract pointwise semantic features using a KPConv-based 3D U-Net, and project them onto a bird's-eye-view (BEV) feature map. A 2D front-end with temporal propagation blocks then performs sequential forecasting on the BEV representation to produce future occupancy grids; we further fine-tune this network in the HA-VLN environment [Thomas et al., 2023]. The network outputs a sequence of future spatio-temporal occupancy grid maps (SOGM), denoted as MSOGM ∈ RC×T×H×W , where H × W is the BEV resolution and T is the prediction horizon. Each channel encodes the occupancy probability of a different obstacle source (e.g., static, movable, and dynamic), enabling source-aware risk assessment.

3.2 Neural-to-Symbolic Compilation

To bridge the gap between raw perception data and LLMbased planners, we introduce the PRA module. The PRA module compiles SOGM into compact, structured risk representations aligned with candidate paths. These representations serve as tactical summaries and spatio-temporal risk annotations for hierarchical LLM reasoning. The module operates in two stages: path-conditioned symbol grounding and risk schema compilation.

Path-Guided Spatio-temporal Grounding. The raw MSOGM is high-dimensional and sparse. We employ a lightweight 3D convolutional encoder, Φenc to compress the MSOGM into a spatio-temporal feature embedding F:

To geometrically align risk features with each candidate path, we resample every path Pk into a fixed-length keypoint sequence Pk = {pk,ℓ}Lℓ=1 , and build a path-environment interaction module by stacking M interaction layers. Each layer follows a standard Transformer decoder structure. We focus on the proposed path-guided sparse cross-attention: for each keypoint pk,ℓ, we define a local spatio-temporal neighborhood centered at its projected location on the feature grid, which can be approximated as a "virtual cylinder" with radius r . Let Ωk,ℓ denote the sparse sampling index set within this neighborhood.

where proj(·) maps world coordinates to feature grid indices. We then perform sparse multi-head cross-attention, where the query qk,ℓ interacts only with keys/values extracted from Ωk,ℓ. For the j-th attention head, the sparse multi-head crossattention is computed as:

This mechanism enforces a physical locality constraint by restricting cross-attention to regions consistent with the path geometry, leading to more geometrically grounded risk attribution.After M iterative interactions, we obtain the riskaware hidden-state sequence {hkℓ }Lℓ=1 , where hkℓ is the hidden token (decoder hidden state) of the ℓ-th keypoint, and serves as the input to subsequent risk prediction and structured compilation.

Risk Schema Compilation. Based on the keypoint representations {hkℓ }Lℓ=1 , the PRA module predicts a concise path-level risk summary as well as the spatio-temporal risks along the path, and deterministically compiles it into a structured risk schema that can be used by the LLM planner. Specifically, the tactical summary head outputs z k lvl ∈ R4 and z k obs ∈ R3 , while the spatio-temporal risk head outputs u k ∈ RL and U k ∈ RL×8 . To ensure the stability and repeatability of the interface, we map the logarithm of the risk level to discrete labels in the following way:

p k lvl = softmax(z k lvl ), level k = arg max p k lvl . (4) The observation feature z k obs is mapped to obstacle-source indicators o k . The tactical summary is then defined as Sk = (level k , o k ). Next, threshold-based segmentation and interval merging are applied to (u k ,U k ) to obtain a set of risk events Rk = {ej}Nk j=1 , where each event is represented as

e = (id, T , tstart, tend, sstart, send, description), (5) with id being a unique event identifier within path k. tstart, tend denote the predicted time window (as discrete forecast-step indices, convertible to seconds via ∆t), and sstart, send denote the affected path segment. The description is a short templated natural-language phrase used in the LLM prompt. The T is a discrete label derived from level k and the primary obstacle source inferred from z k obs .

In this final result, the Brain component uses Sk to compare candidate paths, while the Cerebellum component uses Rk to avoid high-risk intervals when generating action scripts explicitly. This enables the hierarchical LLM planning to maintain controllability and interpretability without directly exposing high-dimensional spatial tensors.

3.3 Hierarchical LLMPlanner

After receiving the compiled symbolic outputs from the PRA module, the system enters the final decision phase. We propose a Parallel Hierarchical LLM Planner (HLP) that decomposes planning into two stages: (i) strategic option selection over candidate paths and (ii) tactical action scripting conditioned on the selected path and its associated risk descriptors. The overall prompt structure and the parallel two-stage workflow are illustrated in Figure 3.

The Brain: Strategic Option Selection. As a high-level planner, the Brain LLM is not responsible for generating lowlevel action sequences. Instead, it evaluates trade-offs and selects a preferred option among K candidate paths. Concretely, we formulate this step as a multimodal option selection problem driven by a structured prompt.

The global context provides essential background for decision-making. It includes the task instruction L, the previous plan Pt-1 summarizing recently executed steps, and a condensed history Ht (with a fixed token budget) that summarizes visited locations and key observations. The second part of the prompt is the candidate set. For each path k ∈ {1, . . . ,K}, the system constructs an option described by three elements: a directional description Dk, a corresponding viewpoint image vk, and a tactical summary Sk produced by the PRA module. The Brain policy πBrain integrates the above information and outputs an index k∗ indicating the selected option:

This design conditions option selection on both task requirements and risk cues, aiming to favor decisions that are better aligned with the instruction while reducing predicted risk exposure.

The Cerebellum: Parallel Tactical Grounding. In parallel with the Brain's deliberation, a Cerebellum cluster performs symbol-to-action scripting. Unlike the Brain, the Cerebellum focuses on how to execute a given candidate path segment under the provided risk descriptors. For each candidate path Pk, we instantiate a dedicated Cerebellum module π(k) Cereb . It receives (i) the assigned local path segment ˜ Pk (e.g., a waypoint sequence for the current horizon), (ii) detailed spatiotemporal risk information Rk generated by the PRA module, and (iii) a predefined discrete action space A. The Cerebellum then produces an action script Πk composed of primitives from A:

Compared with purely reactive policies, conditioning the script on predicted spatio-temporal risk descriptors enables anticipatory maneuvers (e.g., waiting or detouring) rather than responding only to immediate observations. Finally, we cache the output results of the Cerebellum as the candidate script pool. Then, the Execution Selector will match the decision k∗ of the brain with the corresponding script and obtain Πk∗ for execution. Since this selection is a lightweight lookup, the overall planner reduces latency while maintaining controllability via decoupled reasoning and scripting.

4 Experiment

4.1 Experiment Setup

Simulated Environments. Our implementation is based on the HA-VLN simulator and is benchmarked on the HA-R2R instruction dataset. This dataset contains a total of 16,844 natural language instructions, covering 90 building scans. Additionally, HA-VLN provides human-related dynamic configurations (910 human models, 428 regions) in the scenes of the 90 scans, thus making the navigation evaluation more closely resemble real indoor scenarios. The agent's input is language instructions and panoramic visual observations (RGB-D), and the output is a sequence of discrete actions. To assess zeroshot generalization across different scenes, we use a sampled subset of 90 scenarios and 216 trajectories for evaluation. We conducted comparative experiments with a range of learningbased VLN methods and LLM/multi-LLM-based approaches to evaluate the navigator's performance. Among them, the PRA module is a learning-based module. During the training process, we used the AdamW optimizer, with a learning rate of 10-4 and batch size of 4 on two NVIDIA A100 GPUs. In the zero-shot inference stage, the planning module is deployed through the API: Brain uses gpt-4o-2024-05-13, and Cerebellum uses gpt-3.5-turbo.

Evaluation Metrics. We use the HA-VLN evaluation metrics to assess the agent: Navigation Error (NE) represents the average Euclidean distance between the agent's position at the end of navigation and the target point, which is used to measure the arrival accuracy. The total collision rate (TCR) counts the number of collisions between the agent and pedestrians in each episode and takes the average to reflect the overall collision intensity. TCR quantifies the frequency of collisions that occur in areas with human activity. The collision rate (CR) calculates the proportion of episodes in which at least one collision occurs, reflecting the probability of collision occurrence. Finally, the success rate (SR) counts the agent's success rate in the HA-VLN environment when it is within 3 meters of the target and no collision occurs throughout the process. These standards ensure that while taking into account the navigation accuracy, they can objectively measure its safety and compliance with spatial constraints in dynamic crowd scenarios.

4.2 Results in Simulating Environments

Table 1 presents the performance of different methods on HA-R2R under the Retrained and Zero-shot settings. Retrained refers to the strong supervision assumption of further training on the target environment, which usually significantly improves the success rate or reduces the error; while Zero-shot is closer to real deployment, requiring the model to generalize and make safe decisions without additional training. By comparison, many traditional methods often rely on Retraining to achieve better performance, and their Zero-shot performance generally declines. In contrast, AHead-VLN still maintains a high safety rate and success rate even under a complete Zero-shot condition. In the zero-shot setting, our agent achieves an SR of 0.17, which is the best result among the compared methods; relative to the baseline with the highest SR (0.15), this corresponds to an improvement of approximately 13.3%. Since SR in HA-VLN requires not only reaching the target (within 3 meters) but also avoiding collisions throughout the entire episode, this gain demonstrates the simultaneous satisfaction of task completion and safety constraints. From the safety perspective, AHead-VLN achieves TCR and CR, indicating both lower average collision intensity and lower collision occurrence probability; compared with the safest baseline, we further reduce TCR and CR by about 2.0% and 4.35%, respectively. This suggests that the proposed method can perceive and avoid risks more effectively without training-time adaptation, thereby improving the reliability and safety of navigation. Meanwhile, NE remains within a reasonable range, increasing by only 2.46% compared with the best NE baseline, which indicates that the safety benefits do not come from excessive conservatism or a substantial loss of efficiency. Overall, AHead-VLN achieves more reliable safety and stronger collision-free success without retraining.

4.3 Ablation Study

Table 2 shows the ablation results, indicating that each module contributes to the safe arrival. Firstly, PRA is the core of safety: after its removal, the increases in TCR and CR were significant, while SR decreased from 0.17 to 0.06, and NE also deteriorated, indicating that the lack of structured forward risk information would simultaneously undermine risk avoidance and goal advancement. Secondly, changing the parallel script to a serial one would cause TCR and CR to rise to 5.76/0.56, SR to drop to 0.12, and NE to change only slightly, suggesting that the parallel mechanism mainly improves the decision-making timeliness and process stability in dynamic scenarios, rather than the static navigation accuracy. Again, after removing Cerebellum, TCR became 5.41, CR became 0.53, and SR dropped to 0.14, while NE only slightly increased. This indicates that hierarchical control is crucial for the controllability of action execution and fine-grained avoidance. Finally, when both PRA and Cerebellum are removed simultaneously, the result is the worst, verifying their complementarity: PRA determines "where / when to avoid", and HLP ensures "how to safely and stably execute".

The ablation results, together with qualitative failure cases, suggest two remaining limitations: PRA may still misestimate localized risks in occluded layouts such as corners or pillars, while zero-shot LLM planning and Cerebellum scripting may suffer from ambiguous instruction grounding and imperfect action timing

5 Conclusion

We propose AHead-VLN, a human-aware navigation framework that converts predicted risks into path-level event descriptions for hierarchical LLM planning. By separating strategic path selection from tactical action execution, AHead-VLN improves decision stability, interpretability, and safety in dynamic crowd environments. Experiments on HA-VLN demonstrate better zero-shot success and collision avoidance than strong baselines. Future work will explore broader VLN scenarios and real-world deployment.

Acknowledgements

This paper is supported by the Major Scientific and Technological Research Project of Hebei Communications Investment Group Co., Ltd., "Research and Application of BeiDou + 5G Positioning and Navigation Technology" under grant KT2024D1-01.

This paper is supported by the Hebei Provincial Major Science and Technology Support Program project "Beidou + Indoor Distributed Positioning and Navigation Application Demonstration for the Large-Scale Underground Space in Xiong'an New Area" under grant 242Q040Z.