GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning

1 Introduction

The emergence of Vision-Language Models (VLMs) has significantly advanced the capabilities of autonomous agents in perceiving, reasoning, and acting within complex environments [Zhang et al., 2025b]. A promising and increasingly popular research direction is that of GUI Agents, where large language model (LLM)-based agents interact with Graphical user interfaces (GUIs) to accomplish real-world tasks. By bridging visual perception, semantic understanding, and action planning, GUI Agents are poised to unlock a new era of end-to-end automation, transforming how intelligent systems interact with the digital world across domains ranging from productivity to everyday services.

†Corresponding author.

‡Work done at ByteDance.

However, the development of GUI Agents is currently constrained by the availability of high-quality training data. Existing GUI agent trajectories are primarily obtained through manual annotation or synthetic generation. Manual collection involves labeling complete action trajectories and defining high-level tasks, a process that is not only time-consuming and labor-intensive, but also susceptible to inconsistencies in quality and style due to varying annotator expertise. On the other hand, synthetic data generation is typically driven by either predefined task goals or random environment interaction. Task-driven approaches offer clear and structured objectives, but suffer from limited scalability and diversity. In contrast, interaction-based methods promote trajectory diversity, yet often lead to overly divergent behaviors that fail to converge on meaningful task outcomes.

Unlike traditional text or image data, GUI trajectories embody rich patterns of human interaction with graphical interfaces. They are not simple Markovian sequences, but rather unfolding narratives shaped by both intention and exploration. Human behavior in GUI environments typically unfolds through the following progressive stages:

• Exploration and boundary probing: When first encountering an unfamiliar application or interface, users often engage in seemingly random tapping, swiping, and navigating actions to test affordances and interface boundaries;
• Goal formulation and pursuit: As users develop clearer objectives, their actions become more directed and intentional, focusing on accomplishing specific tasks through iterative interactions;
• Cross-application coordination: To fulfill more complex goals, users frequently orchestrate multiple apps in tandem;

• Self-correction and backtracking: Users identify missteps or unreachable states and adapt by revising their strategies, undoing actions, or restarting from known checkpoints.

These patterns reveal that GUI trajectories are neither purely random nor rigidly deterministic-they embody a delicate balance between "chaos" and "order," being structured, goaldriven, and highly adaptive.

To address the limitations of existing data acquisition approaches and better capture the nuanced characteristics of human GUI behavior, we propose Graphical User Interface Reasoning and random Walk (GUI-ReWalk)-a multi-stage framework that integrates stochastic exploration with goaldirected reasoning to synthesize diverse and realistic GUI trajectories. Inspired by how humans explore unfamiliar interfaces, GUI-ReWalk begins with a random walk phase, simulating natural trial-and-error behaviors akin to an uninformed policy over a Markov chain, where each state transition depends only on the current state and available actions. As the trajectory unfolds, a large language model (LLM) acts as a reasoning agent that interprets the partially observed sequence and infers high-level goals, transitioning the framework into a reasoning-guided phase. This phase resembles a policy update in a hierarchical Markov Decision Process (hMDP), where action generation is conditioned not only on the current GUI state but also on the inferred intent-mirroring how users refine their behavior upon gaining contextual understanding. In addition, GUI-ReWalk supports multi-stride task generation, where each stride represents a subtask composed of several low-level actions, sequentially coordinated to complete complex goals across multiple interfaces or applications. By unifying the randomness of Markov chains with the intent-aware adaptability of hMDPs, GUIReWalk produces synthetic interaction data that captures both the long-tail diversity and the structured, goal-driven nature of real-world human-computer interaction.

In our experiments, we developed GUI-ReWalk-7B, built on Qwen2.5-VL-7B, and trained it on synthetic trajectory data generated within a controlled GUI environment. We evaluated its grounding and navigation capabilities across multiple benchmarks, including Screenspot-Pro, OSWorldG, and UI-Vision for grounding, and AndroidControl and GUI-Odyssey for navigation. Results demonstrate that GUIReWalk, leveraging systematic trajectory generation and task-aware supervision, achieves superior coverage of diverse interaction flows, higher trajectory entropy, and realistic user intent, as validated by human evaluations. These findings establish GUI-ReWalk as a highly effective, scalable, and dataefficient solution for advancing human-computer interaction in diverse GUI environments.

In summary, our work makes the following key contributions:

• Human-like modeling of GUI interaction: We formalize GUI trajectories as a hierarchical Markov Decision Process, where each stride combines subgoal abstraction with stride-based reasoning to capture both exploratory and goal-directed behaviors.
• The GUI-ReWalk framework: We introduce a multistage framework integrating random exploration, taskguided completion, and cross-application task initiation, enhanced by retrospective LLM-based annotation and error-recovery mechanisms that mirror real human interaction patterns.
• Dataset analysis and model evaluation: We provide an in-depth analysis of the GUI-ReWalk dataset and demonstrate its effectiveness by training GUI-ReWalk7B, which achieves substantial improvements across grounding and navigation benchmarks.

2 RelatedWorks

2.1 Evolution of GUI Agents

GUI agents have progressively evolved from rule-based systems to data-driven, end-to-end models. Early approaches-including RPA tools [Dobrica, 2022], DART [Memon et al., 2003] relied on predefined heuristics and API invocations to mimic user actions, but exhibited limited flexibility and poor generalization in dynamic or unfamiliar environments. The emergence of modular agent frameworks-integrating foundation models (e.g., GPT-4o [OpenAI, 2024]), memory systems (e.g., Cradle [Tan et al., 2024]), grounding components (e.g., MM-Navigator [Yan et al., 2023]), and tool-use mechanisms (e.g., AutoGPT [Yang et al., 2023])-enabled more adaptive and multi-step interactions. However, these systems often remained constrained by handcrafted workflows, prompt engineering, and brittle module coordination [Xia et al., 2024].

More recently, native agent architectures such as Claude Computer Use [Anthropic, 2024], Aguvis [Xu et al., 2025], OS-Atlas [Wu et al., 2024], and UI-TARS [Qin et al., 2025] have unified perception, reasoning, memory, and action within end-to-end, vision-centric models. These agents operate directly on raw screenshots without relying on structured UI representations (e.g., accessibility trees or HTML), and are trained on large-scale GUI interaction data, achieving improved generalization across platforms such as web, mobile, and desktop. Building on this foundation, recent work has further enhanced native agents through targeted training strategies-including reinforcement learning, supervised fine-tuning, and curriculum learning-along with dedicated datasets for grounded interaction [Yang et al., 2025], complex task reasoning [Tang et al., 2025], and reflective decision-making [Wu et al., 2025].

2.2 GUI Benchmarks and Environments

Benchmark environments play a central role in the development of GUI agents by defining interaction modalities, task formats, and evaluation protocols. Early benchmarks such as MiniWob++ [Liu et al., 2018] and WoB [Shi et al., 2017] provided synthetic but controlled environments-MiniWob++ emphasized UI layout and instruction diversity, while WoB enabled reproducible task execution on real webpages. Subsequent benchmarks moved toward greater realism and task complexity. WebShop [Yao et al., 2022] introduced compositional shopping tasks requiring semantic reasoning and goal-driven navigation, and Mind2Web [Deng et al., 2023] scaled to 2,000 open-ended tasks across 137 websites with fine-grained step annotations. WebArena [Zhou et al., 2024] simulated multimodal websites across diverse domains (e.g., e-commerce, social media), while WebLINX [L`u et al., 2024] extended to long-horizon, multi-turn workflows using retrieval-augmented prompting and expert demonstrations.

Beyond the browser, benchmarks such as OSWorld [Xie et al., 2024] and WindowsAgentArena [Bonatti et al., 2024] enabled agents to interact with full desktop operating systems, supporting complex workflows like file management and multi-application coordination. On mobile platforms, AndroidWorld [Rawles et al., 2025] and GUI-Odyssey [Lu et al., 2024] enabled fine-grained UI interactions across and within apps. Finally, modality-rich and cross-platform benchmarks have emerged to support generalist agents: GUIWorld [Chen et al., 2025] captured video-based GUI behavior grounded in real-world demonstrations, while AgentSynth [Xie et al., 2025a] introduced a modular benchmark that generates long-horizon desktop tasks from atomic subtasks via LLMs, facilitating structured evaluation of planning, perception, and robustness.

2.3 Data Collection and Synthesis for GUI Agents

Training GUI agents depends on large-scale, diverse task trajectories. Early datasets such as WebShop [Yao et al., 2022], Mind2Web [Deng et al., 2023], and AndroidControl [Li et al., 2024] were constructed through human demonstrations to ensure task fidelity and realism. GUI-Odyssey [Lu et al., 2024] further contributed 7,700 mobile interaction episodes spanning both within-app and cross-app workflows. However, the scalability of these human-annotated datasets is hindered by high collection costs.

To overcome this limitation, recent efforts have explored automated data generation techniques. OS-Genesis [Sun et al., 2025] extracts high-quality task trajectories via agentdriven exploration guided by learned reward models. WebSynthesis [Gao et al., 2025] performs world-model-guided search over simulated web interfaces to synthesize interaction traces. GUI-World [Chen et al., 2025] generates videobased interaction data from curated app screenshots, while TongUI [Zhang et al., 2025a] mines web tutorials and converts them into over 143K multimodal, executable task trajectories grounded in realistic application scenarios.

3 Methods

3.1 Overview

The GUI-ReWalk framework is designed to replicate the iterative, exploratory, and goal-oriented behaviors characteristic of human interactions with GUIs. To this end, we formalize the framework as hierarchical Markov Decision Process, consisting of multiple sequentially executed steps. Each stride consists of three distinct phases: a random walk phase, a task-guided completion phase, and a task initiation phase in cross-application. The GUI-ReWalk framework integrates both random exploratory actions and logical reasoning processes, thereby enhancing the diversity and length of interaction trajectories. Between these phases, we introduce retrospective annotation, which uses large language models (LLMs) to perform backward annotation and summarization of trajectories, enabling full automation of the process. In both the task-guided completion phase and the task initiation phase, we further incorporate a error task recovery scheme that generates a new goal whenever the LLM becomes blocked in the current environment, drawing upon the previously failed objective and the current state to ensure continuity and robustness in task execution.

Through this mechanism, GUI-ReWalk effectively generates multi-stride trajectories that closely mirror human multiapplication workflows, preserving logical task coherence while fluidly transitioning between exploratory and goaldirected behaviors. When the framework encounters a dead end that prevents task completion, it invokes a reflective reasoning process to revise the original objectives, leveraging both the initial goals and the interaction history, to formulate new, relevant, and executable targets. This capability enables the system to recover progress, thereby ensuring the continuity and robustness of task execution.

3.2 RandomWalk

The random walk phase reflects human exploration and boundary probing when interacting with unfamiliar interfaces. Inspired by open-world agents that acquire general skills through extensive exploration [Liu et al., 2025], we simulate this stage as trial-and-error interaction without a fixed objective. GUI-ReWalk models this behavior as a Markov chain:

where S denotes the state of the GUI environment, and Ar denotes the primitive GUI actions available in this state. The state transition probability of the random walk is defined as

where st ∈ S and art ∈ Ar denote the state and accessible actions at time step t. During the exploration phase, the action policy is set to a uniform distribution P r (art | st) = 1 |Ar | to maximize state-space coverage and emulate the chaotic probing behavior commonly observed in human users. After this, GUI-ReWalk randomly chooses an executable UI element for the selected action. For input-type actions, such as typing, GUI-ReWalk leverages the LLM to generate contextappropriate text. As the trajectory stride extends over multiple iterations, the length of the random walk is gradually reduced to better reflect the natural shift from broad exploration to focused interaction observed in real human behavior.

3.3 Task-guided Completion

After exploring the environment, human users typically form explicit goals and act purposefully. GUI-ReWalk models the task-guided completion phase as a goal-constrained Markov decision process [Kaelbling et al., 1998]. In this process, we first use the LLM to infer a high-level task goal from the terminal exploration state.

where ΦLLM is the LLM goal inference function, Tr is the terminal time step of the randomwalk. Then the task-guided completion phase is formalized as:

The state transition probability of task-guided completion captures how humans act purposefully once they have a clear goal in mind. It can be formulated as:

where agt ∈ Ag and π(agt | st) is the action policy based on the LLM. π(agt | st) selects the next action based on the current state and the intended goal. When an action cannot be executed within the current environment, π(agt | st) engages a reflective reasoning process to revise the goal g, ensuring that the updated objective remains both relevant and feasible for continued task execution. To reflect the sparsity of meaningful task completion signals, we define a sparse reward [Andrychowicz et al., 2017; Schaul et al., 2015] function as:

where S g = S × G is the task-conditioned state space and G is the goal space. The prompts for ΦLLM is shown in the Appendix.

3.4 Cross-application Task Initiation

Similar to the goal inference in the task-guided completion phase, GUI-ReWalk leverages the LLM to analyze the trajectory and annotations of the current stride Ei. Based on this analysis, it generates a semantically related cross-application goal to initiate the next stride:

where si denotes the final state of the i-th stride, and ΠLLM is the goal-generation policy implemented by the LLM. The inferred goal Gi+1 is then used to initialize the next stride Li+1. Upon switching to a new application, GUI-ReWalk reenters the process, performing a random walk, task-guided completion, and retrospective annotation, thereby constructing a new stride that continues the multi-application trajectory.

This hierarchical orchestration mirrors human multiapplication workflows, where users frequently transition from completing one task to initiating another related task across different applications. It maintains the same alternation between chaotic exploration and goal-directed execution, while ensuring semantic continuity across strides to form coherent, multi-application trajectories.

3.5 Task Recovery

Task recovery addresses scenarios where users deviate from their intended trajectory due to errors, ambiguous goals, or unforeseen interface dynamics. Inspired byMIRROR's intraand inter-reflection mechanisms for optimized reasoning in tool learning [Guo et al., 2025], GUI-ReWalk models this recovery process as an adaptive replanning mechanism built on the interplay between state monitoring and LLM-driven reasoning.

Formally, when the agent detects that the current trajectory τ = {st, at}Tt=1 fails to progress toward the inferred goal g, a recovery trigger is activated:

Once activated, we use the LLM to re-analyze the current environment to update or refine the task objective:

with ΨLLM denoting the goal-revision function. This allows the agent to dynamically adapt its task representation when the original goal g becomes infeasible or underspecified.

The subsequent execution continues under the revised policy π′ (a | s, g′ ), ensuring that the trajectory realigns with a coherent objective. This recovery loop effectively captures human-like resilience in digital environments, enabling GUIReWalk to handle interruptions, erroneous actions, and semantic drift robustly. By incorporating task recovery, the framework closes the loop between exploration, goal-directed execution, and error correction, thereby achieving more reliable and human-aligned multi-application task automation.

4 Experiments and Results

We train GUI-ReWalk on trajectory data generated within our GUI environment, using Qwen-2.5VL-7B as the base model. The evaluation is conducted from two perspectives: grounding and navigation. For grounding, the full controllability of the GUI environment enables the construction of a largescale dataset for model training. For navigation, we apply LLM-based automated filtering and trajectory scoring to select high-quality samples for supervised fine-tuning (SFT).

4.1 Grounding

We evaluate the grounding capability of GUI-ReWalk on several publicly available benchmarks, including Screenspot-Pro [Li et al., 2025], OSWorld-G [Xie et al., 2025b], and UIVision [Nayak et al., 2025]. These benchmarks share a common focus on fine-grained GUI grounding, requiring models to accurately localize target interface elements in complex, real-world desktop or application screenshots based on natural language instructions. The overall results are reported in Table 1, with a detailed breakdown provided in the Appendix.

ScreenSpot-Pro targets professional software with highresolution interfaces, spanning domains such as CAD, programming, creative design, scientific computing, office applications, and operating systems. It emphasizes grounding in visually complex environments, where dense and heterogeneous iconography poses substantial challenges. As shown in Table 1, with 100k generated grounding data, GUI-ReWalk improves upon Qwen2.5-VL-7B by 14.3.

OSWorld-G consists of fine-grained tasks that closely simulate authentic computer usage, requiring text matching, element recognition, layout understanding, precise manipulation, and refusal handling. It provides a holistic evaluation of grounding in real-world digital environments. GUI-ReWalk yields an improvement of 10.7 over Qwen2.5-VL-7B.

UI-Vision is a license-permissive benchmark designed for desktop agents. It evaluates grounding performance across diverse and fine-grained tasks in realistic desktop environments, offering a comprehensive assessment of practical grounding capabilities. GUI-ReWalk achieves an improvement of 2.2 compared with Qwen2.5-VL-7B.

GUI-ReWalk delivers consistent improvements in grounding performance across professional software, realistic desktop tasks, and fine-grained computer-use scenarios. Compared with general-purpose vision-language models of the same scale (e.g., Qwen2.5-VL-7B), GUI-ReWalk demonstrates significantly stronger capabilities in recognizing dense iconography, understanding complex layouts, and grounding actions within diverse GUI contexts. Relative to other GUI-specialized models at the 7B scale, GUI-ReWalk shows greater robustness and adaptability, benefiting from its systematic trajectory generation pipeline. Moreover, the framework proves to be highly data-efficient, achieving substantial gains with moderate training data while retaining scalability for larger settings. These results establish GUI-ReWalk as a more reliable and generalizable solution among models of comparable size, highlighting its potential to advance grounding in practical human-computer interaction tasks.

4.2 Navigation

To evaluate the multi-step decision-making capability of our proposed method, GUI-ReWalk, we conduct experiments on several publicly available benchmarks, including AndroidControl [Li et al., 2024] and GUI-Odyssey [Lu et al., 2024]. The overall results are summarized in Table 2, with detailed analyses provided in the Appendix.

AndroidControl is a static offline benchmark designed to evaluate UI comprehension, task decomposition, and action planning under both low-level and high-level task instructions. Compared with Qwen2.5-VL-7B, GUI-ReWalk achieves consistent improvements across both evaluation metrics. Specifically, on low-level tasks, GUI-ReWalk maintains roughly flat type accuracy and boosts step success rate by 11.3. On high-level tasks, it further achieves gains of 2.2 in type accuracy. These results highlight the model's superior ability in hierarchical planning and abstraction.

GUI-Odyssey provides complementary offline evaluation tasks that emphasize structured reasoning and robust action planning in controlled environments. On this benchmark, GUI-ReWalk outperforms Qwen2.5-VL-7B by 10.1 in type accuracy and 17.9 in step success rate, further validating the model's effectiveness in offline multi-step decision-making and complex task decomposition.

To further assess the agent's capability in handling openended and complex real-world tasks, we conducted a comprehensive evaluation on the challenging OSWorld benchmark. As presented in Table 5, GUI-ReWalk-7B achieves a success rate of 4.7%, effectively matching the performance of the significantly larger Qwen2.5-VL-32B model. This represents a remarkable leap over the base Qwen2.5-VL-7B model, which scores only 1.4%-translating to a nearly 3× improvement in task completion. Such results compellingly demonstrate that our high-quality, reasoning-dense synthetic data can effectively substitute for sheer parameter scale, enabling efficient 7B-scale models to rival the capabilities of 32B models and remain competitive with 72B counterparts in complex, open-ended environments.

To investigate the impact of trajectory stride on model performance, we conduct an ablation study by varying the stride number during training. As summarized in Table 3, a single stride yields suboptimal grounding and navigation due to limited diversity. Increasing to two strides substantially improves performance by capturing multi-step dependencies. However, three strides offer marginal gains and may introduce noise from redundant sub-trajectories.

Overall, GUI-ReWalk significantly improves multi-step decision-making by balancing low-level operations (AndroidControl) and high-level planning (GUI-Odyssey). By capturing the hierarchical nature of human-computer interaction, our synthetic trajectories enable the 7B model to clearly outperform its baseline. This demonstrates that well-structured data can effectively offset model scale limitations, offering a scalable path toward generalizable GUI agents.

5 Conclusion

In this work, we introduced GUI-ReWalk, a reasoningenhanced framework for synthesizing realistic and diverse GUI interaction trajectories. By unifying stochastic exploration with goal-directed reasoning, GUI-ReWalk captures both the long-tail variability and the structured intent of human-computer interactions. Its multi-stride design enables the construction of long-horizon workflows spanning multiple applications, offering a closer reflection of real-world usage patterns than prior datasets. Extensive evaluations show that training models on GUI-ReWalk yields broader interaction coverage, higher trajectory entropy, and more faithful representations of user intent across diverse benchmarks. Beyond providing a scalable data generation pipeline, GUIReWalk underscores the importance of reflective reasoning, error recovery, and platform diversity in advancing GUI agent research, paving the way toward next-generation agents that are both resilient and capable of real-world automation at scale.

Contribution Statement

The authors marked with an asterisk (∗ ) contributed equally to this work.