A Survey of Artificial Intelligence in Endoscopic Surgery Workflow: From Perception to Surgical Support

1 Introduction

Endoscopic surgery reduces patient trauma and accelerates recovery compared to open procedures, but this benefit comes at a perceptual cost. Unlike open surgery, where clinicians have a global anatomical view and direct tactile feedback, endoscopic procedures require surgeons to navigate complex anatomy through a narrow monocular (or near-monocular) video stream that is frequently degraded by motion blur, tissue occlusion, and organ deformation. The requirement to understand a dynamic surgical environment using this 2D video on screen imposes cognitive load, requiring surgeons to mentally reconstruct spatial relationships and infer tissueinstrument interactions from fragmented visual cues. These challenges motivate the development of AI-based tools capable of improving visual quality and extracting geometric information from endoscopic video to assist in instrument localization and manipulation.

∗Corresponding author.

Recent advances in generative models, multimodal large language models (MLLMs), and embodied intelligence have inspired broader research interest in supporting endoscopic surgery. Moving beyond treating surgical videos as passive inputs for offline analysis, these advances point toward a new paradigm of active, spatially grounded surgical assistance.

In this paradigm, learning systems are expected to reconstruct 3D anatomy, localize surgical instruments and lesions in the physical workspace, and ultimately support or automate robotic actions. These capabilities align more closely with actual clinical workflows, offering direct support that is genuinely in demand, especially given concerns about patient safety and procedural variability.

The path toward realizing this vision, however, remains unclear, despite prominent progress across different areas. Advances have been made in perception, understanding, and robotic control, but these developments are often pursued in isolation. The resulting fragmentation obscures key systemlevel bottlenecks, the dependencies between components, and the research directions most likely to yield a clinically meaningful impact. To address this gap, we present a systematic review that synthesizes progress across these interconnected areas and frames them within a unified embodied intelligence perspective. Accordingly, we ask: what are the key steps toward embodied intelligence in endoscopic surgery?

We organize this pathway into four progressively higherlevel stages as shown in Figure 1. (1) Visual Perception and Processing focuses on learning robust visual representations through image quality enhancement, domain and style normalization, depth estimation, and segmentation. (2) Surgical Scene Understanding builds upon these representations to reason about surgical entities and context, including tissues, instruments, and procedural stages. (3) 3D Scene Reconstruction lifts perception into a spatially grounded representation by recovering 3D structure and aligning reconstructed models with the physical surgical workspace. (4) Embodied Surgical Intelligence closes the perception-action loop by integrating visually-conditioned control, decision-making, and action-centric world models. For each stage, we analyze the current methods, identify fundamental bottlenecks and open challenges, and discuss how progress across stages can be jointly leveraged rather than pursued in isolation.

Finally, to advance embodied intelligence research, we first review existing datasets to highlight limitations in current data regimes (e.g., scale, diversity, and action-level supervision) that hinder progress in embodied intelligence. We then outline open research directions and articulate a long-term vision for embodied intelligence in endoscopic surgery and its integration within the operating room.

2 Visual Perception and Processing

Visual perception forms the foundation of surgical intelligence. However, raw surgical footage is often degraded by blood, motion blur, and poor lighting, which can severely compromise downstream analysis. Effective image processing is therefore required to reveal informative visual cues and stabilize subsequent interpretation. Addressing these challenges is essential for enabling higher-level understanding and decision-making. Accordingly, this section focuses on three core problems in visual perception: visual enhancement, segmentation, and geometric perception.

2.1 Visual Enhancement

Visual enhancement transforms degraded endoscopic footage into high-fidelity video suitable for downstream analysis.

Frame Processing. Early methods operate at the frame level. LighTDiff [Chen et al., 2024] employs a diffusion architecture for low-light enhancement, processing at a lower resolution before recovering fine details. LVQIS [Zheng et al., 2023] uses MPRNet to mitigate motion blur and restore sharpness in laparoscopic videos. EndoL2H [Almalioglu et al., 2020] addresses resolution limitations in capsule endoscopy through conditional GANs with spatial attention, achieving up to 12× upscaling.

Video Processing. While efficient, frame-based methods cannot enforce temporal consistency, resulting in visual jitter and flickering across frames. This limitation motivates recent work that shifts toward video-based modeling. DGGAN [Xu et al., 2025a] propagates degradation representations across neighboring frames to address multiple degradations including uneven illumination and motion blur. RVDNet [Zhu et al., 2025] utilizes smoke-free reference frames to guide video desmoking in surgery, maintaining temporal coherence.

Discussion. Despite these advances, several key challenges remain. First, unified models for compound degradations (e.g., simultaneous smoke, blur, and low light) common in surgical settings are still lacking. Second, the unresolved cost-consistency trade-off limits real-time deployment. Third, current methods are largely confined to synthetic environments, and generalization to diverse real-world scenarios remains insufficiently explored.

2.2 Segmentation and Tracking

Surgical segmentation and tracking aim to parse anatomical structures and instruments from endoscopic videos, under challenges that differ fundamentally from natural scenes.

Anatomical Segmentation. Unlike natural images, surgical anatomy exhibits weak texture cues and ambiguous boundaries between adjacent tissues. Existing methods primarily improve what can be segmented by injecting task-specific inductive biases. MoE-SAM [Li et al., 2025b] routes features through multiple organ-specialized experts, enabling finer discrimination when texture statistics vary across organs. While it is effective for known anatomical categories, it assumes stable organ-specific appearance, limiting robustness to unseen anatomy. SAM-EG [Trinh et al., 2024] instead focuses on why boundaries fail by adding explicit edge supervision, improving localization in low-contrast regions. While effective for boundary refinement, it cannot resolve semantic ambiguity when tissues share similar edge patterns, as it does not introduce higher-level anatomical reasoning.

Instrument Tracking. Instrument tracking faces a different set of constraints: identity preservation under occlusion and fast motion. Memory-based approaches such as MASAM2 [Yin et al., 2025] explicitly address temporal discontinuity by re-identifying instruments after occlusion. However, such mechanisms rely on appearance consistency and degrade under severe motion blur. Zhang et al. [Zhang et al., 2025] improve temporal stability by decoupling mask generation from category prediction, reducing jitter during rapid motion, but do not resolve inter-class ambiguity. SurgicalSAM [Yue et al., 2024] tackles this by embedding category priors, enabling discrimination between visually similar tools. Still, these methods remain fundamentally 2D and thus fail to reason about physical interaction. Xu et al. [Xu et al., 2025c] extend tracking to 3D, enabling estimation of instrument-tissue contact and reasoning beyond appearance; however, the approach requires predefined geometry, which restricts scalability and generalization across instruments.

Discussion. Existing methods address isolated failure modes, such as weak texture, boundary ambiguity, or occlusion, through task-specific priors, but often sacrifice generalizability. Moreover, heavy reliance on two-dimensional appearance constrains the ability to reason about interactions between organs and instruments. Overcoming these limitations requires more realistic modeling in real surgical settings.

2.3 Geometric Perception

Geometric perception in surgical video aims to recover spatial structure frommonocular endoscopy. Deformable tissues, specular highlights, and the lack of fixed landmarks make depth inference ill-posed, forcing surgeons to rely on subjective estimation rather than reliable geometric cues.

Monocular Depth Estimation. Most approaches focus on improving relative depth consistency under adverse visual conditions. DARES [Zeinoddin et al., 2024] enhances texture cues to sharpen depth discontinuities, improving local boundary perception under complex lighting. However, operating purely on appearance, it cannot distinguish instruments from tissue, leading to spatially plausible but semantically inconsistent depth predictions. EndoUFM [Yao et al., 2025b] addresses this limitation by coupling depth with semantic labels, allowing geometry to be interpreted in a context-aware manner. While this improves spatial coherence within a frame, it does not address instability across time.

Temporal Stability. EndoStreamDepth [Li et al., 2025a] explicitly targets instability in depth estimation by enforcing temporal smoothness, producing more stable depth streams. Nevertheless, like other monocular methods, it remains limited to relative depth and cannot recover metric scale.

Discussion. Current approaches largely treat depth as a visual reconstruction problem rather than a physical one. Relative depth may suffice for perception enhancement, but it fails to capture the true geometry required for safe interaction, such as estimating clearance, force, or tool reach. In surgical settings, where millimeter-level errors can be critical, the lack of metric depth undermines reliability and limits automation. This gap highlights the core geometric challenge: recovering physically grounded 3D structure under deformable, reflective, and dynamic conditions, which remains unresolved by monocular depth estimation alone.

3 Surgical Understanding

High-quality visual perception alone is insufficient for surgical intelligence; clinical decision-making requires structured semantic understanding and reasoning about procedural context. We distinguish surgical understanding into two complementary levels: workflow understanding, which captures objective surgical facts, and workflow reasoning, which evaluates procedural quality and safety.

3.1 WorkflowUnderstanding

Workflow understanding aims to transform surgical video into objective procedural records and temporal logic, such as operative reports that accurately document what actions occurred, when they occurred, and which surgical phase is currently underway.

Fact Grounding. Recent work introduces domain-specific grounding to address this gap. LLaVA-Endo [Yao et al., 2025a] uses professional terms to describe surgical steps. MML-SurgAdapt [Walimbe et al., 2025] identifies operational intent by analyzing how instruments and tissues interact. SurgVLM [Zeng et al., 2025] further demonstrates that a unified model can support multiple workflow tasks, improving record completeness. Nevertheless, these models remain sensitive to institutional variations in equipment and protocols, leading to unstable performance in unseen environments.

Phase Recognition. Phase recognition supports operating room scheduling and risk warnings by capturing the temporal logic and long-term dependencies of surgical workflows. SurgeNetXL [Jaspers et al., 2025] and Surgformer [Yang et al., 2024a] improve generalization via pretraining and modeling efficiency through hierarchical designs, respectively. Furthermore, by aligning visual observations with professional terminology (e.g., HecVL [Yuan et al., 2024]) and incorporating instrument intent recognition, these methods convert complex visual evolution into structured clinical facts, enabling precise partitioning of complex surgical phases.

Discussion. Existing approaches effectively capture what happens during surgery but rely heavily on learned correlations tied to specific data distributions, making them sensitive to shifts in visual appearance or procedural conventions. Developing methods that are less dependent on institutional style and more deeply grounded in invariant surgical principles remains an important direction for future work.

3.2 WorkflowReasoning

Workflow reasoning shifts from factual description to evaluative judgment, addressing whether surgical actions are safe and clinically appropriate. Traditional approaches rely on low-level kinematic metrics, which capture motion patterns but often fail to reflect adherence to surgical standards.

Recent methods introduce explicit clinical reasoning. SurgVLM [Zeng et al., 2025] and CARES [Low et al., 2025] leverage language-based evaluation and causal modeling to assess procedural quality and identify hazards. Similarly, CholecMamba [Wang et al., 2025] and ORDiRS [Shen et al., 2025] ensure reasoning consistency and operational safety through temporal modeling and digital twin grounding, respectively.

Discussion. While language-based reasoning improves interpretability, current models struggle to maintain logical consistency over long procedures and to ground abstract judgments in actionable signals. The inability to translate reasoning outputs into executable safety constraints remains a key barrier, limiting the integration of workflow reasoning into autonomous surgical systems.

4 3DReconstruction

Three-dimensional (3D) reconstruction provides spatial context that is absent in two-dimensional endoscopy, enabling intuitive perception of anatomical depth and structure. This geometric understanding serves as a foundation for robotic assistance and clinical decision-making. This section examines how different reconstruction paradigms balance fidelity, editability, and real-time performance in surgical settings.

4.1 3DRepresentations

The choice of 3D representation largely determines whether a reconstruction can support intraoperative interaction. Voxel grids explicitly encode volumetric information but scale poorly and incur high memory and computational costs, limiting real-time use. Mesh-based representations reduce storage by modeling surfaces and are well-suited for smooth anatomy; however, their fixed topology makes them fragile under large tissue deformations. Point clouds offer a lightweight alternative but are often too sparse to support precise boundary reasoning.

Implicit neural representations model (e.g., NeRF [Mildenhall et al., 2020]) continuous geometry and appearance, achieving high visual fidelity and reduced artifacts. However, their reliance on iterative optimization and expensive rendering introduce latency that is incompatible with realtime surgical requirements. These limitations have motivated explicit alternatives such as 3D Gaussian Splatting [Kerbl et al., 2023], which represents scenes using learnable Gaussian primitives.

Discussion. Selecting a 3D representation involves a fundamental trade-off between geometric expressiveness and computational efficiency. While continuous representations excel in visual quality, explicit geometric primitives are better aligned with real-time constraints. As a result, Gaussianbased representations have gained traction as a balanced solution that supports interactive performance while maintaining sufficient geometric realism for surgical applications.

4.2 Static Scene Reconstruction

Static reconstruction aims to restore accurate geometry to support spatial reasoning in visually ambiguous surgical scenes, such as texture-poor endoluminal environments. A key challenge is depth ambiguity caused by single-source illumination, which weakens natural shading cues. SVS [Sommersperger et al., 2023] addresses this issue by introducing artificial shadows, enhancing depth perception without altering underlying geometry.

Another major challenge arises from frequent occlusions caused by surgical instruments. SurgicalGS [Chen et al., 2025b] addresses this issue by inferring occluded anatomy to preserve spatial continuity during tool interaction. More recent work moves beyond purely static reconstruction toward interactive representations. For example, SurgTPGS [Huang et al., 2025] integrates geometry with language, allowing surgeons to query spatial structures directly.

Discussion. Static reconstruction assumes an unchanging anatomy, enabling spatial understanding but limiting its applicability to visualization. As a result, such models are limited in supporting tasks that require reasoning about procedural changes or surgical interventions.

4.3 Dynamic and Generative Modeling

Dynamic reconstruction [Chen et al., 2025a] handles evolving surgical scenes involving tissue deformation, instrument interaction, and structural changes from cutting. T2GS [Xu et al., 2025b] enables localized updates by decoupling instrument motion from tissue deformation to reduce drift. SimEndoGS [Yang et al., 2024b] integrates physical constraints to simulate geometric changes after tissue removal and improves scene alignment.

Recent studies increasingly incorporate generative models. EndoGaussian [Liu et al., 2025] uses deformation layers to preserve geometric consistency and reduce flicker during rapid motion. However, drastic topological changes like excision or fragmentation remain difficult to model.

Discussion. Dynamic models move reconstruction from passive display toward interactive simulation, but at the cost of stability and efficiency. While deformation can be tracked, topological changes and long-term consistency remain unresolved. Bridging physical realism with real-time responsiveness is essential for deploying dynamic reconstruction in autonomous or semi-autonomous surgical systems.

5 Embodied Surgical Intelligence

Embodied surgical intelligence extends beyond perception to enable actionable decision support, with the goal of improving safety and efficiency through autonomous or semiautonomous systems. A central challenge is the scarcity of paired data that jointly captures surgical video, language, and robot-executable action representations. To address this limitation, surgical world models simulate procedural dynamics to generate scalable multimodal supervision, while vision-language-action (VLA) models leverage this structure to translate clinical intent into executable behavior.

5.1 Surgical WorldModels

Surgical world models aim to predict how the surgical environment evolves in response to actions, enabling systems to evaluate outcomes before execution. This predictive capability is essential for risk-aware planning, such as anticipating tissue deformation during cutting or visibility changes caused by bleeding. Existing approaches largely diverge into two complementary routes, differing in how explicitly they model visual and physical detail.

High-Fidelity Simulation Route. High-fidelity approaches prioritize pixel-level realism, treating visual accuracy as the foundation for reliable prediction. Endora [Li et al., 2024] achieves realistic endoscopic rendering, and SurgWorld [He et al., 2025] advances this by learning action-controllable dynamics from unlabeled videos, while PhysGaussian [Xie et al., 2024] further constrains models with physical laws to ensure plausible tissue deformation.

Efficient Planning Route. In contrast, efficient planning approaches operate in compact latent spaces, assuming that high-resolution visual detail is largely redundant for control. GAS [Lin et al., 2024] exemplifies this strategy by optimizing robotic actions through forward prediction without full image synthesis, enabling high-frequency control suitable for delicate manipulation.

Discussion. High fidelity simulations model complex phenomena but cause latency unsuitable for real time control. These methods suit offline evaluation rather than direct surgical decision making. Conversely, planning routes prioritize speed for interactive assistance. However, compressed representations may discard clinical details and raise safety concerns. Balancing model compactness with essential medical information remains a major challenge.

Surgical world models have transitioned from theory into essential foundations for modern robotics. The current trajectory involves merging efficient planning with realistic physics. This allows robots to run parallel simulations to anticipate complications and optimize precise tissue handling before execution.

5.2 Surgical Vision-Language-ActionModels

Vision-Language-Action (VLA) models bridge high-level clinical intent and low-level robotic execution, enabling agents to interpret complex instructions in unstructured surgical environments. Current research optimizes these systems across two primary dimensions: task execution quality and physical motion control.

EmbodiedDecisionMaking andExecution. Reliable surgical VLAs require strict adherence to clinical protocols and expert heuristics. Beyond traditional imitation learning, MedFlamingo-PPO [Muppidi and Radfar, 2025] and SER [Fan and Chen, 2025] embed procedural rules and virtual fixtures as explicit policy constraints to circumvent unsafe exploration. Recent advancements focus on expertdriven alignment to incorporate refined clinical logic. For instance, SurgWorld [He et al., 2025] leverages world modeling to align surgical actions with expert text descriptions. This alignment improves the model's generalization ability across different anatomical structures and techniques, ensuring its robustness to intraoperative unpredictability.

Hierarchical Control. Hierarchical control decomposes surgical tasks into distinct layers to separate high-level planning from low-level execution. Precise surgical intervention requires continuous and responsive motion. Discrete action representations often introduce discontinuities, which are unacceptable in surgery due to the risk of tissue damage. To ensure smooth execution, EndoVLA [Ng et al., 2025] utilizes reinforcement learning and continuous control formulations. Additionally, although SRT-H [Fan and Chen, 2025] splits planning and control to reduce delay, this separation causes the system to miss sudden events because it only observes the environment intermittently. To mitigate these risks, SAFeCopilot [Nguyen et al., 2025] introduces semantic arbitration. This framework utilizes safety contracts to dynamically reallocate control authority between the agent and the surgeon, maintaining real-time responsiveness during critical incidents like bleeding or tissue slippage.

Discussion. Current surgical VLA models rely on modular designs that separate perception, reasoning, and control. While this improves stability and reduces latency, it limits the system's ability to respond to unexpected events that unfold continuously during surgery. Motion smoothness and rulebased safety can be enforced locally, but they do not provide the global awareness needed to handle anatomical variability and procedural deviations. Even with the introduction of arbitration mechanisms, achieving reliable real-time responsiveness remains difficult. Future progress depends on the tighter integration of perception and action, enabling decisions to adapt in real time as the surgical scene evolves.

6 Datasets

Data quality and diversity determine the performance of endoscopic vision models. This section analyzes technical challenges, categorizes available datasets, and discusses regulatory constraints governing their clinical application.

6.1 Challenges in Dataset Construction

The development of endoscopic models is primarily hindered by a fundamental trilemma among label precision, visual realism, and data diversity. These constraints necessitate a multisource data paradigm.

First, real surgical videos provide authentic visuals but lack precise labels. Synthetic data offers accurate labels but lacks realistic textures. This gap makes it difficult to train models that achieve both visual robustness and geometric accuracy. Second, surgical data labeling requires specialized clinical knowledge. Manual labeling by experts is expensive and slow. Differences in judgment among annotators introduce noise. Automated label generation in simulated environments is a practical way to acquire large-scale data. Third, most datasets focus on routine procedures, with rare events such as surgical complications being insufficiently represented. This distribution imbalance limits generalization in extreme cases.

6.2 Taxonomy of Endoscopic Datasets

To address these challenges, the research community leverages a diverse set of data resources, which can be broadly categorized into synthetic, phantom and clinical surgical datasets, as summarized in Table 1.

Synthetic datasets use computer graphics to generate largescale data with precise labels, enabling efficient training and low annotation costs, as well as the simulation of rare clinical scenarios that are difficult to capture in vivo. Phantom datasets rely on physical organ models and external tracking to provide accurate labels in controlled settings, offering a more realistic appearance than synthetic data while retaining the precision needed for technical validation under realistic sensing conditions. Clinical datasets are collected during real procedures and provide authentic depictions of tissue deformation and surgical artifacts, but suffer from limited and noisy annotations due to practical constraints. Recently, benchmarks like Gastric-X [Lu et al., 2026] have advanced this by aligning multi-phase CT and endoscopic imagery with biochemical indicators to reflect holistic clinical workflows.

7 FutureWorks

Physically Grounded Clinical Modeling. Most current models rely heavily on visual correlations, with limited incorporation of physical laws or medical principles. Yet safe surgical assistance depends on understanding how tissue deforms, how forces propagate, and how actions affect anatomy and risk. Future work should integrate physical modeling and medical knowledge directly into learning frameworks, enabling systems to reason about deformation, contact, and procedural constraints rather than inferring them implicitly from appearance.

Real-Time Closed-Loop Integration. Many existing approaches are developed without strict real-time requirements, assuming delayed inference or offline optimization. In surgery, however, models must operate under tight latency budgets. Future research should treat real-time operation as a core constraint, favoring models that support incremental updates, low-latency inference, and predictable execution while balancing model expressiveness.

Integrated System. Current systems often separate perception, understanding, and control into loosely connected stages, leading to delayed feedback and brittle behavior. A key future direction is the development of integrated architectures in which perception continuously informs reasoning and reasoning directly shapes action in a closed loop. Such integration would allow surgical systems to adapt online to evolving anatomy and unexpected events.

8 Ethical Considerations

The clinical deployment of AI systems for endoscopic surgery is constrained by regulatory, safety, and ethical requirements. Translating research advances into operatingroom practice requires clear accountability, responsible data governance, and sustained safety assurances.

Data Privacy. Surgical data constitute sensitive patient information, and regulatory restrictions on data sharing often lead to fragmented datasets and institution-specific models. This limits generalizability and raises concerns across clinical settings. Clear standards for data anonymization, data use, and accountability are therefore essential for both ethical compliance and robust model development.

Liability and Explainability. As AI systems increasingly influence surgical decisions and actions, opaque model behavior complicates responsibility assignment in the event of failure. Without transparent and auditable decision processes, regulatory approval is difficult to obtain and clinical trust remains limited, restricting the level of autonomy that can be safely delegated to AI systems.

Safe Deployment. Model deployment requires ongoing oversight beyond initial approval. Surgical environments evolve over time, and models must remain reliable under distribution shifts and rare events. Ethical deployment therefore demands supervision, uncertainty awareness, and postdeployment monitoring to ensure sustained safety.

9 Conclusion

Endoscopic surgical intelligence is moving beyond passive video analysis toward systems that actively support intraoperative decision-making. In this survey, we have framed this evolution through a perception-to-action pipeline. Although notable progress has been made at individual stages, existing methods are often developed in isolation, limiting realtime performance and overall reliability. Because surgery is dynamic and safety-critical, effective systems must be physically grounded, temporally responsive, and tightly integrated. Future advances will therefore depend on unifying perception, reasoning, and control into coherent, real-time frameworks that respect both physical and clinical constraints. We hope this survey draws greater attention to these challenges and highlights new research directions that could stimulate the further development of endoscopic surgical intelligence.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62471306), the Shenzhen Medical Research Fund (D2402010), and the Shenzhen Major Scientific and Technological Project (KJZD20230923114615031).

Contribution Statement

Juyan Ba and Hao Chen contributed equally.