Interactive Open-Set Semantic Mapping with a 3D Scene Graph Backend

1 Introduction

Autonomous mobile robots operating in complex, largescale environments require a semantic understanding that extends beyond pure geometric occupancy to perform highlevel tasks [N¨uchter and Hertzberg, 2008; Igelbrink et al., 2024]. While volumetric representations effectively model surface geometry, they often lack the structural information needed for actual scene understanding. 3D Semantic Scene Graphs (3DSSGs) have emerged as a powerful solution to bridge this gap between sub-symbolic sensor data and symbolic reasoning by representing environments as hierarchical graphs of entities and relationships [Armeni et al., 2019; Rosinol et al., 2020]. Recent works combine the structure of 3DSSGs with the flexibility of open-set embeddings like Contrastive Language-Image Pretraining (CLIP) [Radford et al., 2021], representing instances as feature embeddings rather than a closed semantic vocabulary. However, the field of open-set mapping currently faces a significant scalability gap. Most existing works [Gu et al., 2024; Linok et al., 2025; Peng and Genova, 2023; Jatavallabhula et al., 2023] target small-scale, single-room environments, often requiring significant offline processing. Conversely, frameworks that scale to larger environments typically sacrifice geometric detail or the granularity of individual object instances [Hughes et al., 2022; Maggio et al., 2024; Laina et al., 2025]. Furthermore, many systems treat the scene graph as a mere post-hoc derivative layer, preventing real-time interaction with the map in e. g., exploration or continuous mapping scenarios.

In this system demonstration, we present a mapping architecture with a 3DSSG as the foundational backend for the entire mapping process - based on our recent work [G¨unther et al., 2026; Igelbrink et al., 2026], which details the theoretical underpinnings. Our system is specifically designed to address the challenges of large-scale open-set mapping via the following key features: (1) An architecture optimized for GPU-accelerated operations, capable of managing thousands of unique, spatially grounded object instances across large, multi-story environments. (2) The system maintains the 3DSSG as its primary, incrementally updated knowledge representation - valid at any time step without requiring postprocessing. (3) We automatically integrate well localized CLIP features directly into the graph nodes during the mapping process, enabling immediate natural language queries.

Our live demonstration focuses on our system's interactive capabilities by processing extensive data from the MatterportHabitat (HM3D) dataset [Ramakrishnan et al., 2021] alongside real-time streams from a handheld device, showcasing how it fuses high-bandwidth sensor data into a consistent semantic knowledge base that attendees can query on the fly.

2 SystemArchitecture

The core of our demonstration is a modular, GPU-accelerated pipeline designed to build and maintain the underlying 3DSSG as the foundational representation for the mapping life-cycle and all integrated data. The system takes as input a stream of RGB-D frames and their poses. The system is engineered for high throughput on the GPU, enabling the processing of large-scale environments like HM3D in real-time.

2.1 Scalable Scene Graph Backend

To manage thousands of unique instances across large-scale environments, the backend is organized into three hierarchical abstraction layers:

1. Frames Layer: Tracks 6D poses and raw 2D segmentation masks for all processed frames.
2. Segments Layer: An intermediate GPU-resident layer tracking active 3D object fragments and their visual descriptors used for associating 2D detections to 3D segments.
3. Objects Layer: The persistent global representation where fragments are consolidated into distinct object instances with accumulated geometric and CLIP features.

This structure maintains additional efficient spatial indices ensuring it remains computational efficient and queryable even as the number of nodes grows very large.

2.2 Incremental Open-Set Perception

Our pipeline avoids expensive post-processing by inferring and integrating open-vocabulary features directly during the mapping process instead of inferring them post-hoc from all processed frames.

Segmentation and Feature Extraction. For each frame, we use FastSAM [Zhao et al., 2023] for segmentation and extract a dense feature maps using the DINOv2 [Oquab et al., 2024] and openCLIP [Ilharco et al., 2021] models. Normally, both models generate only a single feature vector per image, which requires a costly cropping strategy to obtain per-mask features [Kassab et al., 2024]. To avoid this, we modify both models based on the MaskCLIP [Dong et al., 2023] approach: we compute local per-patch features from earlier transformer layers rather than the final output, allowing the models to run just once per frame.

In order to ensure high-quality semantic grounding, we modulate the local CLIP features with their global feature embeddings. The resulting features are then weighted using a view quality score based on geometric properties, ensuring that only good observations of the instance contribute to the persistent node feature.

Consistent Integration and Refinement. The transition from raw sensor data to a topologically valid scene graph is performed via a two-stage matching process optimized for the GPU:

1. Greedy Association: Local segments are matched against the global graph using 3D IoU and feature similarity, allowing for early integration of clearly visible segments and creation of new instances.
2. Active Refinement: As a greedy merging approach alone often leads to over-segmentation and the accumulation of spurious artifacts due to temporal inconsistencies in the SAM predictions, we re-evaluate modified nodes within their spatial neighborhood in an additional refinement stage with respect to their more accurate voxel overlaps. This process effectively resolves conflicts in the instances due to over- or under-segmentation and fuses fragmented segments into coherent object instances, maintaining topological consistency during exploration of the scene.

The incremental design of our mapping pipeline ensures that the 3DSSG is valid at any discrete time step, providing a stable foundation for interactive open-set queries without requiring further refinement steps.

3 Demonstration Scenario

The primary goal of this demonstration is to showcase the real-time capabilities and scalability of our presented 3DSSG backend in large, multi-story environments. The demonstration setup consists of a high-performance mobile workstation (NVIDIA RTX 4090 GPU) and a handheld RGB-D device (e. g., Orbbec Femto Bolt, Azure Kinect or iPad). Our demonstration is divided into three parts in order to highlight and showcase its main features, including large-scale processing, semantic interactivity, and live flexibility.

3.1 Large-Scale Incremental Mapping

In the first part, we demonstrate the system's ability to handle the high density of instances found in the HM3D dataset. Our pipeline processes multi-room sequences from HM3D at 3-5 Hz, incrementally building a scene graph that remains consistent and fast even as thousands of objects are instantiated and tracked, see Figures 1-2 for illustrating examples. Attendees will observe:

• 2D Segmentation and Feature Extraction: The raw RGB-D feed from the dataset is segmented in real time using the FastSAM Model and sharp local CLIP and DINO features are extracted.
• Real-Time Instance Refinement: A split-screen visualization shows how over-segmented fragments are fused into coherent semantic objects through our GPUaccelerated active refinement stage.
• Scalable Visualization: The 3D view displays the global 3DSSG, where color-coded instances represent the system's current "source of truth" regarding the environment's structure.

3.2 Interactive Querying

Once the environment is mapped, we invite attendees to interact directly with the generated representation, enabled by interactive and customized querying. Attendees can type natural language queries, such as "vintage wooden chair" or "fire extinguisher", see Figure 1c for an example. Attendees will observe:

• Instant Semantic Grounding: The system computes the cosine similarity between the query embedding and the CLIP features stored in the graph nodes, instantly highlighting matching entities in the 3D map.
• Graph Inspection: Users can click on individual 3D objects to inspect the underlying scene graph structure, including connectivity and visibility of the object during the mapping process.

3.3 Handheld Live Mapping

To showcase the robustness of the pipeline beyond static, synthetic datasets, we will showcase our mapping pipeline on real-world data, which we capture on site. This part demonstrates the system's readiness for real-world robotic deployment, where perception and open-set reasoning must operate in tandem. Attendees will observe:

• Live Data Fusion: Using a handheld RGB-D device, we capture a live dataset of the demonstration area, which is integrated into a scene graph.
• Dynamic Updates: Attendees can see how new observations update the tracked segments and objects in the 3DSSG in real-time during the mapping.

4 Conclusion

In this demonstration, we presented a mapping architecture that bridges the gap between raw sensor data and semantic reasoning by establishing the 3D Semantic Scene Graph (3DSSG) as its foundational backend. By focusing on incremental integration instead of post-processing workflows and utilizing a GPU-accelerated pipeline, we show that openset semantic mapping can scale to large, multi-story environments containing thousands of unique object instances without sacrificing real-time performance. The three-part demonstration highlights the practical utility of our approach:

1. Efficiently processing large-scale datasets,
2. enabling immediate natural language interaction through integrated CLIP features, and
3. maintaining robustness on real-world data.

Overall, our system ensures that the 3DSSG remains the incrementally updated "source of truth", providing a consistent and human-aligned knowledge base at every time step. We believe that this high-performance, explicit structure is a crucial enabler for next-generation hybrid intelligence systems. Here, perception and reasoning must operate in tandem to tackle unlabeled, complex real-world scenarios with enhanced interpretability and alignment to human concepts, as well as to connect to knowledge-grounded representations, e.g., [Renz et al., 2025].

Acknowledgements

This work is supported by the ExPrIS, LIEREx and AMENABLE projects through grants from the German Federal Ministry of Research, Technology and Space (BMFTR) with grant numbers 16IW23001, 16IW24004 and 16IW26002. The DFKI Niedersachsen (DFKI NI) is sponsored by the Ministry of Science and Culture of Lower Saxony and the Volkswagen Foundation.