FSR-VLN: Fast and Slow Reasoning for Vision-Language Navigation (arXiv 2025)

Created on June 11, 2026

2026   ·   vln   navigation   mapping   foundation-model   ·   paper-review

Item Details
Authors Xiaolin Zhou, Tingyang Xiao, Liu Liu, Yucheng Wang, Maiyue Chen, Xinrui Meng, Xinjie Wang, Wei Feng, Wei Sui, Zhizhong Su (Horizon Robotics / D-Robotics)
Venue arXiv 2509.13733 (v3 2025-11-25), cs.RO — conference acceptance not confirmed
Links arXiv · Project (fsr-vln folder; project page only, no runnable code released)

One-line Summary

FSR-VLN is a “fast-slow (dual-process)” object search system. It first builds a four-level Hierarchical Multi-modal Scene Graph (HMSG) of floor → room → view → object, then uses CLIP-based fast matching (intuition) on top of it to pick candidate rooms, views, and objects, and calls VLM (GPT-4o) based slow reasoning (verification and refinement) only when the fast match looks doubtful. On 87 instructions in real-world long-horizon office environments collected with a Unitree-G1 humanoid, it reports 92% SR with a mean response time of 5.5 s, cutting response time by 82% compared with MobilityVLA-style methods that run the VLM on every query.

  • Geometric/semantic-map-based ObjNav. The VLMap, OK-Robot, and OVL-MAP family embeds features from vision foundation models such as CLIP, OWL-ViT, and LSeg into voxels or point clouds. They support geometric consistency and open-vocabulary queries, but are vulnerable to reconstruction noise and odometry drift and are hard to interface directly with LLMs/VLMs.
  • 3D scene graphs. ConceptGraphs, HOVSG, DOVSG, and OpenIN abstract floors/rooms/objects into nodes and provide an LLM interface. However, they rely on rigid instruction formats such as “object A in region B on floor C” and on CLIP similarity search, so they do not fully exploit VLM reasoning.
  • Image topological graphs. MobilityVLA (goal-frame retrieval with a long-context VLM), Uni-NaVid, ReMEmbR, MapGPT, Astra, RoboHop, TANGO. They preserve raw images and pair well with VLMs, but lack 3D structure, depend on video captioning that is inefficient over long sequences, and suffer semantic-geometric mismatch.
  • This paper’s shift. It proposes HMSG, which merges the two lines (hierarchical scene graph + image topology) into one. The key is inserting a view node layer between the traditional three levels so that an image-level representation for VLM reasoning is held explicitly. On top of that it adds a Kahneman-style System 1/2 conditional call, fast (CLIP) → slow (VLM), to capture both accuracy and latency.
  • Contemporaries. This sits in a wave of VLM-based VLN work such as osmAG-LLM (the HM3D-SEM baseline), JanusVLN, and DreamNav. Since the first release was 2025-09, downstream citation impact is not yet confirmed.

Problem and Motivation

Existing VLN is weak at long-horizon spatial reasoning; in particular, success rates are low and inference latency is high in indoor spaces with long corridors and multiple rooms. The authors identify the root bottleneck as the absence of a persistent long-horizon spatial memory that encodes, organizes, and retrieves environment knowledge. Geometric maps and 3D scene graphs are geometrically consistent but depend on pre-extracted features and interact weakly with VLMs. Conversely, image topologies are VLM-friendly but lack 3D structure, and captioning long videos is inefficient. Running a VLM over the whole video for every query is accurate but slow (MobilityVLA about 30 s). The motivation is that “always using heavy reasoning is wasteful.”

Key Ideas and Method

The method has three parts: (A) the HMSG representation, (B) HMSG construction, and (C) fast-to-slow reasoning.

HMSG (Hierarchical Multi-modal Scene Graph)

Four node levels; each node carries geometric, semantic, and topological multimodal features.

Node Geometry Semantics Connections
Floor min/max height, PLY point cloud id/name list of child rooms
Room 2D polygon boundary, point cloud name + CLIP embedding connected views and objects
Object 3D bounding box, point cloud CLIP embedding parent room, views it is visible in
View (new) camera pose CLIP embedding + VLM (GPT-4o) generated text description visible objects (visibility); undirected view-view edges = relative pose

A traditional scene graph only knows “there is a chair in the room” and relies solely on CLIP matching, so its visual and spatial understanding is thin. Inserting a view layer lets the VLM directly read and reason about “from this viewpoint, this object looks like this,” strengthening both image-level navigation and object localization at once. The relative-pose edges between view nodes are used for global path planning.

HMSG Construction

  • SLAM uses FAST-LIVO2 (LiDAR-inertial-visual odometry) to extract RGBD and poses, and an instance-level open-vocabulary map is built in the HOVSG manner. Objects and views are assigned to rooms by geometric overlap on the top-down map.
  • Floors and rooms are added sequentially and given CLIP features. Unlike HOVSG, room names are inferred from image views with GPT-4o.
  • Each view stores a CLIP embedding + VLM caption + camera pose, and visible objects are linked to the view by edges.
  • For each object, the mean depth of the views it appears in is computed and the nearest (minimum-depth) view is designated the “best view” (representative viewpoint). This becomes the verification reference for the later slow reasoning.

Fast-to-Slow Reasoning

  1. LLM-based instruction understanding. For spatially explicit instructions (“go to the blue cylindrical stool in the office”), the LLM acts as a hierarchical concept parser that decomposes them into floor/region/object and maps them to scene graph nodes. For non-spatial instructions (“I’m tired, where can I rest?”, “I’m thirsty”), the LLM acts as a goal inference agent that infers the most relevant object and region from user intent.
  2. Fast Matching (CLIP). If a room name is given, the room is matched first and views/objects are matched within it. The goal view is chosen by CLIP similarity between the query text and view embeddings, and candidate targets by similarity between the query text and object embeddings. It is fast, but CLIP’s limitations mean it can be wrong.
  3. Slow Reasoning (GPT-4o, called conditionally).
    • Verification: the best view of the fast-matched object is shown to GPT-4o with the question “is this object in this view?” Since in the HMSG an object must appear in its own best view, if GPT-4o says “no,” the match is judged untrustworthy.
    • View reselection: if wrong, the LLM uses the text descriptions of the unmatched views to pick the most semantically consistent image as view-1, then the fast-matched view and view-1 are compared by VLM reasoning to decide the final goal image.
    • Object rematching: once the goal image is fixed, the object list of that image is traversed, query-object CLIP similarity is recomputed, and the goal object is updated.

By analogy, it is a person who comes up with an answer on quick instinct, and only when that instinct feels doubtful goes back to inspect the photo carefully and double-check.

Experiments and Results

Setup. Hardware is a Unitree-G1 humanoid + Intel RealSense D455 RGBD + Livox Mid360 LiDAR. Speech recognition (FunASR + VAD), LLM target inference, HMSG, path planning, and whole-body control are integrated; inference runs in the cloud while the robot handles data collection and control. The datasets are four real-world long-horizon office scenes collected by the humanoid (long corridors + multiple rooms) and eight HM3D-SEM scenes. Following the MobilityVLA setup, 87 crowdsourced instructions are split into four types: RF (Reasoning-Free), RR (Reasoning-Required), SO (Small Objects), and ST (Spatial Target, testing long-horizon memory via room type). Metrics are SR and RSR top-n@k (one of the top-n predictions lies within k m of GT). Baselines are OK-Robot, HOVSG, and MobilityVLA (not public, so re-implemented by the authors with GPT-4o: CLIP top-50 frame selection followed by reasoning); on HM3D-SEM, HOVSG and osmAG-LLM.

Real-world results (87 instructions).

Method SR Response time
OK-Robot 0.609 0.2 s
HOVSG 0.517 0.2 s
MobilityVLA (re-implemented) 0.345 ~30 s
FSR-VLN 0.920 5.5 s (fast only: 1.5 s)
  1. SR 92% (80/87). Relative improvements of +167%, +51%, and +77% over MobilityVLA, OK-Robot, and HOVSG respectively. FSR-VLN applied a stricter criterion: the robot must be inside the correct room to count as success.
  2. Response time 5.5 s. Because slow reasoning is called only when fast matching fails, and the VLM is applied only to candidate views rather than the whole sequence, response time is 82% shorter than MobilityVLA.
  3. Long-horizon robustness. RSR@Top1 is 96.6% (84/87) at the 4-5 m threshold, the best across all distances. MobilityVLA is the worst at short range (1-2 m) but second after FSR-VLN at 3-5 m, showing its sequence-reasoning capability.

HM3D-SEM (RSR@Top1 at 1/2/3 m). FSR-VLN 0.87/0.88/0.88 vs HOVSG 0.52/0.64/0.70 and osmAG-LLM 0.28/0.50/0.69. osmAG-LLM discards visual CLIP features and uses only XML text, so its Top1 is much lower.

Ablation (RSR@Top1, 1 m). Neither ST nor NR: 0.724 → with ST: 0.816 (room information restricts the search to that room) → ST + NR: 0.920 (adds VLM verification and refinement). Both room-level guidance and slow reasoning contribute; NR raises RSR at the cost of 1.5 s → 5.5 s.

Points to keep in mind when reading.

  • MobilityVLA is not public, so the authors re-implemented it with GPT-4o. It may be weaker than the original, so the 167% margin risks being overstated. OK-Robot/MobilityVLA have no room information and cannot interpret ST instructions; the authors say they compensated by “counting same-category objects in other rooms as success,” but a comparison asymmetry remains.
  • The real-world evaluation is small (four self-collected rooms, 87 instructions) and not a public benchmark, so external validation is hard.
  • The metrics are RSR (retrieval success rate) centric. They mainly measure whether the “target coordinate retrieval” was right rather than actual driving success, and there is no quantitative report of path planning, control, or actual arrival success (SPL etc.). It is closer to “goal retrieval” success than “navigation” success.
  • There are no ablations removing the view layer itself, comparing GPT-4o with open VLMs, or reporting slow-reasoning call frequency and failure rates. The slow-call ratio, which is the denominator of the “82% reduction,” may be dataset-dependent. The qualitative examples show only a success case in one room, with no failure-mode analysis.

Conclusion and Significance

HMSG, which stitches together hierarchical scene graphs (geometry, long horizon) and image topologies (VLM-friendly) via view nodes, is a practical blueprint for VLN map design in the VLM era. The conditional cascade that calls the slow VLM only when fast intuition fails, rather than “always VLM,” is a direct answer to the latency budget problem of real-world humanoids and has implications for cost- and latency-sensitive on-device/cloud hybrid deployments. The view layer + conditional fast/slow appears to be a practical and sensible engineering integration, and the latency-accuracy trade-off improvement is convincing. However, the “SOTA” claim and the size of the quantitative margins must be read against the limitation of a self-built benchmark and re-implemented baselines.

From a robotics practice viewpoint, the takeaways are as follows.

  • Where it can be used. For autonomous mobile robots in multi-floor indoor buildings, a “floor → room → view → object” hierarchical map fits environments with clear room structure such as offices and hospitals. In particular, for operating scenarios where users name the room (“the water dispenser in the pantry”), the ablation showing RSR rising from 0.724 to 0.816 with ST guidance alone is worth applying immediately.
  • Latency budget design. Whether an open VLM or an API VLM, the cascade structure that does not call it on every query and only invokes it when low-cost embedding matching has low confidence is a pattern portable to any system.
  • Limitations. HMSG construction is slow, so real-time mapping is not possible (acknowledged by the authors), and a static environment is assumed, so environments where furniture is moved cannot be handled. Since slow reasoning depends on GPT-4o’s visual judgment, there is no analysis of misjudgments under occlusion, similar objects, or lighting, and the feasibility of substituting an open VLM and the resulting performance gap are unknown.
  • Reproduction difficulty is high. The fsr-vln folder on GitHub contains only the project page (index.html + static) with no runnable code or training scripts (checked 2026-06), and the repository has no LICENSE file, so usage rights are uncertain. Checkpoints, the 87 instructions, and the four-room dataset are also unreleased. The dependency stack (FAST-LIVO2, HOVSG, GPT-4o API, FunASR, Unitree-G1 + D455 + Mid360) is heavy as well. That said, the algorithm itself can be reproduced by implementing the view layer and GPT-4o verification logic on top of HOVSG (public), since Algorithm 1 and the reasoning flow are relatively clear. Without the same benchmark, numerical agreement cannot be verified.

References

  • Paper: arXiv 2509.13733
  • Project: HorizonRobotics/robot_lab (code not released)
  • HOVSG: Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (RSS 2024)
  • MobilityVLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs (CoRL 2024)
  • OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics (RSS 2024)
  • FAST-LIVO2: Fast, Direct LiDAR-Inertial-Visual Odometry (T-RO 2025)
  • ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (ICRA 2024)