VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning (arXiv 2025)

Created on June 11, 2026

2026   ·   vln   rl   foundation-model   navigation   ·   paper-review

Item Details
Authors Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, Hengshuang Zhao (HKU, Shanghai AI Lab)
Venue arXiv 2506.17221v2 (2025-06-25), preprint (under review)
Links arXiv · Code (Apache-2.0)

One-line Summary

VLN-R1 is an end-to-end framework that, without a separate map, depth sensor, or waypoint predictor, feeds only first-person video into a large vision-language model (Qwen2-VL) and has it directly predict four low-level actions, FORWARD/TURN-LEFT/TURN-RIGHT/STOP, in continuous environments (VLN-CE). After SFT on expert demonstrations, it is post-trained with DeepSeek-R1-style GRPO reinforcement fine-tuning (RFT) combined with a Time-Decayed Reward (TDR). As a result, it surpassed task-specific methods that use maps, odometry, and depth using RGB alone, and a 2B model caught up with a 7B SFT model through RFT alone.

  • Graph-based, discrete VLN. It began with R2R (Anderson et al., 2018) and RxR (multilingual, dense grounding). The agent teleports between predefined nodes while aligning language and vision, which was weak at open-world generalization.
  • Transition 1: continuous-environment VLN-CE. Krantz et al. (2020), “Beyond the Nav-Graph,” replaced discrete waypoints with low-level motor control (turn angle, step length), introducing free movement inside the simulator. However, it depended on additional modules such as depth maps and global maps, waypoint predictors (Hong 2022, Krantz 2021), and CLIP-based alignment.
  • Transition 2: LLMs/LVLMs as planners. NavGPT, MapGPT, InstructNav, and others converted vision to text and had an LLM plan the route. They remain confined to discrete graphs and third-person views, and errors accumulate. NaVid/Uni-NaVid (RSS 2024/2025) advanced to video-based VLMs but rely on modular visual pipelines.
  • Transition 3: bringing RFT to VLN (this paper). It is the first to transplant the DeepSeek-R1/R1-Zero finding that “RL alone improves reasoning ability” and GRPO (DeepSeekMath) into VLN. It claims to be the first case applying the verifiable-reward (RLVR) paradigm to continuous navigation decision making.
  • Afterward. RFT-based VLN lines such as ETP-R1 (“Evolving Topological Planning with RFT for VLN-CE”) are being derived.

Problem and Motivation

  1. Existing LLM-based VLN is tied to discrete topological graphs and does not generalize to continuous, unseen environments.
  2. Continuous-environment (VLN-CE) methods require additional sensors and modules such as depth, maps, and CLIP, which reduces scalability and generality in human-agent interaction.
  3. SFT alone may fit the overall text output but does not provide precise action-level control, and in particular fails to capture the temporal dependencies of multi-step future actions.

Therefore the starting point is to build “an end-to-end agent in which an LVLM issues actions directly from first-person RGB video alone” and post-train it with action-level rewards.

Key Ideas and Method

Data engine VLN-Ego

On the Habitat simulator (Matterport3D 90 scenes: train 61 / val-seen 11 / val-unseen 18), it automatically annotates pairs of “first-person video + the next 6 actions (ground truth)” along R2R and RxR trajectories. Each sample has three parts: (1) Instruction: system message + natural-language instruction, (2) Vision: past frames <History Memory> + <Current Observation>, (3) Action: four multiple-choice options A/B/C/D = FORWARD 25 cm / TURN-LEFT 30° / TURN-RIGHT 30° / STOP. About 1.8M SFT samples were obtained from R2R 630K + RxR 1.2M.

Long-Short Memory Sampling

This is a frame-selection strategy that “looks densely at recent frames and sparsely at the distant past.” Uniform sampling ignores recency, and exponential decay loses long-term context. So short-term memory is sampled densely with a small interval \(\delta_1\), long-term memory sparsely with a larger interval \(\delta_2 > \delta_1\), and the union is taken to preserve both recent detail and long-range context.

Stage 1: SFT

The model autoregressively generates the next \(n=6\) actions as text of the form “A. Move forward 25cm, B. Turn left 30°, …, D. Stop.” The option identifiers (A/B/C/D) and the action descriptions are supervised together with cross-entropy. Training the model to generate the discrete symbol (which action) and the natural-language description (what the motion is) at the same time induces compositional reasoning.

Stage 2: RFT with GRPO + Time-Decayed Reward

GRPO. It needs neither PPO’s critic nor DPO’s offline preference data. For one question, \(G\) responses (8 here) are sampled, and the rewards within the group are normalized by mean and standard deviation to compute the relative advantage \(A_i\). Good responses are pushed up slightly and bad ones pushed down cautiously (clipping), and a KL penalty prevents excessive drift from the reference model and reward hacking.

TDR (the key contribution). Only the option \(\alpha_{t+k}\) is extracted from the predicted action sequence and compared with the ground truth, but multiplied by an exponential decay \(\gamma^k\).

\[R_{\text{nav}} = \sum_{k=0}^{n-1} \gamma^k \cdot \mathbb{I}\left(\alpha_{t+k} = \alpha^{*}_{t+k}\right)\]

The intuition is that “getting near-future actions right matters much more than distant ones.” The motivation is that simple binary matching or text-similarity rewards such as ROUGE-L cannot capture temporal order, and it provides position-aware supervision of the action sequence that SFT cannot.

Experiments and Results

Setup. The backbone is Qwen2-VL-2B/7B (7B on 8 A800s + DeepSpeed ZeRO-3). SFT is 1 epoch on 1.8M samples, about 36 hours (global batch 64, lr 5e-6 cosine). RFT uses 10K from each dataset for a total of 20K, GRPO with 8 generations, lr 1e-6, \(\beta = 0.04\), about 12 hours per epoch. The input is 16 images per instance (up to 65,536 px, about 4.1K tokens). Evaluation uses SR/OS/SPL/NE/TL on VLN-CE Val-Unseen (18 fully unseen scenes), with RGB video as the only input.

Key results.

Setting SR (R2R Val-Unseen)
R2R-CMTP (map + odometry + depth) 26.4
VLN-SFT-2B 21.2
VLN-SFT-7B 24.9
VLN-R1-2B 25.6
VLN-R1-7B 30.2 (OS 41.2 / SPL 21.8 / NE 7.0)
  1. SOTA without additional sensors. VLN-R1-7B surpassed a task-specific method using maps, odometry, and depth (R2R-CMTP SR 26.4) with RGB alone.
  2. A small model catches up with a large one via RFT. 2B+RFT (SR 25.6) exceeded 7B-SFT (SR 24.9). This reproduces in VLN the DeepSeek-R1 observation that “a small model reaches large-model level through RFT.”
  3. Data-efficient cross-domain adaptation. After SFT on R2R only, RFT on RxR with just 10K samples performs better than the version trained on all of RxR (VLN-R1-7B RxR SR 22.7 vs full-data 22.3). On RxR Val-Unseen it exceeds comparison methods such as A2Nav (16.8) and WS-MGMap (15.0).

Ablation (7B, trained on R2R only).

  • Action prediction length: single action SR 15.1 → 4 actions 21.4 → 6 actions 24.9 → 8 actions 22.7. Six is optimal.
  • Frame selection: uniform 8 20.8 / uniform 16 22.0 / exponential decay 23.8 / Long-Short 24.9 (though for OS, exponential decay’s 34.3 is higher).
  • GRPO generations: k=2 24.7 → k=4 26.5 → k=6 28.4 → k=8 30.2. Converges at 8.
  • Reward function: Hard 23.8 < Uniform 25.0 < Linear 28.3 < Exponential (TDR) 30.2.

Points to keep in mind when reading. The task-specific comparison methods in the R2R table, AG-CMTP/R2R-CMTP (2021), are somewhat dated, and the main-text tables lack a same-condition comparison with recent video-VLM baselines such as NaVid/Uni-NaVid. “SOTA with RGB only” is sensitive to the choice of comparison methods. Absolute performance is also in the 30% SR range on Val-Unseen, far from practical use (the paper acknowledges this). The action space is four discrete actions with TURN ±30° and a fixed FORWARD distance, so although it is called a “continuous environment,” control is effectively quantized. Evaluation is limited to the Habitat indoor simulator with no real-robot validation. The EQA results are relegated to the appendix, so the main-text evidence is weak.

Conclusion and Significance

VLN-R1 redefined VLN as “a downstream task of an LVLM” and was the first to apply verifiable rewards (RLVR) + GRPO to continuous navigation decision making. It showed competitive results with a simple “video → action” pipeline after stripping out heavy modules such as maps, depth, and waypoint predictors, and in particular the fact that cross-domain transfer works with small-data RFT has strong practical implications in the embedded field, where data collection is expensive.

From a robotics practice perspective, the takeaways are as follows.

  • Where it can be used. When extending autonomous mobile-robot navigation in multi-floor indoor buildings to a new building or floor, the strategy of adapting a model trained in one domain with RFT on about 10K samples of new-domain data is worth adopting as is. A time-weighted reward design like TDR is applicable to any policy that outputs multi-step actions.
  • Practicality of small models. The result that a 2B model catches up with 7B SFT through RFT is meaningful from an onboard deployment perspective. However, the paper does not report latency or real-time figures.
  • Limitations. Sim-to-real is unverified since it is limited to indoor simulation; precise control and avoidance of dynamic obstacles and people are not considered due to the discrete, coarse action space; and it is unclear whether the choices of \(\gamma\) and \(n=6\) in TDR remain valid for longer horizons or outdoors. Since the GRPO reward is “match with the GT action sequence,” it is dependent on expert trajectory quality, and reward design for situations with multiple valid paths is an open problem.
  • Reproduction difficulty is moderate to high. The repo is Apache-2.0, and the training data and data generation pipeline are released. However, at the time of checking, the SFT training, testing, and RFT sections were marked “in progress,” so whether the core training and evaluation code is fully released is uncertain, and a VLN-R1-specific trained checkpoint was not clearly confirmed either. Training the 7B model requires 36 hours of SFT + 12 hours per RFT epoch on 8 A800s.

References

  • Paper: arXiv 2506.17221
  • Code: Qi-Zhangyang/GPT4Scene-and-VLN-R1
  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
  • DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — GRPO (2024)
  • Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution (2024)
  • Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (ECCV 2020)
  • NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation (RSS 2024)