NaVILA: Legged Robot Vision-Language-Action Model for Navigation (RSS 2025)

Created on June 11, 2026

2026   ·   vln   vla   legged   navigation   ·   paper-review

Item Details
Authors An-Chieh Cheng, Yandong Ji, Zhaojing Yang (co-first authors), Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, Xiaolong Wang (UC San Diego, USC, NVIDIA)
Venue arXiv 2412.04453 (2024-12, v2 2025-02), accepted at RSS 2025
Links Project · Code (Apache-2.0) · Benchmark (MIT)

One-line Summary

NaVILA solves Vision-and-Language Navigation (VLN) for legged robots such as quadrupeds and humanoids in two levels. The upper-level VLA (Vision-Language-Action) model outputs only natural-language mid-level actions such as “move forward 75 cm” or “turn right 30 degrees,” and the lower-level vision-based locomotion RL policy executes those commands as actual joint control. With this structure it surpassed prior SOTA methods that relied on panoramas, depth, odometry, and waypoint predictors while using only a single RGB camera, and showed an average success rate of 88% over 25 instructions on real Unitree Go2, H1, and Booster T1 robots.

The developments leading up to this paper can be organized into five threads.

  1. The discrete VLN era. After R2R (Anderson et al., 2018), the mainstream was research that followed instructions by teleporting between nodes of a navigation graph in simulators such as Matterport3D. It was strong at high-level decision making but ignored actual motion control.
  2. Transition to continuous-environment VLN-CE. Krantz et al. (2020) proposed VLN-CE, which requires low-level actions such as forward and turn instead of teleportation. Subsequently, ETPNav, BEVBert, Reborn, and others improved performance by predicting candidate positions with a waypoint predictor (Hong et al.) pretrained in simulation, but because of the simulation dependence, generalization was fragile and obstacle avoidance and low-level motion remained unsolved.
  3. Emergence of single-view video VLMs. NaVid (RSS 2024) predicted the next action from a single-viewpoint RGB video without panoramas or depth. It is NaVILA’s closest predecessor and its key baseline.
  4. The manipulation VLA trend. RT-2, OpenVLA, and Open X-Embodiment established the paradigm of fine-tuning VLMs on robot demonstrations to output low-level actions directly. However, most focused on manipulation and used quantized low-level commands.
  5. Spatial-reasoning VLMs. SpatialVLM and SpatialRGPT gave VLMs the ability to reason about distance and direction. This is the foundation for NaVILA’s idea of “outputting 75 cm / 30 degrees in language.”

NaVILA’s turning point is breaking the convention from (4) that “the VLA outputs low-level actions directly”: the VLA outputs only natural-language mid-level actions and low-level control is separated. It simultaneously overcomes the waypoint-predictor dependence of (2) and the “solve only the high level and ignore control” stance of (3), and is the first end-to-end legged-VLN demonstration to connect high-level reasoning and actual leg joint control in one pipeline. Since then, NaVILA has been cited as a reference point in the 2025 navigation foundation model discussion, including Uni-NaVid, StreamVLN, and NavFoM, and “language = mid-level action” + “VLA/locomotion separation” is establishing itself as a design pattern.

Problem and Motivation

VLN, where a human commands in natural language and the robot follows through an unknown environment without a map, is attractive from both the interface and generalization perspectives. Extending it to legged robots is highly valuable because they can reach narrow, cluttered spaces and unstructured terrain such as rocks, holes, and stairs, but the gap between “language → leg joint torques” is very large.

If the VLM directly outputs quantized low-level actions as in existing end-to-end VLAs, two problems arise. First, the prior knowledge of the LLM/VLM trained in natural language is not exploited. Second, the model overfits to the action space of a specific robot, making platform transfer and use of real-world data difficult. The authors’ starting question was “is there a better way to represent actions beyond low-level commands?”

Key Ideas and Method

In one sentence: “the VLA outputs only natural-language mid-level actions such as turn right 30°, forward 75cm, and a separate visual locomotion RL policy executes them as joint control.” By analogy, the VLA is the navigator in the passenger seat who looks at the map and says “turn right at the next intersection,” and the locomotion policy is the driver who hears that, operates the wheel and pedals, and avoids obstacles ahead on their own. Even if the vehicle (robot) changes, the navigator does not need to relearn how to drive; only the driver is swapped.

High level: VLA (VLM fine-tuning)

  • The backbone is NVIDIA’s VILA (an image-based VLM strong at multi-image reasoning). An image VLM was chosen over a video encoder because of the generalization advantage that comes from large-scale image-text data.
  • Both input and output are kept in the language domain and special tokens are minimized to make the most of the LLM’s reasoning ability. The input is the current observation frame plus past frames (memory), distinguished by text cues such as current observation: and a video of historical observations:. The latest frame t is always the current observation, and the rest are uniformly sampled from up to t-1 (experiments with 8-64 frames).
  • The output is a continuous-valued mid-level action of the form {forward d cm, turn left/right θ°, stop}. At inference a regex parser extracts the action and its arguments, and the paper reports a 100% match rate across all experiments.

Data pipeline (a hidden contribution)

  • From 2K first-person YouTube tour videos, 20K trajectories are created by entropy-based trajectory sampling (borrowed from YouTube-VLN), metric camera poses are estimated with MASt3R to extract per-step actions, and instructions are generated with VLM captions and LLM rephrasing. This is the first case of directly training continuous-environment navigation from human videos (prior work used them only as pretraining aids).
  • The SFT data blend has four parts: (1) real-video navigation data, (2) simulated navigation data (R2R-CE and RxR-CE converted via Habitat shortest-path following, with consecutive actions merged and stop labels rebalanced), (3) auxiliary navigation data (EnvDrop augmented instructions, trajectory summarization, ScanQA 3D QA), and (4) general VQA (to preserve general capabilities).

Low level: visual locomotion policy (Unitree Go2)

  • The Go2 has an L1 LiDAR (15 Hz, 360°×90°) and 18 DoF (base 6 + legs 3×4), and the policy outputs target positions for the 12 leg joints.
  • The LiDAR point cloud is converted into a 2.5D height map (minimum value per voxel, max filter over the last 5 frames). LiDAR was chosen as the primary sensor to sense safely even where RGB/depth fail, such as under strong sunlight and with transparent objects like glass.
  • Training is single-stage PPO (no two-stage teacher-student distillation). Using ray-casting in Isaac Sim/Isaac Lab, it achieves a throughput of over 60K FPS on a single RTX 4090. The critic observes privileged information (terrain height scan) while the actor receives only observations obtainable in the real world. The VLM’s action vocabulary is mapped to fixed command velocities {0.5 m/s, ±π/6 rad/s, 0} and executed for the specified duration.

Dual frequency

The heavy, slow VLA issues high-level commands at a low frequency (about 1 FPS), while the locomotion policy avoids obstacles in real time. Separating the two timescales secures robustness and real-time performance at the same time.

Experiments and Results

Setup. The platforms are the Unitree Go2 (primary) and the Unitree H1 and Booster T1 humanoids (the same VLA transferred without retraining). The simulators are Habitat (high-level evaluation) and the new VLN-CE-Isaac (Isaac Sim, reflecting legged-robot joints and physics; 1,077 navigable trajectories selected from the 1,839 R2R Val-Unseen trajectories). The benchmarks are R2R-CE and RxR-CE (NE/OS/SR/SPL/nDTW), ScanQA (CIDEr etc.), VLN-CE-Isaac, and real-world 25 instructions × 3 trials. The baselines are ETPNav, BEVBert, Reborn (using waypoint predictors), NaVid (single-view SOTA), ROA (locomotion distillation), and GPT-4o (real world).

Key results.

Item Value
R2R-CE Val-Unseen NE 5.22 / OS 62.5 / SR 54.0 / SPL 49.0 (single RGB)
RxR-CE Val-Unseen (trained on R2R only, zero-shot) SR 34.3 vs NaVid 23.8 (+10 pp)
ScanQA CIDEr 102.7 (64 frames), +20 points over NaviLLM, on par with 3D-input LEO
VLN-CE-Isaac vision policy over blind: Go2 +14 pp, H1 +21 pp SR; versus Oracle low-level policy: Go2 -15 pp, H1 -27 pp
Low-level policy collision rate 3.09 → 0.81 versus ROA
Real world (Go2, 25 instructions) average SR 88% (compound instructions 75%), far ahead of GPT-4o
  • Matching or exceeding models that use panoramas, odometry, and waypoint predictors with single-view RGB alone is a first in this setting. In the table, methods using waypoint predictors are separated with a distinct marker, and NaVILA’s claim takes the restrained form of “strongest among methods without a predictor, and on par with those using one.”
  • Adding human (YouTube) videos improves OS, SR, and SPL by about 5 pp, and in the real world outdoor generalization and SR across all environments also rise (ablation).
  • For memory frames, 8 frames cover most instruction horizons on R2R-CE, and 8 frames are also used in the real world due to latency constraints.
  • Quantization: converting to W4A16 with AWQ reduces latency from 594 to 368 ms and memory from 18.5 to 8.6 GB while navigation performance is almost preserved. This suggests the feasibility of onboard deployment.

Points to keep in mind when reading. The “17% improvement” the paper highlights is a figure within the group not using waypoint predictors, so there is a risk of overinterpretation if only the headline is read. The RxR Val-Unseen SR is in the 34% range in absolute terms, so real-world difficulty remains high. Also, the claim that “language representation is better than quantized commands” is demonstrated indirectly through SOTA comparison rather than a same-condition A/B comparison.

Conclusion and Significance

NaVILA broke the convention that “the VLA must output low-level actions directly,” and with a two-level design in which the VLA outputs only natural-language mid-level actions and low-level control is separated, it connected high-level reasoning and actual leg joint control using only a single RGB camera. The contributions to the field are threefold: (1) the first end-to-end demonstration of legged-robot VLN, (2) the reusable design pattern of the “language = mid-level action” interface, and (3) the VLN-CE-Isaac benchmark reflecting legged-robot physics.

From a robotics practice perspective, the takeaways are as follows.

  • Where it can be used. When attaching “natural-language command → movement” to autonomous mobile-robot navigation in multi-floor indoor buildings or driving on unstructured outdoor terrain, the architecture itself, separating a high-level VLA from a low-level controller, is directly worth referencing. The fact that only the locomotion policy needs to be replaced, without retraining the VLA, when the platform changes is particularly advantageous in settings that operate several robots together.
  • If you are building a quadruped simulation testbed, the VLN-CE-Isaac benchmark (MIT license) and the single-stage PPO + LiDAR height map configuration are a starting point. The figure of 60K FPS on a single RTX 4090 is a training scale reproducible even in a small lab.
  • Reproduction difficulty is moderate to high. Inference and evaluation are immediately possible with the released HuggingFace checkpoints (a8cheng/navila-llama3-8b-8f etc.) and code. On the other hand, the training code is listed as to-be-released (TODO) in the README, the YouTube data provides only video IDs and annotations due to copyright, and SFT takes 18 hours on 4 nodes × 8 A100s. Real-robot deployment code is not clearly present in the repo, so the locomotion training and deployment stack must be implemented in-house. This is the biggest barrier.
  • Limitations. Open issues include the absence of error recovery (no self-correction after deviating from the path), the heavy compute of the image VLM (about 1 FPS), the effect of angle and distance quantization in natural-language actions on narrow corridors and precise maneuvering (H1 performs lower than Go2), and the 8-frame memory limit on long-horizon compound instructions.

References

  • Paper: arXiv 2412.04453 · Project page
  • Code: AnjieCheng/NaVILA · NaVILA-Bench
  • NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation (RSS 2024)
  • VILA: On Pre-training for Visual Language Models (CVPR 2024)
  • Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (ECCV 2020)
  • MASt3R: Grounding Image Matching in 3D (ECCV 2024)
  • AWQ: Activation-aware Weight Quantization (MLSys 2024)