π0.5: a Vision-Language-Action Model with Open-World Generalization (CoRL 2025)

Created on June 11, 2026

2026   ·   vla   foundation-model   navigation   ·   paper-review

Item Details
Authors Kevin Black, Noah Brown, Danny Driess, Chelsea Finn, Karol Hausman, Brian Ichter, Sergey Levine, Karl Pertsch, Lucy Xiaoyang Shi, Quan Vuong, and many others (Physical Intelligence)
Venue arXiv 2504.16054 (2025-04-22), CoRL 2025 Oral
Links arXiv · Code (openpi, Apache-2.0)

One-line Summary

π0.5 is the first VLA to combine co-training on heterogeneous data sources with hierarchical reasoning (high-level subtask text → low-level actions) and perform 10-15 minute long-horizon multi-stage mobile manipulation (kitchen and bedroom cleaning) end-to-end in new homes that never appeared in training. It is the direct successor of π0 and shifts the center of gravity from “demonstrating dexterous capability” to “open-world generalization.”

The lineage runs RT-1/RT-2 → OpenVLA → π0 (PaliGemma backbone + flow matching action expert) → π0.5 → π0-FAST → Knowledge Insulation (KI) → π0.6. Where π0 showed dexterity, π0.5 targets generalization. π0 demonstrated dexterous tasks such as laundry folding but was evaluated in environments close to the training distribution, whereas π0.5 set the goal of “working out of the box in a new home.”

The four key changes from π0 are:

  1. Data diversity. The mobile manipulation data that is the evaluation target is only about 400 hours, 2.4% of the total. The remaining 97.6% is indirect knowledge from other robots, the web, high-level labels, and so on. It directly tests the proposition that “scale alone is insufficient for generalization; knowledge transfer across multiple levels of abstraction is required.”
  2. Hierarchical reasoning. The same model first predicts high-level subtask text (“pick up the plate”), then generates a low-level action chunk conditioned on that text. It evolves from π0’s flat policy into a chain-of-thought-style two stages.
  3. Hybrid action representation. Pretraining uses FAST discrete tokens (fast training), and post-training uses a flow matching continuous action expert (real-time inference and precision). The two representations are separated by the attention mask.
  4. Verbal Instruction modality. “Verbal demonstrations,” in which a person coaches the robot step by step in real-time language, are used to train the high-level policy.

Contemporary competitors include GR00T N1 (NVIDIA), AgiBot World, and RT-H. π0.5’s differentiator is the evaluation difficulty of “long-horizon tasks in a new home.” Among related prior work, SayCan is a two-model structure where an LLM selects high-level steps and a separate low-level policy executes them; π0.5 unifies this into one model. Embodied Chain-of-Thought runs high-level reasoning at the same frequency as low-level control, whereas π0.5 runs the high level at low frequency for efficiency.

Problem and Motivation

Open-world generalization is the biggest challenge of physical intelligence. Robots become useful only when they can handle diverse situations they have never seen outside the lab. Taking cleaning a new kitchen as an example, generalization is needed at several levels.

  • Behaviors that generalize directly when sufficiently represented in the data (picking up a plate)
  • Modifying existing skills in new ways or new orders
  • Semantic understanding of the scene (which drawer to open, which object on the counter is the dish rack)

Simple scaling struggles to fill the second and third. Humans synthesize not only direct experience but what they hear from others, what they read in books, and insights from other tasks. The paper’s hypothesis is that robot learning systems must likewise transfer knowledge from diverse information sources. Above all, for long-horizon, complex tasks like kitchen cleaning, brute-force collection of robot data covering every plausible scenario is practically impossible, so the claim is that co-training on heterogeneous sources is the only way.

Key Ideas and Method

One Model, Two Roles (Hierarchical Policy)

The policy distribution is factored as follows.

\[\pi_\theta(a, \hat{\ell} \mid o, \ell) = \pi_\theta(\hat{\ell} \mid o, \ell) \cdot \pi_\theta(a \mid o, \hat{\ell})\]
  • High level: from the observation \(o\) and the overall command \(\ell\) (“clean the kitchen”), predict the subtask text \(\hat{\ell}\) (“pick up the plate”). Relevant bounding boxes are predicted first as well.
  • Low level: from the observation \(o\) and the subtask \(\hat{\ell}\), generate the action chunk \(a\). Importantly, the actions depend only on the subtask \(\hat{\ell}\), not on the original command \(\ell\).

By analogy, one person “writes a one-line note of the next thing to do in their head” (high level) and moves their hands by looking at that note (low level). Same brain, two modes. The high level runs at low frequency, the low level at high frequency.

Hybrid Discrete + Continuous Actions

Discrete tokens (FAST) train fast but require expensive autoregressive decoding at inference, making them unsuitable for real-time control. Flow matching continuous actions infer fast but train slowly. π0.5 learns both representations but divides their roles by stage.

Stage Action representation Loss weight Steps
Pretraining FAST discrete tokens (next-token like a standard VLM) \(\alpha = 0\) 280k
Post-training Add flow matching action expert (300M, randomly initialized) \(\alpha = 10\) 80k

The combined loss is the cross-entropy over text tokens (including FAST-encoded action tokens) plus a flow matching term. For noise \(\omega \sim \mathcal{N}(0, I)\) it forms \(a^{\tau, \omega} = \tau a + (1 - \tau)\omega\) and trains the model to predict the vector field \(\omega - a\).

\[\mathcal{L} = \mathcal{L}_{\text{CE}}(\text{text, FAST tokens}) + \alpha \left\| (\omega - a) - f^a_\theta(a^{\tau,\omega}, o, \hat{\ell}) \right\|^2\]

The two action representations are separated by the attention mask so they cannot see each other, blocking information leakage. At inference, the text \(\hat{\ell}\) is decoded autoregressively, then continuous actions are generated with 10 denoising steps conditioned on it. By analogy, studying for an exam memorizes concepts quickly with multiple choice, while the real exam is answered precisely in essay form.

Heterogeneous Co-training Data

Abbrev. Source Role
MM Diverse Mobile Manipulator: about 100 homes, about 400 hours Most directly related to evaluation
ME Multi-Environment non-mobile robots Light and easy to transport, so collected in more homes (different embodiment)
CE Cross-Embodiment lab data Single- and dual-arm, static/mobile, including OXE open-source data
HL High-Level subtask prediction Manual subtask labels + bounding box annotations on robot data
WD Multimodal Web Data Captioning (CapsFusion, COCO), VQA (Cambrian-7M, PixMo, VQAv2), object localization (additional indoor-scene collection)
VI Verbal Instruction (post-training only) Experts “teleoperate” the low-level policy with real-time language, demonstrating good high-level outputs

VI is only about 11% of all high-level mobile examples, but it is decisive. All action data is trained to predict target joint and end-effector poses; actions are normalized to \([-1, 1]\) using the 1% and 99% quantiles of each dimension, and lower-dimensional robots are zero-padded to the largest action space.

Backbone and Architecture

  • Backbone: PaliGemma VLM = SigLIP (400M) vision + Gemma (2B). Width 2048, depth 18, MLP 16384, 18 heads.
  • Action expert: 300M (width 1024, MLP 4096). Similar to MoE, only action tokens are processed with separate weights. The action horizon is 50.
  • Timestep injection: unlike π0, \(\tau\) is not fused into the noisy action but injected into each layer through a separate MLP and adaptive RMSNorm. The flow timestep is sampled from a Beta distribution that emphasizes the low range.
  • Attention: images, prompt, and proprioception use a bidirectional prefix mask. FAST tokens attend autoregressively to the prefix and previous action tokens. Action expert embeddings attend only to the prefix and to each other, and do not see FAST tokens. That is, information flows one way from the VLM to the action expert. This design is the precursor of the later Knowledge Insulation and π0.6.

Experiments and Results

Robot system. Two kinds of mobile manipulators. 6-DoF dual arms + parallel grippers, a holonomic wheeled base (3 DoF), a torso lift (1-2 DoF), and 4 cameras (front/rear + both wrists). State/action is 18-19 DoF. π0.5 directly commands arm, gripper, and lift target poses and base velocities at 50 Hz (action chunking); there is only a simple PD tracker with no trajectory planning or collision detection. Both navigation and manipulation are end-to-end.

Evaluation environments. All experiments were run only in new environments absent from training. Quantitative comparisons use reproducible mock rooms; the final evaluation uses 3 real homes.

The three key results are as follows.

1. Generalization to new homes. In 3 real homes entirely absent from training, it consistently succeeded at multi-stage kitchen and bedroom cleaning tasks (putting dishes in the sink, tidying drawers, laundry basket, etc.) lasting 2-5 minutes, some 10-15 minutes. The authors claim this is the first case of an end-to-end learned system performing long-horizon dexterous tasks at this level in a new home.

2. Indirect data is essential. Removing either ME or CE (other robots) sharply degrades performance, and removing both is worse. Cross-embodiment transfer is thus key. Removing WD (web) made no significant difference on the 4 mock tasks, but language following on unseen-category (OOD) objects deteriorated substantially. Removing VI caused the high-level policy to collapse.

3. Scaling with the number of environments. Increasing the number of training environments through 3, 12, 22, 53, 82, and 104 monotonically increased average performance on the 4 tasks. The 104-environment model approached the performance of a control model that included the test home directly in training, despite having zero test-home data. A baseline using only mobile data without co-training was far worse. Language following rate and success rate also improved with the number of environments, with in-distribution rising faster.

Details.

  • Versus π0. π0.5 significantly outperformed both the original π0 and the improved π0-FAST+Flow (joint diffusion + FAST trained on action data only, without HL and WD). The result held even when π0 was trained further to 300k steps, reconfirming the FAST paper’s conclusion that FAST token training is more compute-efficient than pure diffusion.
  • High-level reasoning ablation. Full π0.5 was the best, even higher than the human oracle (human HL). Surprisingly, second was implicit HL (no runtime high-level reasoning, HL data included only in training). This means much of the gain from explicit reasoning comes simply from mixing in HL data. No HL (data removed as well) dropped substantially. Using zero-shot GPT-4 as the high-level policy was the worst, showing the importance of adapting the VLM on robot data.
  • Per-task breakdown. “Items in Drawer” collapsed when web data was removed (semantic knowledge matters for generalization to less-seen objects), while “Laundry Basket” and “Dishes in Sink” were less sensitive to the high-level policy choice (shorter horizon or less semantic reasoning needed).
  • The project blog reports language following/success figures of about 83% in-distribution and about 94% OOD when the number of environments is sufficient.

Claims vs reality. “The first demonstration of long-horizon dexterous manipulation in a new home” mostly holds given that evaluation is restricted to new environments, although the quantitative evaluation is mock-room centric and the real-home sample is small (3 homes, 10 trials). “Co-training is the essence of generalization” is highly convincing because the ablations isolate each source’s contribution. However, “why implicit HL is so strong” is only partially explained, and “104 environments ≈ control” is limited to mock environments and 4 tasks, so over-interpretation should be avoided.

Conclusion and Significance

π0.5 is the turning point that shifted the center of gravity of VLA research from “capability demonstration” to “open-world generalization.” It demonstrated that designing a heterogeneous knowledge transfer recipe, rather than blindly increasing robot data, is the key to generalization. The “same-model hierarchical reasoning” and “discrete pretraining + continuous post-training” hybrid became standard patterns afterwards, and the attention separation design leads directly to π0.6 via Knowledge Insulation (blocking the action expert’s gradient from backpropagating into the VLM backbone to preserve pretrained knowledge).

Its applicability and limitations from a robotics practice viewpoint are as follows.

  • Fine-tuning and inference are accessible. openpi (JAX/Flax + PyTorch) releases the pi05_base (for fine-tuning), pi05_libero (LIBERO), and pi05_droid (DROID, with KI) checkpoints under Apache-2.0. Inference needs 8 GB+, LoRA fine-tuning 22.5 GB+ (RTX 4090), and full fine-tuning 70 GB+ (A100/H100). Reproducing the LIBERO/DROID benchmarks is of medium difficulty.
  • Reproducing the paper’s results is practically impossible. New-home generalization depends on two kinds of in-house mobile manipulators, about 400 hours of home data, and 100 environments, all private. Reproduction difficulty is high.
  • Implications for autonomous mobile robot navigation in multi-floor indoor buildings. Handling both navigation and manipulation end-to-end with no trajectory planning or collision detection is interesting, but the paper explicitly notes the limitation that context and memory are thin, so it is weak at partially observable tasks such as moving between rooms or remembering object locations. Scenarios requiring floor changes or long routes would need separate memory and planning modules.
  • It still makes mistakes. Reported failures include unfamiliar drawer handles, cabinets that are physically hard to open, partial observation (the arm occluding a spill that needs wiping), and distracted high-level reasoning (repeatedly opening and closing a drawer). It handles only simple prompts; complex preferences and instructions would need richer annotations.
  • Open questions. The mechanism by which implicit HL approaches explicit reasoning, extension to longer horizons spanning multiple rooms, and ways of injecting human knowledge other than VI remain unexplored.

References

  • Paper: arXiv 2504.16054
  • Code: openpi (Apache-2.0)
  • π0: A Vision-Language-Action Flow Model for General Robot Control (Black et al., 2024) — direct predecessor
  • FAST: Efficient Action Tokenization for Vision-Language-Action Models (Pertsch et al., 2025) — discrete action tokenizer for pretraining
  • PaliGemma: A versatile 3B VLM for transfer (Beyer et al., 2024) — vision-language backbone
  • Do As I Can, Not As I Say / SayCan (Ahn et al., 2022) — representative prior work on two-model hierarchies
  • RT-H: Action Hierarchies Using Language (Belkhale et al., 2024) — prior work on language-based action hierarchies