π0: A Vision-Language-Action Flow Model for General Robot Control (arXiv 2024)

Created on June 11, 2026

2026   ·   vla   foundation-model   ·   paper-review

Item Details
Authors Kevin Black, Noah Brown, Danny Driess, Michael Equi, Chelsea Finn, Karol Hausman, Brian Ichter, Liyiming Ke, Sergey Levine, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Quan Vuong, et al. (Physical Intelligence, San Francisco)
Venue arXiv 2410.24164, preprint 2024-10-31 (no separate conference publication confirmed; released together with the company blog)
Links arXiv · Blog · Code (openpi, Apache-2.0)

One-line Summary

π0 (pi-zero) is a robot foundation model that attaches a separate action expert (about 0.3B) generating continuous actions via flow matching on top of a pretrained VLM (PaliGemma 3B). After pretraining on cross-embodiment data from 7 robot configurations, 68 tasks, and about 10,000 hours, then post-training on high-quality task-specific data, it performs long-horizon dexterous manipulation lasting tens of minutes, such as laundry folding, table bussing, and box assembly, at 50 Hz. It is Physical Intelligence’s first flagship model and the starting point of the lineage that continues with π0.5 and π*0.6.

The background of π0 can be summarized as the meeting point of four streams.

  1. Autoregressive token VLAs. RT-2 and OpenVLA fine-tune a pretrained VLM on robot demonstrations but output actions discretized like text tokens. This is the core stream that transferred internet knowledge to robots, but because discrete tokens are decoded autoregressively, they cannot support action chunking and decoding is slow, making them weak at high-frequency dexterous manipulation.
  2. Diffusion-based action generation. Diffusion Policy and Octo generate actions with diffusion models and represent multimodal, continuous distributions well. However, most lack a VLM backbone and cannot bring in internet-scale semantic knowledge.
  3. Action chunking. ACT (ALOHA) predicts a chunk of H action steps at once, enabling bimanual dexterous manipulation, but was limited to single tasks with tens to hundreds of demonstrations.
  4. Per-token loss mixing. Transfusion trained next-token prediction and diffusion together within one transformer, and Playground v3 gave separate weights to diffusion tokens. π0 brings this idea to VLAs and handles “cross-entropy for images and text, flow matching for actions” in a single transformer.

π0 unifies these four streams. It bundles VLM pretraining, flow matching, action chunking, cross-embodiment, and an LLM-style pre/post-training recipe, and the authors claim it is “the first flow matching VLA that outputs high-frequency action chunks.”

The subsequent lineage is as follows. π0-FAST is a version with the FAST tokenizer that improves autoregressive tokenization. π0.5 strengthens open-world generalization and introduces Knowledge Insulation (a training method that uses stop-gradient so the action expert does not corrupt backbone representations). π*0.6 (2025-11) extends the model to learn from its own successes and failures with an offline RL recipe called RECAP. In other words, the π0 line is moving from imitation learning toward experience-based RL.

Problem and Motivation

The three major bottlenecks in robot learning are data scarcity, generalization, and robustness. π0 proposes solving them with the success formula of LLMs/VLMs: “pretrain on diverse large-scale data, then align on narrow high-quality data.” Transferring this to robots, however, requires three things at once.

  • Scale. Pretraining gains do not appear at small scale.
  • Architecture. It must absorb diverse data while still representing dexterous and subtle physical behavior.
  • Training recipe. The authors emphasize this as the most important.

As an intuitive analogy, a model trained only on high-quality data never learns to recover from mistakes and is brittle, while a model trained only on low-quality pretraining data cannot act fluently and efficiently. Combining the two yields a model that “acts as close to high quality as possible, but has a recovery repertoire when it errs.”

Key Ideas and Method

VLM Backbone: The Import Port for Internet Knowledge

The backbone is initialized from PaliGemma 3B (SigLIP image encoder + Gemma 2B language model). Images and language are placed in the same embedding space via late fusion, inheriting semantic reasoning and language understanding as-is. By analogy, it is attaching hands and feet to “a brain that already knows how to read the world.” The paper states that PaliGemma was chosen only for convenience and its small size (favorable for real-time control), and that the framework itself is compatible with any pretrained VLM.

Action Expert: The Continuous Action Output Module

This is a separate set of weights (about 0.3B) that handles inputs the VLM never had (proprioceptive state \(q_t\)) and outputs (action chunk \(A_t\)). The key trick is to view a single transformer as two experts (a 2-expert MoE). Image and text tokens are routed to the large VLM expert, state and action tokens to the small action expert, and the two experts interact only in the self-attention layers. The action expert is shrunk to width 1024 and MLP 4096 (the backbone Gemma 2B has width 2048 and MLP 16384) to preserve inference speed even with multiple forward passes during flow integration.

Flow Matching: How the Action Distribution Is Drawn

The action expert generates a continuous action chunk \(A_t = [a_t, \ldots, a_{t+H-1}]\) (\(H = 50\)) with conditional flow matching. The training objective is as follows.

\[L^\tau(\theta) = \mathbb{E}_{p(A_t|o_t),\, q(A_t^\tau|A_t)} \left\| v_\theta(A_t^\tau, o_t) - u(A_t^\tau | A_t) \right\|^2\]
The probability path is a simple linear-Gaussian (OT) path $$q(A^\tau A) = \mathcal{N}(\tau A, (1-\tau) I)\(. From noise\)\epsilon \sim \mathcal{N}(0, I)\(it forms\)A^\tau = \tau A + (1-\tau)\epsilon\(and trains the network output\)v_\theta\(to match the vector field\)u = A - \epsilon\(. At inference, it integrates with forward Euler from\)\tau = 0\((pure noise) to\)\tau = 1$$ (actions).
\[A^{\tau + \delta} = A^\tau + \delta \, v_\theta(A^\tau, o_t), \quad \delta = 0.1\]

That is, 10 steps suffice. It is a sibling of diffusion, but the straight path converges in fewer steps, and it suits dexterous, multimodal, high-precision action representation, enabling 50 Hz control. Instead of cross-entropy on discrete tokens, the flow matching loss is applied only to action tokens.

Blockwise Causal Attention and KV Caching

The attention mask is split into three blocks: [image + language] / [state \(q\)] / [noisy actions \(A^\tau\)]. Attention within each block is bidirectional, and later blocks are not visible to earlier ones. There are two effects. First, the keys/values of the observation (prefix) are computed once and cached, and only the action tokens are recomputed across the 10 flow-integration iterations, making inference efficient. Second, it prevents the inputs seen during VLM pretraining from attending to the newly added inputs, minimizing distribution shift.

Timestep Sampling Design

During training, the flow timestep \(\tau\) is sampled not uniformly but from a beta distribution that emphasizes the low-timestep (high-noise) region.

\[p(\tau) = \mathrm{Beta}\left(\frac{s - \tau}{s};\, 1.5,\, 1\right), \quad s = 0.999\]
This design stems from the hypothesis that, unlike image synthesis, in robotics the observation strongly constrains the action, so “mean action prediction” $$\mathbb{E}[A o]$$ is harder. It trains more on the high-noise region.

Pre/Post-training Recipe

Stage Data Purpose
Pretraining OXE + Bridge v2 + DROID (9.1% of total) + in-house 903M steps (single-arm 106M, bimanual 797M, 68 tasks) Acquire broad physical skills and recovery behaviors
Post-training 5 to 100+ hours of high-quality curated data per task Align fluent, consistent strategies

The pretraining mix weights each task-robot combination by \(n^{0.43}\) for its sample count \(n\), down-weighting over-represented combinations. State and action vectors are zero-padded to the largest robot dimension (18-D: two 6-DoF arms, two grippers, a mobile base, a vertical torso), and missing image slots are masked. Language labels combine task names with segment annotations about 2 s long.

For tasks requiring semantic reasoning such as table bussing, a high-level VLM policy decomposes the task SayCan-style into intermediate language commands such as “pick up the napkin” and “put it in the trash,” and π0 executes them.

Experiments and Results

System configuration. PaliGemma 3B + action expert 0.3B, 3.3B in total. To isolate the effect of VLM pretraining, a non-VLM baseline π0-small (470M, DistilBERT + ViT + DiT) was also trained. The in-house data covers 7 configurations: UR5e, Bimanual UR5e, Franka, Bimanual Trossen (ALOHA), Bimanual ARX and AgileX, Mobile Trossen and ARX, and Mobile Fibocom. The main model was trained for 700k steps, and a 160k-step version was also evaluated for compute parity with the baselines.

Inference cost (RTX 4090). Image encoding 14 ms + observation forward 32 ms + 10 flow steps 27 ms, for 73 ms onboard (86 ms offboard). Action chunks are executed open-loop; temporal ensembling actually hurt performance and was discarded. 20 Hz robots re-infer every 0.8 s (16 actions), 50 Hz robots every 0.5 s (25 actions).

Evaluation groups. (1) Direct prompting without post-training (shirt folding, bussing easy/hard, grocery bagging, toast); (2) language instruction following (flat / human intermediate commands / high-level VLM); (3) fine-tuning on new dexterous tasks (1/5/10 hours of data); (4) highly complex multi-stage tasks (laundry folding, box building, packing eggs, etc.). The comparison set is OpenVLA (7B), Octo (93M), ACT, and Diffusion Policy.

The three key results are as follows.

1. Dominant out of the box. π0 (700k) was the best on every task and nearly perfect on shirt folding and easy bussing. Even the 160k parity version beat all baselines, and π0-small without a VLM beat OpenVLA and Octo. OpenVLA does not support action chunks and collapsed on high-frequency dexterous tasks.

2. VLM pretraining lifts language following. In language instruction following, π0 far outperformed π0-small and benefited from both human intermediate commands (π0-human) and high-level VLM guidance (π0-HL). π0-small has limited language ability and gained nothing from high-level guidance.

3. Very long-horizon dexterous tasks achieved. On 5-20 minute tasks such as laundry folding (taking out, flattening, and folding from an arbitrarily crumpled state), mobile laundry, dryer unloading, table bussing with unseen objects, box building, packing eggs, and to-go boxes, the full recipe (pretraining + fine-tuning) achieved over 50% of the maximum score on every task. The authors consider these the longest dexterous tasks in the end-to-end robot learning literature.

Fine-tuning details. In fine-tuning on new tasks, π0 was generally superior to ACT, Diffusion Policy, OpenVLA, and Octo. Interestingly, for the existing models training from scratch was better than from pretraining, evidence that pretraining transfer is hard with prior approaches. For π0, pretraining showed up to 2x improvement over scratch, and the pretraining gain was larger with less data and harder tasks.

Claims vs reality. The claim of “the first flow matching VLA” is accepted as novel as an integrative contribution, but the authors themselves state that the individual components build on prior work. The “largest pretraining mix ever” appears factually true in scale, but the in-house data is private, so external verification is impossible. The “new SOTA dexterous manipulation” is backed by strong real-robot demonstrations, but since few existing models can run tasks of this difficulty, some comparisons are limited to ablations, and the sample of 10 trials per task is small.

Conclusion and Significance

π0 is the turning point that moved VLAs from “discrete-token autoregression” to “VLM + continuous flow matching action expert.” Since then nearly every dexterous manipulation VLA has followed the blueprint of action expert + flow/diffusion + action chunking. The more important message is the recipe rather than the architecture. It presented empirical evidence that LLM-style pre/post-training works for robots too, with pretraining providing generality and resilience and post-training providing fluency. With the release of openpi, it became the standard starting point for reproduction and extension.

Its applicability and limitations from a robotics practice viewpoint are as follows.

  • Fine-tuning the public base checkpoints is the realistic path. openpi (Apache-2.0, JAX + PyTorch support since 2025-09) provides π0, π0-FAST, and π0.5 base checkpoints plus ALOHA/DROID/LIBERO fine-tuned checkpoints. Inference needs 8 GB+ (RTX 4090 class), LoRA fine-tuning 22.5 GB+, and full fine-tuning 70 GB+ (A100/H100). At 73 ms, inference is light on a single 4090.
  • Reproducing pretraining from scratch is practically impossible. The in-house 10,000-hour data is private, and 7 kinds of real robots plus large-scale GPUs are required. Reproduction difficulty is medium for fine-tuning and inference, high for the full paper.
  • There is no principle for composing the pretraining mix. There is no explanation of which data to mix and in what proportion (acknowledged by the authors), and it is impossible to predict how much data is needed to reach near-perfect. Some tasks fall short in reliability.
  • Transfer to other domains is unconfirmed. Whether positive transfer is universal to very different domains such as autonomous driving, navigation, and legged locomotion has not been verified. Applying it to autonomous mobile robot navigation in multi-floor indoor buildings or a quadruped simulation testbed would require separate validation.
  • Responsiveness of open-loop chunk execution. Since temporal ensembling hurt performance and was discarded, concerns about reduced responsiveness and error accumulation over the long term remain.

References

  • Paper: arXiv 2410.24164 · Physical Intelligence blog
  • Code: openpi (Apache-2.0)
  • PaliGemma: A versatile 3B VLM for transfer (Beyer et al., 2024) — the VLM backbone of π0
  • Flow Matching for Generative Modeling (Lipman et al., 2022) — the original flow matching paper
  • Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (Zhou et al., 2024) — direct inspiration for per-token loss mixing
  • Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware / ACT (Zhao et al., 2023) — the basis of action chunking
  • Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (Chi et al., 2023) — key comparison baseline