π*0.6: a VLA That Learns From Experience (arXiv 2025)

Created on June 11, 2026

2026   ·   vla   rl   foundation-model   ·   paper-review

Item Details
Authors Ali Amin, Ashwin Balakrishna, Kevin Black, Danny Driess, Chelsea Finn, Karol Hausman, Brian Ichter, Liyiming Ke, Sergey Levine, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Jost Tobias Springenberg, Kyle Stachowicz, Quan Vuong, Zhiyuan Zhou, et al. (Physical Intelligence, about 56 authors in total)
Venue arXiv 2511.14759 (v2, 2025-11-19, cs.LG preprint). No separate conference publication confirmed; released together with the company blog
Links arXiv · Blog · Code not released (as of 2026-06, openpi contains only π0, π0-FAST, and π0.5; no RECAP or π*0.6 checkpoints confirmed)

One-line Summary

π*0.6 takes a VLA trained only on demonstrations (π0.6) and applies reinforcement learning from its own experience (autonomous rollouts), human intervention corrections, and sparse rewards, achieving over 90% success and more than 2x throughput on 5-15 minute very long-horizon dexterous manipulation such as laundry folding, espresso making, and box assembly. The core is an iterative offline RL recipe called RECAP (RL with Experience and Corrections via Advantage-conditioned Policies). Instead of policy gradients (PPO), it conditions the policy by inserting an advantage indicator, “is this action better than average,” into the prompt as text (“Advantage: positive/negative”), and trains the entire flow-matching VLA end-to-end.

The line from π0 → π0.5 → π0.6 → π*0.6 can be summarized as follows.

  1. π0 (2024). The first flagship VLA, imitation-learned from cross-embodiment demonstrations with a VLM (PaliGemma 3B) + flow-matching action expert. It showed 50 Hz dexterous manipulation but was confined by the limits of imitation learning (compounding error, demonstration quality ceiling).
  2. π0.5 (CoRL 2025). The successor for open-world generalization. Web co-training, subtask text reasoning (high level), and the Knowledge Insulation (KI) training recipe. Still supervised (imitation) based.
  3. π0.6 (2025, model card). An incremental improvement of π0.5. More robot platform data, backbone upgraded to Gemma 3 4B, action expert enlarged to 860M. This is the base VLA of this paper.
  4. π*0.6 (this paper, 2025). A version of π0.6 modified to enable RL by adding advantage conditioning capability (a binarized advantage indicator input). In other words, π0.6 + RECAP.

By analogy, π0 through π0.6 are “a student who studied from the textbook (demonstrations),” and π*0.6 is “a student who takes the exam themselves (autonomous rollouts), receives a coach’s corrections (intervention), and reviews to improve.” The paper’s first sentence is also “Practice makes perfect.”

Methodologically, reward-conditioned policies (Kumar, Peng, Levine 2019) and the Decision Transformer family are the direct ancestors of advantage conditioning, and CFGRL (Frans et al. 2025) provides the theoretical basis that “diffusion guidance is a controllable policy improvement operator.” Expert intervention follows human-gated DAgger (Kelly et al. 2019), and the critic uses distributional RL (Bellemare et al. 2017). Many recent studies have applied PPO and its variants to VLA fine-tuning, but scaling to real-world RL has been difficult, and residual policies, action-head fine-tuning, and optimization in diffusion noise space mostly used discrete actions or simple Gaussian distributions. This paper’s differentiator is training the entire expressive flow-matching VLA end-to-end with iterative offline RL.

Problem and Motivation

Imitation learning policies have three fundamental limits. (a) They cannot do better than the demonstrations, (b) they cannot fix the mistakes they actually make during deployment, and (c) they cannot exceed the speed and robustness of human teleoperation (compounding error).

RL formalizes “mastery through practice,” but applying real-world RL to large VLAs was an unsolved problem. It requires stable, scalable RL for large models, integration of heterogeneous off-policy data from multiple policies (demonstrations, autonomous rollouts, human interventions), and handling the ambiguous, stochastic rewards of the real world. Critically, flow-matching/diffusion VLAs have no tractable log-likelihood, so policy gradients such as PPO/REINFORCE cannot be attached directly. A training method that bypasses policy gradients was needed.

Key Ideas and Method

RECAP is an iteration of three subroutines: (1) data collection → (2) value function learning → (3) advantage-conditioned policy learning. Only the data changes between stages.

Module 1: Distributional Multi-task Value Function (Critic)

Same architecture as the VLA but with a smaller VLM backbone (670M, Gemma 3 initialized). The inputs are identical to the VLA: 3 camera streams + joints + language.

Rewards are sparse and generic. Only a success/failure label is attached at the end of an episode, and the reward is defined as follows.

\[r_t = \begin{cases} 0 & \text{success termination} \\ -C_{\text{fail}} & \text{failure termination} \\ -1 & \text{otherwise} \end{cases}\]

The value function therefore predicts “the negative of the number of steps remaining until success,” normalized to \((-1, 0)\) by the per-task maximum length. The value is represented as a distribution \(p_\phi(V \mid o_t, \ell)\) over \(B = 201\) bins, and the empirical return \(R_t(\tau)\) is discretized and learned by Monte-Carlo with cross-entropy. The continuous value is extracted as \(V(o_t, \ell) = \sum_b p_\phi(V = b \mid o_t) \, v(b)\). It is co-trained with web multimodal data to prevent overfitting.

Intuitively, this critic is a referee that reads “how close this scene is to success right now, and whether a mistake was just made.” The paper’s visualization shows the value dropping and then rising as laundry is crumpled and then recovered.

Module 2: Advantage-conditioned Policy Extraction (The Key Trick)

The advantage is “how much better this action is than the average for that state,” computed with an n-step estimate.

\[A(o_t, a_t) = \mathbb{E}\left[\sum_{t'=t}^{t+N-1} r_{t'} + V(o_{t+N})\right] - V(o_t)\]

Post-training uses \(N = 50\), pretraining uses \(N = T\) (the whole episode). This is binarized with a per-task threshold \(\epsilon_\ell\) to form the improvement indicator \(I_t\) (positive/negative). \(\epsilon_\ell\) is usually set so that the top 30-40% of the data is positive (about 30% of demonstrations in pretraining, about 40% of rollouts in fine-tuning; for t-shirt laundry only about 10% is positive because the demonstrations are slow).

The indicator is simply injected into the policy as text. In the sequence, “Advantage: positive” or “Advantage: negative” is inserted after the subtask \(\hat{\ell}\) and before the actions, and all data, good or bad, is trained in a supervised manner. At inference, conditioning on positive draws only good actions. The training objective is as follows.

\[\mathcal{L}(\theta) = -\log \pi_\theta(a_t \mid o_t, \ell) - \alpha \log \pi_\theta(a_t \mid I_t, o_t, \ell), \quad I_t = \mathbb{1}[A > \epsilon_\ell]\]

In the actual implementation, instead of \(\alpha\), an advantage-conditioning dropout of 30% is used to learn the conditional and unconditional together.

The theoretical basis is the CFGRL family. If the improved policy is defined as \(\hat{\pi}(a \mid o) \propto \pi_{\text{ref}}(a \mid o) \cdot p(I \mid A)^\beta\), then \(J(\hat{\pi}) \geq J(\pi_{\text{ref}})\) is guaranteed. By Bayes’ rule, \(p(I \mid A) = \pi_{\text{ref}}(a \mid I, o) / \pi_{\text{ref}}(a \mid o)\), so with \(\beta = 1\) we get \(\hat{\pi}(a \mid o) = \pi_{\text{ref}}(a \mid I = \text{True}, o)\). That is, there is no need to represent the improvement-probability term separately; it suffices to have the model learn both the conditional and the unconditional, which is isomorphic to classifier-free guidance. After training, the policy can also be sharpened with \(\beta > 1\).

\[\nabla \log \pi(\cdot \mid o) + \beta \left( \nabla \log \pi(\cdot \mid I, o) - \nabla \log \pi(\cdot \mid o) \right)\]

However, increasing \(\beta\) too much pushes the action distribution to the boundary of the support and makes it aggressive, so control is mainly via \(\epsilon_\ell\) and \(\beta\) is kept around 1.5-2.5.

Why not PPO/AWR. PPO is hard to apply to flow matching because there is no log-likelihood, and it is unstable off-policy. AWR loses by discarding data (filtered imitation) or down-weighting it. Advantage conditioning uses all data in a supervised manner while removing the complexity of policy gradients, so it scales to large VLAs.

Since exact likelihood evaluation is impossible for the continuous action part, the parallel between flow matching and diffusion (diffusion gives a log-likelihood lower bound) is used to motivate the discrete action log-likelihood + continuous flow-matching loss as a lower bound on the full action likelihood.

Module 3: Data Collection (Experience + Corrections)

  • Autonomous rollouts. The policy is run on real robots to collect episodes, which are labeled success/failure.
  • Expert intervention (human-gated DAgger). A teleoperator steps in just before failure and corrects. Corrective actions are forced to \(I_t = \text{True}\) unconditionally (assuming the expert is always right).
  • Whole episodes, both autonomous and corrected portions, are accumulated into the dataset.

The key insight is that intervention alone is insufficient. Intervention only addresses large mistakes and exploration; fine improvements such as speed are handled by autonomous RL. The combination of intervention (coaching) and autonomous RL (practice) is essential.

The Full Loop

Stage Content
Pretraining Train the value function + advantage-conditioned policy on all demonstrations (tens of thousands of hours, diverse robots) → π*0.6 generalist
Post-training initialization SFT on demonstrations (fine-tuning with \(I_t = \text{True}\) fixed) → initial policy
Iteration Data collection (autonomous + intervention) → retrain value function → retrain policy

Each iteration fine-tunes from the pretrained checkpoint rather than the previous model to prevent drift. A single iteration already yields large improvements, and difficult tasks run two. In one line, “convert the critic’s advantage into prompt text and feed it to the VLA”; reducing RL to supervised next-token/flow-matching training is the essence of RECAP.

Experiments and Results

Hardware. A static bimanual system, two 6-DoF arms + parallel grippers, 50 Hz joint control, 3 cameras (1 base + 2 wrist). Pretraining uses diverse robots; the iterative improvement experiments use this fixed setup.

Model. π0.6 = Gemma 3 4B backbone + 860M flow-matching action expert. Trained with the KI recipe and also predicts discrete actions with the FAST tokenizer (for KI). Subtask text \(\hat{\ell}\) is predicted at low frequency to condition the actions.

Tasks (3, each 5-15 minutes).

Task Content Time limit
Laundry T-shirts and shorts / 11 diverse garments (measured on button-up shirts) / strict-criterion failure-removal ablation 200-500 s
Cafe Double shot on a commercial espresso machine (portafilter, grinding, tamping, locking, extraction, serving) 200 s
Box assembly Flat cardboard → assembly → labeling → crate loading, deployed in a real factory 600 s

Baselines. π0.5 / π0.6 (supervised) / RL pre-trained π0.6 / π0.6 offline-RL+SFT / π*0.6 (full RECAP, default \(\beta = 1\)). For policy extraction comparisons, AWR and PPO (DPPO/FPO variants + SPO trust region) were compared on the same data.

Metrics. Throughput (successful completions per hour, combining speed and success rate) and success rate (based on human annotation).

Data scale. Laundry (t-shirt): 4 robots × 300 autonomous; diverse laundry: 450 evaluation + 287 corrections; failure removal: about 1000 autonomous + 280 + 378 corrections (3 robots); cafe: 414 autonomous + 429 corrections (1 iteration); box: 600 demonstrations + 360 corrections per iteration (3 robots).

The three key results are as follows.

1. More than 2x throughput and roughly half the failure rate on the hardest tasks. On diverse laundry and espresso, the addition of on-robot data in full RECAP (offline-RL+SFT → final π0.6) was the key contribution. The final π0.6 outperformed all baselines.

2. Practical-level robustness. It achieved 13 hours of continuous espresso operation (05:30-23:30), over 2 hours of uninterrupted novel laundry in a new home, and assembly of packaging boxes in a real factory (59 boxes of chocolate packaging). Success rate is over 90% on all tasks except diverse laundry.

3. Advantage conditioning dominates AWR and PPO. Compared on the same data, RECAP far exceeded both methods in throughput. PPO was unstable off-policy without a trust region (\(\eta = 0.01\)), and AWR achieved success rates but produced slow policies with low throughput.

Details.

  • Effect of iteration. Laundry reached over 90% success after one iteration, after which improvement was mainly in throughput (cumulative +50%). Box assembly, being a long-horizon task, improved more slowly and reached 2x throughput after the second iteration. Some dropped in the first iteration and then recovered.
  • Failure mode removal. Under the strict criterion of “fold with the collar facing up” and adversarial initial conditions, two RECAP iterations (600 trajectories each) alone reached 97% success. This showed that specific mistakes can be targeted and removed by pure RL without demonstrations or interventions.
  • Box assembly by stage. π*0.6 was the best in all four stages: picking, assembly, labeling, and loading. Most failures were timeouts, and assembly and labeling reached about 90% within 600 s.

Claims vs reality. The authors claim to be “the first to show that a general RL recipe using human reward feedback + interventions meaningfully improves VLA robustness and throughput from deployment experience alone.” At the same time they candidly acknowledge that the individual algorithmic elements of RECAP (advantage conditioning, CFGRL, distributional VF, HG-DAgger, the AWR/IQL family) are existing work. What is new is the combination, the scaling to a large flow-matching VLA, and the demonstration on real-world long-horizon dexterous tasks. The core contribution is the integration, system, and proof of a “recipe that works” rather than algorithmic innovation.

Conclusion and Significance

π*0.6 is the first demonstration that brings “VLA + real-world RL” to the level of hours of uninterrupted operation in actual commercial settings (factory box assembly, office cafe). It provided a concrete recipe for exceeding the ceiling of imitation learning (demonstration quality and speed) with RL, and established the “advantage as prompt text condition” pattern as a practical solution for applying RL to flow-matching VLAs. Since it uses all off-policy data without policy gradients, it serves as a bridge for transferring LLM RLHF infrastructure assets to robots. The authors’ conclusion is that “training VLAs with RL is the most direct path to production-level performance.”

Its applicability and limitations from a robotics practice viewpoint are as follows.

  • No code or checkpoints. Code and checkpoints for RECAP, π*0.6, and π0.6 are confirmed unreleased as of 2026-06. openpi contains only π0, π0-FAST, and π0.5. The algorithm itself (advantage conditioning + distributional VF) could be re-implemented on top of openpi’s π0.5, but reproducing paper-level results depends on private resources in data, real robots, and human operation, so reproduction difficulty is high. Validating the RECAP idea in simulation or small-scale environments, such as a quadruped simulation testbed, is feasible.
  • It is not fully autonomous. Reward labeling, intervention, and environment resets all depend on human labor. The authors themselves identify VLM-based reset/reward automation as future work. It presupposes end-to-end training of a 4B VLM + 860M action expert, simultaneous training of a 670M critic, on-the-fly value inference, and operation of 3-4 multi-robot stations.
  • Exploration is naive (greedy). It relies only on policy stochasticity and human intervention, so it works only when the initial imitation policy is passable. There is no sophisticated exploration.
  • Iterative offline, not online. The “collect → retrain → repeat” structure is not real-time concurrent online RL, and the value function is an on-policy Monte-Carlo estimate, less optimal than Q-learning-style off-policy methods. This is a choice prioritizing simplicity and stability, and off-policy extension is an open problem.
  • Reward ambiguity. With sparse binary success labels, partial success and subtle quality differences are hard to capture, and it depends on human annotation quality.

References

  • Paper: arXiv 2511.14759 · Physical Intelligence blog
  • Code: not released (not confirmed). Related repo: openpi (π0, π0-FAST, π0.5 only)
  • π0.5: a Vision-Language-Action Model with Open-World Generalization (Black et al., CoRL 2025) — immediate base
  • Diffusion Guidance Is a Controllable Policy Improvement Operator / CFGRL (Frans et al., 2025) — theoretical origin of advantage conditioning
  • Advantage-Weighted Regression / AWR (Peng et al., 2019) — policy extraction comparison
  • Knowledge Insulation for Training Vision-Language-Action Models (Driess et al., NeurIPS 2025) — backbone of the π0.6 training recipe
  • HG-DAgger: Interactive Imitation Learning with Human Experts (Kelly et al., ICRA 2019) — source of the expert intervention mechanism