Online RL · Flow-based VLA

π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

Online RL for flow-based VLA policies using Flow-SDE exploration and variance-normalized, step-wise contrastive ranking, without computing final-action likelihoods.

Siting Wang, Xiaofeng Wang, Zheng Zhu, Minnan Pei, Xinyu Cui, Cheng Deng, Jian Zhao, Guan Huang, Haifeng Zhang, Jun Wang

TL;DR

π-StepNFT maps binary outcomes or critic-derived feedback to local velocity updates for flow-based VLA policies, without computing final-action likelihoods.
Key insight: wider exploration space needs finer, step-wise alignment — achieved via Flow-SDE exploration + step-wise contrastive ranking.

Highlights

Binary-outcome updates require no critic; the same ranking objective also accepts critic-derived feedback.
Flow-SDE sampling explores a wider local stochastic transition space.
Variance-normalized ranking supervises the adjacent transitions observed during sampling.
Shared-initialization gains on LIBERO and stronger reported mean success on five ManiSkill visual shifts.

Method at a Glance

Figure 1 — Wider stochastic exploration with finer step-wise supervision
Figure 1 — Wider stochastic exploration with finer step-wise supervision. ODE and naïve SDE baselines use terminal-derived targets; variance normalization accounts for local stochastic transitions; π-StepNFT directly supervises the observed adjacent solver transitions.
How π-StepNFT works (conceptually):
1) Explore local stochastic transitions with Flow-SDE rollouts.
2) Use observed adjacent solver states as supervision and normalize transition errors by local SDE variance.
3) Rank mirrored positive and negative velocity branches using rollout feedback.
4) Update from one sampled solver transition using binary outcomes or critic-derived feedback, without computing final-action likelihoods.

Why It Matters

Online reinforcement learning for flow-based VLAs must translate rollout feedback into updates along a multi-step denoising trajectory. Stochastic rollouts explore a wider transition space, but supervision reconstructed from the final output does not directly use the adjacent transitions observed during sampling.

π-StepNFT addresses this exploration–supervision mismatch by ranking variance-normalized errors on observed SDE transitions, connecting outcome feedback to local velocity updates.

Main Results

Updated 28 October 2026 to match the current manuscript revision. All results are success rates (%). π-StepNFT results are three-seed means; πRL baselines use the results reported by their original authors.

LIBERO · Few-shot online improvement
ModelSpatialObjectGoalLongAvg.Δ Avg. (pp)
Full SFT
Octo78.985.784.651.175.1—
OpenVLA84.788.479.253.776.5—
πfast96.496.888.660.285.5—
OpenVLA-OFT91.695.390.686.591.0—
π096.898.895.885.294.2—
π0.598.898.298.092.496.9—
Few-shot SFT + RL (π0)
SFT65.364.449.851.257.6—
πRL (Flow-SDE + PPO)98.499.496.290.296.0+38.4
πRL (Flow-SDE + GRPO)97.897.883.281.490.0+32.4
π-StepNFT95.498.087.986.792.0+34.4
π-StepNFT + Critic97.698.089.890.093.9+36.3
Few-shot SFT + RL (π0.5)
SFT84.695.484.643.977.1—
πRL (Flow-SDE + PPO)99.610098.893.097.9+20.8
πRL (Flow-SDE + GRPO)97.499.891.277.691.5+14.4
π-StepNFT97.810098.279.894.0+16.9
π-StepNFT + Critic98.099.398.288.496.0+18.9
Table 1 — LIBERO results from manuscript Table 1. Online RL methods share the same few-shot SFT initialization within each backbone. Δ Avg. is the gain over that initialization in percentage points (pp).
ManiSkill · In-distribution and OOD
ModelINDVisual (5)Visual-language (6)Semantic (2)Execution (2)
π0
Full SFT38.433.733.16.413.3
πRL (Flow-SDE + PPO)78.860.461.122.331.5
πRL (Flow-Noise + PPO)77.864.464.720.724.2
π-StepNFT73.666.565.821.829.7
π0.5
Full SFT40.137.338.814.322.4
πRL (Flow-SDE + PPO)90.966.268.033.145.4
πRL (Flow-Noise + PPO)89.769.872.433.354.9
π-StepNFT84.375.376.630.542.2
Table 2 — ManiSkill category results from manuscript Table 12. Visual averages cover five appearance perturbations; visual-language averages additionally include instruction-phrasing shifts. Semantic and execution averages each use two shared variants with results available for all methods. Bold marks the best result within each backbone and column.

LIBERO: π-StepNFT raises average success from 57.6% to 92.0% for π0 (+34.4 pp) and from 77.1% to 94.0% for π0.5 (+16.9 pp). Critic-derived feedback further improves the averages to 93.9% and 96.0%, respectively. PPO retains the highest overall means.

ManiSkill visual shifts: π0.5-StepNFT reaches 75.3% across five test-only visual perturbations, compared with 66.2% for Flow-SDE and 69.8% for Flow-Noise. These are gains of 9.1 pp and 5.5 pp. Its robustness advantage is concentrated in visual appearance shifts; PPO baselines remain stronger in-distribution and on semantic and execution category averages.

Key Ablations

(a) Rollout and noise treatment
(a) Rollout and noise treatment
(b) Regression target
(b) Regression target
(c) Step selection strategy
(c) Step selection strategy
(d) Learning objective
(d) Learning objective
(e) Preference signal
(e) Preference signal
(f) Subgoal feedback
(f) Subgoal feedback
Figure 2 — Component-wise ablations of π-StepNFT. (a) Rollout weighting and variance normalization under terminal targets. (b) Immediate next-step targets stabilize stochastic training. (c) Random step selection improves stability over fixed steps. (d) Ranking outperforms weighted-MSE and isolated positive/negative branches. (e) Binary outcomes perform best on the short-horizon ablation; ranking also supports soft feedback. (f) Relative subgoal feedback helps in a controlled long-horizon CALVIN diagnostic. Curves show five-seed means with ±1 standard deviation.
(a) Noise level σ
(a) Noise level σ
(b) EMA decay α
(b) EMA decay α
(c) Branch displacement scale β
(c) Branch displacement scale β
(d) Trust-region weight λTR
(d) Trust-region weight λTR
Figure 3 — Hyperparameter sensitivity analysis. Curves show five-seed means with ±1 standard deviation; bold pink marks the main-experiment configuration.

Excessive SDE noise impedes convergence, while insufficient noise limits local exploration. Dynamic EMA decay balances rollout-policy lag and update stability. Branch displacement scales β ∈ [1, 2] perform best, and a trust-region weight of λTR = 10−4 regularizes updates without slowing convergence.

Citation

@misc{wang2026pistepnftwiderspaceneeds,
      title={$\pi$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs}, 
      author={Siting Wang and Xiaofeng Wang and Zheng Zhu and Minnan Pei and Xinyu Cui and Cheng Deng and Jian Zhao and Guan Huang and Haifeng Zhang and Jun Wang},
      year={2026},
      eprint={2603.02083},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.02083}, 
}