Diffusion policy RL has a hidden unification problem — and it's slowing everyone down

Diffusion and flow models have become the default policy representation for dexterous robot learning. Every major lab uses a variant. The challenge is that each team's RL fine-tuning approach looks completely different, because the field never agreed on why these methods work. Without log-probabilities, vanilla policy gradient estimators break on diffusion policies, and the dozen fixes proposed so far share no common language.

This gap is structural, rather than technical. FlowRL builds a taxonomy that maps all existing RL-with-diffusion-policy methods onto a unified framework. This framework exposes which design choices are genuinely distinct and which are surface-level variations of the same underlying idea. The modular JAX (Just-in-Time compiled array computing framework)-based codebase operationalizes this; each algorithmic component is swappable independently. This allows a team to test whether their "novel" RL approach is actually a new mechanism or just a different parameterization of an existing one. JIT (Just-In-Time) compilation drives throughput high enough to make systematic ablations tractable, a type of controlled comparison currently rare in this subfield.

The limitation is real: a taxonomy and codebase provide infrastructure, not results. FlowRL doesn't claim a new algorithm outperforms prior methods by a specific margin; its contribution is the scaffold that makes fair comparison possible. For teams actively doing robot learning research, this distinction matters. The value compounds, offering faster iteration cycles and a shared vocabulary that allows findings to transfer across groups.

Key takeaways:

• The absence of log-probabilities in diffusion/flow policies breaks standard policy gradient. Existing fixes address this differently at the surface, but the taxonomy makes their deeper structural patterns explicit.
• A field without unified vocabulary accumulates redundant work; reproducibility gaps compound over time, and this framework directly targets that failure mode.
• Teams building RL on top of diffusion policies should audit their current approach against the taxonomy before designing new experiments, because what looks like a novel contribution may already have a tested analog in the framework.

Source: FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies