Entropic Flow Policy: Contracting Goal-Reachability Entropy Instead of Maximizing Reward

Entropy as the Only Supervisory Signal

Most imitation learning either copies actions directly (behavior cloning) or infers a reward and optimizes it (inverse RL). EFP does neither. It defines goal-reachability entropy: from a state s, how many distinct trajectories of length ≤ H still reach the goal s_g? Far from the goal, many paths remain viable — entropy is high. Near the goal, the funnel narrows — entropy collapses toward zero. A policy that always moves toward lower entropy is, by construction, moving toward the goal.

Demonstrations are treated as an empirical trace of that funnel, not as actions to imitate. That distinction gives EFP a self-correcting property plain behavior cloning lacks: if the agent drifts off the demonstrated path, the entropy gradient still points back toward the funnel, because entropy is defined over the whole state space, not just the states a demo happened to visit.

Aspect	Standard RL	MaxEnt RL	EFP (this work)
Entropy's role	not used	maximize policy entropy	minimize reachability entropy
Reward signal	extrinsic	extrinsic + entropy bonus	entropy is the signal
Core quantity	`V(s)`	`V_soft(s)`	`H(s, s_g)` = log-reachability
Diffusion connection	none	none	state entropy sets the sampler's noise level

The synthesis

Diffusion policies already contract entropy — in action space, across denoising steps. EFP contracts entropy in state space, across real timesteps. Entropic Flow Policy links the two: the current state entropy sets the noise scale for the action sampler — broad, exploratory noise far from the goal; tight, precise noise near it — and the denoising score is nudged by −∇_aH_θ at every step.

How it actually trains, in five stages

There is no reward model anywhere in this pipeline — every stage is supervised either by the demonstrations' own timestamps or by the dynamics the policy induces.

#	Stage	What it's trained on
1	Label entropy from demo ordering	H_label(s_t) = log(steps-remaining-to-goal + 1) — a free label from timestamps alone
2	Train the entropy estimator H_θ(s, s_g)	ranking loss (later states score lower than earlier ones) + anchor loss (H(s_g,s_g)=0) + soft regression to the label
3	Train a dynamics model f_φ(s, a) → s′	plain MSE — needed so ∇_aH can be computed by differentiating through the predicted next state
4	Train a DDPM score network ε_ψ	standard diffusion-policy noise-prediction objective on expert actions
5	Sample actions at inference	denoise as usual, but steer every step by −∇_aH_θ(f(s,a), s_g), with the initial noise scale set by the current H — broad far from goal, tight near it

What the learned entropy landscape looks like

Contour map of the learned entropy field around the goal, showing entropy decreasing toward a basin

Entropy value plotted along a successful rollout, decreasing monotonically toward zero at the goal

Left — H_θ(s, s_g) evaluated over the state space: a basin centered on the goal, with entropy rising smoothly the farther a state is from it, exactly the funnel the contraction principle assumes. Right — entropy sampled along one successful rollout, trending down toward zero at the goal.

Three Tasks, Escalating in How Much They Punish Mode-Averaging

Each experiment exists because the previous one wasn't hard enough to separate EFP from behavior cloning. The task design itself is a finding.

Exp. 1 — 3-link planar arm, reaching

A redundant 3-DoF arm reaching for 10 targets, 200 demos. Each target has a single unambiguous solution — no real multimodality, so this is a sanity check that entropy contraction works at all before asking it to out-perform anything.

Full six-panel result figure for the 3-link arm reaching experiment: entropy along trajectories, training losses, success rate, final distance distribution, end-effector trajectories, and the learned entropy landscape

EFP 70% vs. BC 70% — entropy contraction confirmed (entropy falls monotonically on successful rollouts, stays high on failures), but the task is too easy to differentiate methods.

Exp. 2 — 2D navigation around a road-block obstacle

A point mass must get from a start region to a goal star past two rectangular blocks. First pass: the blocks leave a narrow center gap, so upper/lower/middle are all technically viable — 291 demos.

Result

BC 100% vs. EFP 72% — BC's mode-average sneaks straight through the gap, so the task didn't punish averaging. EFP led only under perturbation (93% vs. 80% for a mixture-density baseline).

Closing the gap entirely forces a genuine binary choice — strictly upper or strictly lower, nothing in between. If the "BC averages two modes into a useless path" story were the whole explanation, EFP should win here. It doesn't: BC 100% vs. EFP 56% on the in-distribution, closed-gap variant — a real neural net doesn't literally output the arithmetic mean of two modes, so plain BC still commits to one path and wins. This is why Push-T (Exp. 3) exists: it needed a task where averaging two valid solutions produces something that isn't just "a slightly worse path" but a physically ineffective one.

Learned entropy landscape for the open-gap obstacle navigation task, showing a funnel splitting around two blocks toward the goal

Learned entropy landscape for the closed-gap obstacle navigation task, showing a flatter, less directional gradient around the block

Left, open-gap: one funnel per block, both merging cleanly toward the goal — the field itself doesn't distinguish the gap as a shortcut. Right, closed-gap: flatter and less directional around the block — a plausible reason the guidance signal underperforms plain imitation here.

Every iteration, in full (v2 → v6)

Five task-design passes on the same road-block environment, in order. BC/MDN-BC beat EFP in every one except the open-gap perturbation case.

Full result figure for obstacle navigation v2: open center gap

v2 — open center gap: EFP 72%/68%/93% (In-Dist/OOD/Perturb) vs. BC & MDN-BC ~100%/100%/93%,80%.

Full result figure for obstacle navigation v3: failure supervision added

v3 — failure supervision added (120 failure trajectories, 1179 failure states): EFP 56%/72%/53% — failure demos alone don't close the gap to BC/MDN-BC (~100%/100%/93%).

Full result figure for obstacle navigation v4: refined failure gradient

v4 — refined failure gradient (left-approach, boundary-only, normalized): EFP improves to 72%/76%/60%, still behind BC/MDN-BC (~100%/100%/87–93%).

Full result figure for obstacle navigation v5: closed gap, strictly upper or lower paths

v5 — gap fully closed (genuine binary choice): EFP drops to 56%/52%/67% — still loses to BC/MDN-BC (~100%/100%/93%) even without an averaging escape route.

Full result figure for obstacle navigation v6: fixed start zone with flow matching and Diffusion Policy baseline

v6 — fixed start zone + flow matching + DP baseline: the task reverts to effectively single-path; all four methods land at ~92–100% and stop being informative.

Exp. 3 — Push-T (standard diffusion-policy benchmark)

Push a T-block into a target pose. No gap to sneak through: averaging two valid pushes produces a push that works for neither.

Result

EFP + failure demos wins outright — the headline result below.

On Push-T, Failure Demonstrations Are the Ingredient That Makes Entropy Guidance Work

Two conditions were tested: Standard (agent starts in a normal position) and Perturb (agent starts directly above the block — the setup that, without correction, drives the block away from the goal). EFP was trained two ways: with a small set of labeled failure trajectories added to its entropy estimator, and without.

Push-T benchmark: mean max coverage and success rate for EFP with/without failure demos, Diffusion Policy, BC, and MDN-BC, under Standard and Perturb start conditions

Success % (coverage > 0.35) under both conditions. Standard: EFP+failure 64%, Diffusion Policy 24%, everything else 0%. Perturb (adversarial start): EFP+failure 56%, every other method 0%.

Why failure demos matter

Without them, EFP's entropy gradient has never seen what "getting worse" looks like, so under perturbation it has no signal to correct course, and its guidance term actively fights the diffusion prior. Coverage collapses to 0.000 in both conditions, worse than plain Diffusion Policy's 0.143 / 0.010.

Add failure trajectories, and the same architecture becomes the only method that survives the adversarial start: 64% / 56% success, 0.359 / 0.285 coverage, versus ≤24% / 0% for every baseline.

Standard start

rollouts for all five methods, same seed family

EFP (w/ failure)

64% success

EFP (no failure)

0% success

Diffusion Policy

24% success

0% success

MDN-BC

0% success

Perturb start (adversarial)

agent starts directly above the block

EFP (w/ failure)

56% success

EFP (no failure)

0% success

Diffusion Policy

0% success

MDN-BC

0% success

How Sensitive Is This to the Guidance Scale?

A separate ablation applies entropy guidance on top of plain BC (rather than a diffusion sampler) and sweeps the guidance scale from 0.05 to 5.0.

Guidance scale sweep for entropy-guided BC, showing success collapsing to zero above guidance scale 0.1

Entropy-guided BC only works in a narrow band around guidance-scale 0.05–0.1 (coverage 0.31–0.41, up to 100% success in Standard); above ~0.3 it collapses to 0% in both conditions. The entropy gradient is a knife-edge correction, not a signal that tolerates being over-applied.

Does This Replicate on Real Human Demonstrations?

Everything above uses a hand-scripted synthetic expert. Re-running the same comparison on the official Push-T dataset (206 human-teleoperated episodes) tells a different story:

EFP vs Diffusion Policy on the real human-demonstration Push-T dataset, showing nearly all diffusion-based methods at 100% success in both conditions

On the real dataset, EFP (w/ and w/o failure, w/ and w/o guidance) and plain Diffusion Policy all reach ~100% success in both Standard and Perturb. Only BC (Mean) and MDN-BC fail — BC's coverage holds up standard (100%) but collapses under perturbation (13%); MDN-BC fails outright.

The honest caveat

Failure supervision's advantage on synthetic Push-T exists because the hand-scripted expert gave plain Diffusion Policy a narrow, brittle demonstration set to imitate — DP only scored 24%/0% there. Human demonstrations are naturally more diverse, and standard Diffusion Policy trained on them is already robust to the adversarial start, without any entropy guidance at all. The failure-supervision finding is real, but it's a finding about compensating for weak demonstrations, not a universal improvement over Diffusion Policy.

Where the Entropy-Contraction Idea Goes Next

Formal bound on the contraction principle — when exactly does "lower reachability entropy implies closer to goal" break down? Exp. 2 already shows one failure mode (a gap wide enough that the averaged path is still viable).
Making the synthetic Push-T finding robust — the failure-supervision advantage disappears on real human demonstrations (above), because the baseline Diffusion Policy is already strong there. Finding a task where it helps regardless of demonstration quality is the natural next step.
Table-tennis contact events — a natural, sharp entropy bottleneck (pre- vs. post-contact) to test the framework on a real robot.

A related, code-independent project applies the same entropy-contraction thesis to the denoising process itself — see Heat Dissipation Policy.