Constant-Entropy Video Generation: Predicting Frames by Conserving Spectral Entropy

The Hypothesis: Frames Change a Lot, Entropy Barely Moves

A synthetic dataset of 500 short videos (20 frames each, moving shapes on a drifting-color background) gives full control over ground truth: every frame-to-frame change is smooth motion, with no scene cuts or lighting changes that would genuinely destroy or create information. Three entropy measures were computed on every consecutive frame pair — spectral entropy (Shannon entropy of the Fourier power spectrum), pixel-histogram entropy, and spatial variance as a cheap entropy proxy.

Measure	Mean rel. change	Std	Max
Spectral entropy	1.22%	1.43%	14.00%
Pixel-histogram entropy	2.32%	3.17%	74.36%
Spatial variance	2.00%	1.77%	14.25%

Relative change |H(f_t+1) − H(f_t)| / H(f_t), aggregated over 20 batches of 64 real consecutive-frame pairs. Spectral entropy is both the smallest and the most consistent of the three — the measure this project builds on.

Spectral entropy plotted across the 20 frames of a single synthetic video, showing a gentle downward drift with an 11.39% relative range

Spectral entropy across one full 20-frame sequence. It isn't perfectly flat — there's a gentle downward trend as shapes drift and overlap changes — but the total range (11.4% peak-to-trough over 20 frames) is small next to how much the pixels themselves change, consistent with the near-conservation hypothesis.

FlowNet: Predicting a Delta, Not a New Frame

Rather than predicting frame t+1 directly, the network predicts a flow field v such that u_t+1 = u_t + v(u_t) — a U-Net-style encoder/decoder with residual blocks, whose final output layer is zero-initialized so the network starts out as the identity map (predict no change at all) and has to earn every pixel of motion during training. Training combines two losses: standard reconstruction MSE against the true next frame, plus an entropy-conservation term that penalizes any difference between the spectral entropy of the input frame and the spectral entropy of the prediction.

L = ‖ u_t + v(u_t) − u_t+1 ‖² + λ · ( H(u_t+v(u_t)) − H(u_t) )² Reconstruction loss plus an entropy-conservation penalty, λ = 1.0

Setting	Value
Dataset	500 synthetic 20-frame videos, 64×64, moving shapes — 9,500 consecutive-frame pairs
Model size	4,171,331 parameters
Optimizer	AdamW, lr 2e-4, cosine annealed, gradient-clipped at 1.0
Training	100 epochs, batch size 32
Loss weight	λ = 1.0 for the entropy-conservation term

Training loss curves: reconstruction and total loss decreasing smoothly on a log scale, and entropy-conservation loss decreasing with more noise over 100 epochs

Left: reconstruction loss falls from 0.00399 to 0.00033 over 100 epochs. Right: the entropy-conservation loss falls too (0.000202 → 0.000013), though noisily — consistent with it being a much smaller, secondary term competing with the dominant reconstruction objective.

From Frozen to Tracking: Rollout Over Training

Every 20 epochs, the model performs a 16-step autoregressive rollout — feeding its own prediction back in as the next input — starting from the first real frame of a held-out sequence.

Rollout comparison at epoch 1: predicted frames barely move and accumulate color smearing — Epoch 1 — barely trained. The near-identity initialization keeps shapes roughly in place instead of tracking their motion, and color artifacts accumulate step by step.

Rollout comparison at epoch 100: predicted frames track the ground-truth motion closely, with growing but modest visual artifacts — Epoch 100 — fully trained. The predicted rollout (bottom) tracks the ground truth (top) motion closely for all 16 steps, though a faint trailing smear and a color halo build up around each shape as steps accumulate.

But Entropy Still Drifts, Even Here

The entropy-conservation loss shrinks the entropy gap on individual training pairs, but it doesn't prevent compounding drift once the model's own predictions become its next input.

Spectral entropy of ground-truth vs predicted rollout over 16 steps at epoch 100, showing predicted entropy climbing steadily above ground truth despite the entropy-conservation loss

Ground-truth entropy (blue) drifts gently downward across this 16-step sequence, matching the single-video plot above. The model's own rollout (red) instead climbs steadily upward, diverging further from ground truth with every step — the entropy loss constrains single-step predictions, not multi-step autoregressive behavior.

The Long Rollout: 40 Steps, 30% Entropy Drift

Pushed out to 40 autoregressive steps — twice the length of any training sequence — the model keeps both shapes recognizable and roughly on their correct trajectories, but visible noise and texture accumulate around their edges.

Animated 40-step autoregressive rollout showing a triangle and circle moving while accumulating visual noise and texture around their edges

Sampled frames from a 40-step autoregressive rollout, showing shapes tracking their motion while accumulating a jittery, high-frequency texture

16 sampled frames spanning the full 40-step rollout. Shape position and identity survive the whole way, but a jittery, high-frequency texture visibly accumulates — exactly the kind of change that spectral entropy is built to detect.

Spectral entropy over 40 rollout steps, climbing steadily from 0.594 to 0.774, a 30.23% drift

Spectral entropy climbs from 0.594 to 0.774 over 40 steps — a 30.23% drift, and visibly correlated with the accumulating texture noise in the frames above.

The honest takeaway

The core hypothesis held up: real consecutive frames really do have near-constant spectral entropy (~1.2% mean frame-to-frame change), and training with an explicit entropy-conservation loss produces a model that tracks motion far better than one trained on reconstruction alone. But conserving entropy on individual training pairs is a different thing from conserving it across an autoregressive chain of the model's own predictions — small per-step errors compound into a fairly large drift after enough steps, the same exposure-bias problem that affects autoregressive next-frame prediction in general. An entropy penalty computed on single steps doesn't automatically buy long-horizon stability.