A Kinova Jaco2 arm with a spoon taped to its gripper scoops pasta from a container and transfers it to a plate — 37 real recorded episodes. Can NVIDIA's Cosmos3 world model generate more episodes like it, either to train a policy directly or as a way to stress-test whether video world models are a viable synthetic-data source for real robot manipulation at all?
Collecting real robot demonstrations is slow and expensive. World models like Cosmos3 can generate plausible video conditioned on an image and either an action sequence or a text prompt — which raises an obvious question for imitation learning: could a world model generate usable training episodes for a real manipulation task, cheaply, without a robot in the loop? This project is a hands-on log of testing that on one concrete task — pasta scooping and transfer — with two different generation strategies, and an attempt to close the loop by recovering action labels from the generated video.
| Approach | Outcome |
|---|---|
| Stock environment smoke test (no robot data) | worked clean, artifact-free camera trajectory |
| Exact trajectory conditioning at native ~2.67Hz sample rate | wrong timing outside the model's supported fps range |
| Same, resampled to 15fps (SLERP-interpolated) | accurate matched real wrist rotation and object interaction |
| Chained 25× into a ~27s rollout, each chunk conditioned on the last frame | collapsed artifact noise by ~10 seconds |
| Chained with 5-frame context instead of 1 frame per chunk | collapsed faster more synthetic context made drift worse |
| One real photo + one sentence describing the task, no trajectory | best result correct task execution, no drift |
| Re-tracking the generated video (Depth Anything + FoundationPose) | worked accurate 6-DoF labels, no ground truth needed |
The dataset: 37 real episodes of a Kinova Jaco2 arm re-executing a tracked scooping trajectory, recorded from two fixed RGB cameras at roughly 150 waypoints per episode. Before any generation could be evaluated, every episode needed accurate 6-DoF spoon-pose labels — the exact thing a model would be conditioned on or judged against.
The first labeling attempt used robot forward-kinematics plus a calibrated tool offset, but this was unreliable — 10–20cm off on some frames, from calibration error amplified by the wrist-to-spoon-tip lever arm. The fix was to track the spoon directly: Depth Anything V2 for metric depth (this recording had no depth sensor) feeding FoundationPose for 6-DoF pose from a CAD mesh, with periodic re-registration every 30 frames — a single registration drifts unrecoverably over a ~150-frame sequence.
Cosmos3's forward_dynamics mode takes a starting image plus a sequence of relative 6-DoF pose
deltas (translation, rotation, gripper state) and generates the corresponding video — the literal way to
ask "show me this exact motion."
Interpolating the tracked trajectory (linear translation + SLERP rotation) onto a continuous 15fps timeline — inside the model's supported range — fixed it. Tested on both a near-static window and a genuinely high-motion one (spoon dipping deeper into the container, wrist visibly rotating):
Each forward_dynamics call only produces a fixed ~1-second chunk (16 actions → 17
frames). A longer synthetic episode means chaining: feed the last generated frame back in as the next
chunk's starting image, and repeat.
The natural fix to try: condition each new chunk on the last several generated frames (as a short video) instead of just one, hoping more temporal context would stabilize things.
This suggests the fix for long rollouts isn't more synthetic self-conditioning — it's periodically re-anchoring against real frames instead, which wasn't tested here but is the natural next experiment.
Cosmos3 also has an image2video mode: one real photo, one text prompt, no trajectory data of
any kind. Up to 300 frames in a single call at this resolution — no chaining, no chunk-boundary seams, and
no compounding drift by construction.
"A robotic arm with a spoon attached to its gripper scoops pasta from a plastic container on a wooden table and transfers it into a white plate. The camera is static, looking down at the table. The arm dips the spoon into the container, lifts a scoop of pasta, moves over to the plate, and drops the pasta onto the plate."
The obvious tradeoff: you give up exact trajectory control. You can't specify "the spoon must be at this precise pose at t=1.2s" — only describe the task and let the model improvise plausible motion. For generating training data specifically, rather than a precise digital twin, that's arguably a reasonable trade — real demonstrations have natural variation too.
A generated clip is only useful as training data if action labels can be recovered from it. The same
labeling pipeline from the setup — Depth Anything for per-frame metric depth, then FoundationPose with the
same spoon mesh for 6-DoF tracking — was run directly on the image2video output above.
This is the first fully closed loop in this project: real photo + one sentence → Cosmos3 generation → Depth Anything → FoundationPose → validated 6-DoF action labels, with no ground-truth trajectory anywhere in the pipeline.
Every script referenced above — Cosmos3 inference wrappers, the FoundationPose/Depth Anything labeling pipeline, and the overlay-rendering QC tool — is on GitHub, with notes on what each one does and why.
cosmos_framework.scripts.inference
Both pipelines ran in separate Docker containers on a single machine with 2× RTX A6000. Models used:
Cosmos3 /
cosmos-framework
(Cosmos3-Edge, 4B),
FoundationPose, and
Depth Anything V2
(metric, indoor).