A brief, high-level look at what I'm working on right now — teaching a robot arm to track its own progress through a multi-step task, and understanding how that compares to a general-purpose vision-language-action model.
Most imitation-learning policies decide their next action from a short window of recent observations. That's fine for a single pick-and-place, but it breaks down on tasks that revisit the same objects more than once — scoop from box A, then box B, then back to box A again, for instance. A brief window of frames looks nearly identical on the first and second visit to box A; the only thing that actually distinguishes them is how much of the task has already happened. A policy with no memory of its own history has no way to make that distinction.
I'm building a diffusion policy with an explicit, growing memory of the episode so far: every past frame is encoded and kept in a memory bank, and at each control step an attention mechanism retrieves whatever part of that history is relevant to the current decision. That memory summary, combined with the current observation, conditions a diffusion head that generates the next chunk of robot motion.
Training data comes from a two-stage collection process on a Kinova Jaco2 arm: a human demonstrates the task once by teleoperation, and the robot then autonomously replays that same trajectory itself. Training on the robot's own replay — rather than the original human-hand footage — means what the policy sees during training matches what it sees at deployment.
Alongside the memory-diffusion approach, I'm also finetuning and testing a pretrained vision-language-action model on the same long-horizon task, to understand where a strong generalist policy holds up and where it needs task-specific help — memory of episode progress being one candidate gap.
Actively training and testing several variants of the memory-diffusion policy — different camera configurations and a synthetic-viewpoint data augmentation step — with real-hardware evaluation ongoing on the Kinova Jaco2 arm. This page will be updated as the work progresses; detailed write-ups will follow once results settle.