Latent actions from unlabeled video

The internet is full of video of things being done, and almost none of it comes with action labels. If world models are going to learn from observation at scale, they have to invent the controls themselves.

The idea behind latent actions is simple: between any two consecutive frames, something happened. A hand moved, a wheel turned, a door swung. We train an encoder to compress that "something" into a small discrete code, jointly with a world model that must use the code to predict the second frame from the first. If the code is too small to smuggle pixels through, the only way to lower the loss is for it to represent the action itself.

What emerges is a control interface no one designed. On game footage, the codes align almost perfectly with the controller inputs the model never saw. On robot data, they recover arm primitives. On human video, they land on something like a coarse verb vocabulary: reach, grasp, place, pour.

The practical consequence is that a policy can be trained inside the world model using these invented actions, then grounded to a real robot's actuators with a small amount of labeled data. The expensive part, learning how the world responds to being acted on, comes from ordinary video.

We are releasing our training recipe and checkpoints. The main failure modes involve wind, water, and other agents, which move on their own without being acted on.