What a policy sees

The model that lifted the mug in our kitchen example was never told where the mug was. It received RGB images from its cameras, the measured pose of its own gripper, and the calibration of the cameras that took the images. The mug’s coordinates were in the simulator the whole time. They never crossed the wire.

That boundary is the observation contract, and it was the first thing we decided about RobotGym’s evaluation path. This post is about what crosses it, what does not, and why we record exactly what crossed.

What crosses the wire

Every model we run, GPT-6 Astra, Claude Fable 5 or Claude Fable 5.1, receives the same packet at each turn. Three RGB cameras, including the wrist camera. The measured pose of the gripper: where the point between the fingertips is, in metres from the robot’s base, which way it points, and how far open the fingers are. The calibration of each camera, meaning its intrinsics and where it sits relative to the robot, so a model that wants to reason from a pixel to a point in space has the numbers to do it. The task in plain text. The result of its previous command.

Images and state are sampled from the same physical snapshot, so a picture and the pose that arrived with it describe the same instant.

What does not cross: object positions, contact state, the value of the success predicate, segmentation masks. Those exist in the simulator and are saved beside the episode for grading. A model that received them would be solving a different problem, and a comparison against a model that did not would be meaningless.

There is one optional assist, and it is labelled. Ideal depth assistance lets a model ask for the depth of the visible surface at a pixel it chooses. It is a lookup, frozen and recorded per run. It is not object recognition and it is not a grasp planner. A real rig has a depth camera and calibration, so we consider this sensor math rather than added capability. It is still switched on or off per run, and the run says which.

What sits under one target

A language model does not steer an arm. It names a place for the hand to go. Everything between that sentence and a moving motor is deterministic code, and it is the same code for every model.

Read the stack from the bottom. A motor needs a torque about a thousand times a second. A joint servo produces that torque from the gap between where the joint is and where it was told to be. Inverse kinematics turns a desired hand pose into a set of joint angles. A waypoint generator, running at 20 Hz, splits one requested move into a stream of small hand poses so the motion is smooth and stays inside speed limits. The model supplies the target at the top.

One target from the model becomes a stream of waypoints, each solved into joint angles, each tracked by servos over many ticks. The model spoke once. When someone says a language model controlled a robot, this is what happened: it chose the destination and textbook engineering did the rest. We keep that stack identical across models so that when two of them differ on a scene, the difference is in what they decided and not in what was underneath them.

Achieved, not commanded

The pose in the packet is the pose the arm reached, not the pose it was asked to reach. This sounds like a detail. It is the difference between a run that recovers and a run that drifts.

A command is a target. The arm may lag behind it, be stopped by the countertop, or reject a waypoint it cannot reach. If the next decision starts from the commanded pose, it starts from a place the arm never went, and every error compounds into the next. Starting from the measured pose means the model sees what actually happened, including the times its plan did not work.

The gripper is the clearest case. When the fingers close on a mug they stop early. The commanded gap is smaller than the measured gap, and that difference is the squeeze. Report the command as the state and the model believes the gripper closed on nothing. Overwrite the command with the measurement while the model is thinking and the grip goes slack. We report the measured opening and hold the last commanded target through every pause. A gripper that quietly opens during inference is a bug in the platform, not a decision by the model.

The pixels the model saw

Each run saves two kinds of image. The spectator view is for you: a camera placed to make the scene legible. For our public examples we go further and re-render the recorded poses in Cycles with presentation lighting. The other kind is the exact PNGs the model received, byte for byte, at the resolution it received them.

They are different pictures. In a custom scene the model’s frames come from MuJoCo’s own renderer, and in the starter office scene they are Cycles frames, but in neither case are they the video you watch afterwards, and the presentation lighting is never a policy input. When a grasp misses, the question is what the model saw at that turn, and the answer has to be the actual file, not a nicer view of the same instant. The evidence ZIP keeps both, side by side and labelled.

Both clocks are recorded with every observation too. Simulated time is the physics clock, and it advances only while the arm moves. Wall time is ours, and it keeps running while the model thinks and while frames render. Physics pauses during rendering and inference, so a 60-second simulated attempt can take several minutes of real time. We record both on every observation and in every result, and we never report one as the other. That deserves its own post, and it will get one.

The mug’s position is in the evidence ZIP, inside the saved simulator state, for anyone grading the run. It was never in the prompt.