Declared, not assumed

A wrist camera whose position is measured from the wrist flange instead of from the fingertips shifts every point the model sees by the length of the gripper. Nothing crashes. The arm goes somewhere plausible and wrong, and the run is recorded as a policy failure. We have lost time to that class of mistake and to its cousins: a rotation written in one convention and read in another, a camera principal point with the opposite sign to the library everyone assumes, a gripper whose closed reading nobody wrote down.

Every one of those bugs is an assumption that could have been a declaration. This post is about the conventions we build RobotGym against so that they are declarations. It describes how we work, not how anyone else should.

Five contracts

The platform is held together by five contracts. Everything a robot, a policy, a task or a recording does is declared against one of them.

• The embodiment description is the datasheet a robot hands the platform: its joints and their limits, its gripper and what its open and closed readings are, its cameras and their calibration, its controller and control rate, the action families it accepts, and the frames and units it uses.
• The action contract says what a command means: which family, in which frame, in which units, with which limits and which gripper semantics.
• The observation contract says what a policy receives: which images, which state fields, their units and frames, the calibration, and the instant they were sampled.
• The episode contract says what one attempt is: the reset and seed, the limits, the success predicate, the clock mode and the set of termination reasons.
• The recording contract says what gets saved, in what layout, with what provenance and license.

A policy declares what it needs from the action and observation contracts. An embodiment declares what it offers. Nothing is inferred from the length of an action array, from the direction a number moves, or from what the last person to touch the code remembers. Units are metres, radians and seconds. The quaternion order is written down, XYZW, rather than assumed. The state a policy receives is the achieved state, never the commanded one, and the point called the hand is the point between the fingertips.

Each contract is versioned, and a run freezes the versions it ran under, together with the hashes of the scene and the source and the model identifier the provider actually returned. A run from months ago has to remain interpretable without anyone’s memory.

Both forms of every action

An action passes through three forms on its way to the motors. The policy emits it in canonical form, which for a language model is an absolute hand pose and a gripper fraction. The adapter converts it to the native command the controller executes. The arm then does something, and that is the achieved state.

We record all three. The transcript holds what the model asked for. The command receipts hold what the controller was sent, with identifiers and revisions. The trajectory holds what the arm did, at 20 Hz.

The reason is that recording one form and deriving the others later is how datasets rot. The adapter changes version, the derivation changes with it, and a corpus that looked consistent is now two corpora with the same file names. Someone with the same robot stack wants the native command to reproduce a motion exactly. Someone training across robots wants the canonical form. The gap between the command and the achieved state is itself the most useful signal in the file. Keeping all three costs disk. Losing one costs the dataset.

Primitives that compile down

A language model does not emit twenty small vectors a second. It names a destination and a condition. So the interface we give it is a set of primitives: move_to, move_by, rotate_down, grip, hold, look (also called checkpoint), program and finish. Same names and same meanings on every robot that has an adapter, with the robot-specific behaviour inside the adapter and not in the prompt.

Each primitive compiles into a bounded chunk of the same raw actions a learned policy would emit, through the same 20 Hz waypoint generator, the same inverse kinematics and the same joint servos. Whether the model calls one primitive per turn or writes a short program over several, the controller cannot tell the difference. That is what lets a result from one door be set beside a result from the other, and it is why the saved programs in our audits replayed with bit-identical actions.

There is a line here that we hold. Anything that names an object, a hypothetical locate(mug) or pick(mug), adds capability the model did not have, and a run that used it is a measurement of a different system. Our one assist, ideal depth assistance, is a visible-surface depth lookup at a pixel the model chooses. It is sensor math rather than perception, and even so it is frozen and labelled per run so that nobody compares a run that had it with one that did not.

Admission before compute

Conventions that are not enforced are suggestions. The enforcement point is the moment a run is requested.

Before any GPU time is spent, the platform checks that the policy’s declared needs match what the embodiment offers, that the requested robot has a tested adapter, and that the configuration is one it can run. A mismatch returns an error that says what mismatched, and nothing runs. A match freezes the scene, the controller, the prompt and the model, reserves a slot, and starts. The server rejects a robot mismatch before the queue ever sees the request.

This is also why the workspace exposes one robot. Six arms times forty LIBERO tasks passed startup, camera and observation checks on our side. That is infrastructure validation, and it says nothing about whether any policy succeeds on them. The Franka Panda is the reference robot because it is the one with a tested adapter, a validated gripper mapping and recorded trials behind it. The others stay out of the menu until they can pass the same admission.

Four anti-patterns follow from all of this, and the platform is built to refuse each of them rather than to warn about them.

• Padding every robot’s actions into one giant vector with masks for the unused slots. It works inside a single training run and poisons an interface.
• Inferring gripper semantics from whether a number goes up or down. The reading for closed is a declared field.
• Reporting a lockstep result as real-time performance. Both clocks are recorded on every run, and the clock mode is part of the result.
• Recording only the canonical action and regenerating the native command later.

The wrist camera’s transform is now recomputed at every observation and saved beside the images it produced. Nothing in RobotGym is inferred from the length of an action array.