Platform · Evaluation

Test it in simulation, with a verdict you did not write

Run your task against the twin with declared budgets. The simulator grades the outcome, records why the run ended and keeps every attempt. Start with a built-in model, or talk to us about connecting your own policy.

Run an evaluation Run limits in the docs

The policy asks to stop. The simulator decides whether it succeeded.

Primitivesshared harness · same for every model

move_to(x, y, z, open, steps)   absolute world target · ≤ 8 mm per axis per step
move_by(dx, dy, dz)             relative move from the achieved pose
rotate_down(yaw)                point the gripper down at a yaw
grip(open)                      0 closed to 1 open
hold(seconds)                   wait and let contact settle
look()                          checkpoint · fresh images from the cameras
program(code)                   a short bounded program over the same calls
finish(note)                    a stop request, not a verdict

External camera view the policy received, 256 by 256 pixels — External camera · policy view · 256 px

Wrist camera view the policy received, 256 by 256 pixels — Wrist camera · policy view · 256 px

harness

What every policy gets

Three RGB cameras, the measured gripper pose and the camera calibration. Primitives it can call one at a time or compose into a short bounded program. Underneath, one 20 Hz waypoint generator, inverse kinematics and joint servos. The controller is the same for every run, so differences belong to the policy. Today the harness drives the twin with GPT-6 Astra, Claude Fable 5 or Claude Fable 5.1 at the controls.

No object positions and no hidden success signal. Cameras and proprioception only.
Successive targets move at most 8 mm per axis per control step. Correction is capped so a lagging arm catches up without lunging.
Optional ideal depth assistance: a depth reading at one chosen pixel, frozen and labelled per run.

Run limitsdefaults · ranges · set per run

Simulated60s · 1 to 300

Wall clock600s · 30 to 1,800

Model calls161 to 64

Response tokens8,1921,024 to 32,768

Program steps12,000100 to 50,000

Final settling2.0s · 0 to 5

model          claude-fable-5-1
layout         offset
image_history  2          1 to 4 observations keep their images
depth          off
task           "Stack the purple cylinder on the red can."
success        decided by the simulator, shown before you start

budgets

Declare the budget

Limits are set per run and shown with the result. Physics pauses during rendering and inference, so a sixty-second simulated attempt can take several minutes of real time. We show both clocks and never call the wall clock a benchmark. In the recorded kitchen run, 16.45 seconds of simulated motion took 4 minutes 22 seconds of wall time, with a median of 14.2 seconds per model call.

Simulated seconds, wall-clock seconds, model calls, tokens per response, program steps and final settling.
Final settling is a declared hold after the model finishes or a limit is hit, so an object that slips is counted as slipped.
Two layouts for the office starter, original and shifted, so a model cannot rely on remembered positions.

metrics.jsonfields from the current release · kitchen values

{
  "success": true,
  "termination": "model_finish",
  "control_steps": 329,
  "sim_seconds": 16.45,
  "wall_seconds": 262.02,
  "model_requests": 10,
  "final_settle_seconds": 2.0,
  "grade": { "lift_m": 0.082, "required_lift_m": 0.080,
             "left_pad_contact": true, "right_pad_contact": true },
  "limits": { … as declared before the run },
  "returned_models": ["gpt-6-astra"]
}

verdicts

The simulator decides

A model saying it is done is a stop request. Success is read from simulator state against the predicate shown before the run began. When a run ends for another reason, that reason is recorded on its own, and the attempt stays in the history. The model IDs each provider returned are written into the metrics, so you know what actually answered.

Custom scenes use a rigid-object lift predicate: required height reached while both fingertip pads touch the object for half a second.
Termination reasons are separate fields: finish, simulated time, call budget, token budget, cancellation, infrastructure error.
Failed attempts are never removed. History shows every run on the scene.

Recorded trialspublic evidence

Recorded trial	Controller	Steps	Calls	Simulated	Result
Kitchen mug lift rebuilt from one photograph	Atom Code · GPT-6 Astra	329	10	16.45 s	success · lift 8.2 cm
Bowl onto plate LIBERO spatial · seed 7 · earlier release	π0.5 · hosted	81	17	4.05 s	success

compare

Compare attempts on one scene

Run the same task with each model and read the attempts side by side: the frames each one received, the commands it sent, the notes it wrote and how each run ended. Two recorded trials are public today. The π0.5 run comes from an earlier release and stays as a recorded demonstration.

GPT-6 Astra, Claude Fable 5 and Claude Fable 5.1 in the current release.
Every attempt keeps its spectator view, model frames, commands, transcript and verdict.
Saved programs replay with zero new model calls on the same scene and seed.

See it work