Solutions · Embodied AI evaluation

Put frontier models in a body

Evaluate GPT-6 Astra, Claude Fable 5 and Claude Fable 5.1 on physical tasks through one shared harness, with the cameras, primitives, budgets and judge held fixed.

Run an evaluation The harness in detail

Text and code get evaluated every day. A room with a robot in it almost never does.

What the model receivedkitchen · step 282success

The 224 by 224 pixel camera frame the model received — Policy camera · 224 px · unchanged

Spectator render of the same moment — Spectator render · not a model input

harness

Everything fixed except the model

Every model receives three RGB cameras, the measured gripper pose and the camera calibration. No object positions and no hidden success signal. It acts through the same primitives, the same 20 Hz controller and the same budgets. Differences between attempts belong to the model.

Cameras and proprioception only. Labels in images are scene content, not instructions.
move_to, move_by, rotate_down, grip, hold, look, program and finish.
A finish call is a stop request. The simulator’s predicate decides.

atom-program.pyeveryday kitchen · program 10 of 10

# Program 10 at control step 282
grip(0, steps=24)
current = pose()
move_to(current[0], current[1], current[2] + 0.12, steps=32)
checkpoint(note="Inspect whether the mug lifted with the gripper
  and is held between the fingertip pads.")

two doors

One call at a time, or one program

A model can call primitives one at a time, looking between calls, or write a short bounded program that runs several moves between calls and then checkpoints to look again. The kitchen run used ten programs. Programs run in a restricted language in a separate process with no provider keys, no network and no access to other episodes.

Tool calls with fresh images between turns.
Programs over the same primitives, sandboxed.
Saved programs replay with zero new model calls.

Usagerun d9e05837 · 10 requests

Input tokens34,352

Output tokens6,075

Cached tokens12,420

Cache writes1,380

Median inference14.2s

Failed attempts0

cost

Costs and clocks, measured

Model calls are the expensive part of a run. The harness caches the stable prefix of every request explicitly, and every request’s tokens and timing are in the archive. In the kitchen run, ten requests used 34,352 input tokens with 12,420 of them cached. Median inference was 14.2 seconds per call, and physics paused throughout.

Earlier audits measured a 21 percent input-cost cut on the one-call path and 31 percent on the program path.
Simulated and wall clocks recorded separately, never conflated.
Returned model IDs written into the metrics.

Terminationseparate from success

Reason	Recorded as	Graded
`model_finish`	stop requested by the model	yes, by the predicate
`sim_time_limit`	simulated seconds exhausted	at the limit
`model_call_limit`	call budget exhausted	at the limit
`response_token_limit`	a reply ran out of tokens	at the limit
`provider_refusal`	the provider declined	kept in history

verdicts

A judge the model cannot talk to

Success is read from simulator state against a predicate declared before the run. Termination reasons are recorded on their own, failed attempts stay in the history, and the archive holds the provider requests and responses so a reviewer can read exactly what each model was told and what it answered.

Rigid lift predicate: height reached, both fingertip pads in contact for half a second.
Provider traffic stored with returned model IDs.
Optional ideal depth assistance is labelled on the run. It is not object recognition.

See it work