Two doors into one controller

In the final version of our agent-path audit, the median model call took 7.01 seconds, and every motion of the arm cost one call. A pick that needs a hover, a descent, a grip and a lift is at least that many calls, and that many waits. The arm sits still through each of them.

That measurement is the reason RobotGym has two ways for a language model to drive an arm. This post describes both, what each is good for, what the audits measured, and why we ship them as one harness rather than as two products.

One controller under both

Before the doors, the room they open onto. Every model we run receives the same observation: three RGB cameras, the measured gripper pose and camera calibration. Every model has the same eight primitives: move_to, move_by, rotate_down, grip, hold, look (also called checkpoint), program and finish. And every primitive compiles to the same thing underneath: a bounded chunk of waypoints from one 20 Hz generator, solved by inverse kinematics, tracked by joint servos. Same scene, same limits, same success check decided by simulator state.

Nothing in the controller knows which door a command came through. That is deliberate, and it is the whole argument of the last section.

Door one, a call per move

The first door is the tool-call loop. The model looks at the images and the pose readout, then calls one primitive, say move_to with a target a little above the mug. The harness interpolates a straight path from the measured pose to that target within the declared speed limit, executes it, and sends back fresh images and the pose the arm actually reached. The model looks again and calls again.

This door is good at exactly what a language model is good at. It reads the scene and adjusts after every motion, and because every request and response is recorded, the transcript reads afterwards as a legible account of what it intended. When a grasp misses, the next images show the miss and the next call corrects for it.

The cost is the round trip. Every motion waits for a model, and every wait is several seconds. A trial has a model-call limit, and in the audited version that limit was 24 calls, so a model that spends calls on small corrections runs out of them before it runs out of task. Fine contact is hard too, because the model is typing coordinates it estimated from a picture.

Door two, a program per call

The second door is a program. The model writes a short program over the same primitives, in a restricted language, and submits it in a single call. The program runs locally in a separate process that has no provider keys, no network and no access to other episodes. An interpreter walks it and hands out one primitive at a time, each compiling to the same bounded chunk of waypoints as before. A program can read the measured pose at the moment it executes, so “lift from wherever you are” adapts to reality without asking the model. The model is consulted again only when the program ends, reaches a checkpoint, or fails.

Two things follow. One model call can buy several motions instead of one. And after a clean success, the sequence of programs is frozen and can be run again in fresh physics with the model switched off entirely.

The program could also delegate. In the audited version it could hand a segment of the task to π0.5, a learned policy, and those calls were counted separately from the model’s.

What the audits measured

The agent path succeeded on 3 of 6 declared development cases in its final version. Of the three failures, one was the model stopping before the task was complete, one ran out of control steps, and one ran out of model calls. All 18 declared trials across the three versions are retained, including the earlier versions that did worse.

The code path succeeded on 4 of 4 declared cases. Two of those four were bowl placements where the program delegated the grasp to π0.5, so the language model wrote the plan and a learned policy did the contact. The other two were a cup lift with π0.5 excluded, requiring an 8 cm lift with both fingertip pads in contact. All four saved programs, run again in fresh physics with no model access, succeeded with actions and joint positions bit-identical to the runs that generated them. A saved cup program ran in about ten seconds of wall time. Generating it had taken over a minute.

Explicit prompt caching measured a 21 percent reduction in input cost on the agent path and 31 percent on the code path. That is input cost only, and it did not make the reasoning faster. The longer cached prompt on the code path used more requests and more wall time on the cup, not fewer.

Now the caveats, which are the part we care about. Six cases and four cases are declared development samples, not populations. They are history, not a success rate. The custom cup scene has fixed object positions, so its two seeds do not show scene diversity. The saved-program replays are same-reset reproducibility, not adaptation: a changed scene or seed needs a new evaluation. And the code path only looks fast when the model knows where the object is. When it is unsure, its programs shrink to one or two moves followed by a checkpoint, and the code loop degenerates into the agent loop with extra syntax. The kitchen mug run was ten programs, and nine of them were that short. Generation there was not faster than the agent path. The measured wins were the replays and the hand-offs, not the first attempt.

So why one harness with both doors, rather than picking a winner. Because the two doors are good at different moments of the same task, and the model is the one that knows which moment it is in. Unsure where the mug is, it takes one step and looks. Confident, it writes the whole lift. Because a comparison between GPT-6 Astra and Claude Fable 5.1 is only fair if both had the same primitives, the same limits and the same controller, whichever door each chose. And because the evidence has to be the same shape either way: the exact images, the commands with their receipts, the transcript, the trajectory, the verdict.

Both doors open onto the same 20 Hz controller. That is the only reason a result from one can be set beside a result from the other.