Both clocks, every run

The arm in our kitchen example moved for 16.45 seconds. The run took 4 minutes and 22 seconds. Both numbers are correct, both are in the evidence, and only one of them says anything about the robot.

This post is about the gap between them: why it exists, why we keep it visible, and how the limits on a run turn a comparison between two models into a fair one.

Lockstep

On a real robot the world does not wait for the brain. If the model takes two seconds to decide and the cup is sliding, the decision is about a cup that has already moved. In simulation you can do something a real robot cannot: freeze physics while the model thinks, then resume. That is lockstep, and it is how every RobotGym run works today.

Lockstep answers one question honestly. Can this model do the task at all, given as long as it needs to think between motions. It does not answer the other question, whether the model could do it at speed with the world moving underneath it. Those are different measurements. A result from the first, described as if it were the second, is the kind of claim we refuse to make. When wall time appears anywhere on our site, physics paused while the model thought.

Physics also pauses while frames render. Each observation is a set of camera images at a specific instant, and the simulator holds that instant until the images exist and the model has answered.

Why sixteen seconds takes four minutes

The kitchen run had 329 control steps at 20 Hz, which is 16.45 seconds of simulated motion. It also had 10 model calls. The median call took 14.2 seconds and the 95th percentile was 28.2. Add the setup, the rendering of the cameras before each call, and the interpreter stepping through each program, and 16.45 seconds of motion took 262 seconds of wall time.

Nothing was wrong. That is what a reasoning model costs per decision, and the arm stood still, in frozen physics, while each decision was made. The general form of that sentence is in our docs: a 60-second simulated attempt can take several minutes of real time.

A learned policy shows the same gap, smaller. Our recorded bowl demonstration from an earlier release, hosted π0.5 on a LIBERO scene, ran 81 control steps, 4.05 seconds of simulated motion, in 15.46 seconds of wall time across 17 policy calls. Far faster per call, still not real time, and recorded the same way.

Two clocks on every run

So every run carries both clocks. Simulated seconds, which advance only while the controller steps physics. Wall seconds, which advance always. They are recorded per observation, per model call and in the final result, and the comparison view shows both beside the outcome, the seed and the step count, so that a change in conditions is visible rather than inferred.

The same rule holds for video. Our presentation videos preserve the recorded simulator poses at 20 Hz and re-render them in Cycles, with the thinking pauses omitted and a two-second hold on the final frame. The kitchen video is therefore 16.45 seconds of motion plus two seconds. The caption says the pauses are gone. The run page says how long they were.

While a run is live, the page shows a ticking wait timer during each model call and states that physics is paused. We added that after noticing our own status line said the arm was executing while the model was, in fact, thinking.

Limits that make a comparison fair

A comparison between two models is only fair if neither had more of anything the other lacked. Six limits are set per run and shown with it.

Two are time. Simulated seconds bound how much physics a model may spend, and because the limit is shown with the result you can see that two runs being compared had the same amount. Wall-clock seconds bound our compute and catch a stalled provider, and a run that hits them ends with time exhaustion as its recorded reason, not as a failure of the task.

Two are the model. Model calls bound how many times a model may look and decide. Tokens per response bound how long each answer may be, so a model that writes a longer program or a longer chain of reasoning does not buy itself more than its rival within one call. A run that exceeds them ends with token exhaustion, again a separate reason.

Two are motion. Program steps bound how far a single program can carry the arm before it must look again, which keeps the code door from running blind. Final settling is a fixed period after the last command during which physics continues and the model does not, so the verdict is read from a scene that has come to rest rather than from the instant the model said finish.

Success is decided by simulator state. Time exhaustion, token exhaustion, cancellation and infrastructure errors are separate termination reasons, recorded as what they are, and failed attempts stay in the history. A model calling finish is a request to stop. It does not set the verdict, and in the audits a model that called it too early was recorded as unsuccessful.

The evidence ZIP for the kitchen run has the trajectory at 20 Hz and the timestamps of every model call. Lay them side by side and you can see where the arm froze for each of the ten calls.