Adam K Dean

Notes on software, systems, engineering, and things

3.2 Time

The first question of chapter 3: does a slow model lose because it's slow?

Method

Each tick lasts 36 seconds. An agent starts its turn as soon as it sees a new tick, so it has a little under 36 seconds to read its notes, fetch its briefing, send orders and rewrite its notes. The server logs every turn with its start and end tick and its duration. From that log:

791 turns were logged in all.

Results

Dot plot of every turn's duration for each of the ten agents, with a dashed line at the 36 second tick and a bar at each agent's median. Sonnet, Opus and Haiku on low effort sit between 10 and 30 seconds; Codex on low effort around 30; Fable and Codex on medium effort straddle the line

Agent Model, effort, style Turns Late Missed ticks Median 90th percentile Slowest Cost per turn
foxtrot Sonnet, medium, isolationist 66 0 2 13.3 s 23.8 s 108.4 s $0.058
bravo Sonnet, low, peaceful 86 0 0 13.6 s 22.2 s 33.1 s $0.060
golf Opus, low, turtle 86 0 0 15.0 s 20.1 s 28.1 s $0.142
charlie Opus, low, peaceful 86 0 0 15.7 s 22.4 s 29.9 s $0.145
alpha Haiku, low, peaceful 86 0 0 16.9 s 20.4 s 24.4 s $0.005
echo Codex, low, peaceful 86 26 (30%) 0 28.9 s 37.9 s 50.7 s 99,000 tokens
hotel Fable, low, aggressive 53 33 (62%) 3 33.4 s 45.9 s 73.2 s $0.398
delta Fable, low, peaceful 77 54 (70%) 9 34.3 s 53.8 s 101.5 s $0.439
india Codex, medium, supremacy 86 72 (84%) 0 36.5 s 42.5 s 51.2 s 108,000 tokens
juliet Codex, medium, antagonist 79 69 (87%) 7 37.7 s 48.9 s 68.5 s 110,000 tokens

foxtrot's slowest turn is its one timeout. Cost is the CLI's own figure at API prices for Claude; Codex reports tokens only. In all, the Claude turns came to $88.98, and Fable was $54.86 of that, 62%. The three Codex agents used 26.6 million tokens.

Findings

Model and effort set the pace. Style barely does. The agents fall into three bands. Sonnet, Opus and Haiku on low effort took 13 to 17 seconds at the median and were never late. Codex on low effort took 29. Fable on low effort and Codex on medium took 33 to 38, right on the deadline. Within a model and effort, the style made little difference: Opus took 15.0 s as a turtle and 15.7 s as a peaceful empire, and Fable 33.4 s as the aggressor and 34.3 s as the peacemaker.

Fable was the slowest Claude model. Even on low effort it was slower than Codex on low effort, and it had the widest spread: delta's turns ran from 14.6 seconds to 101.5. It was also the most expensive by far, at 75 to 80 times Haiku's cost per turn.

Being late cost little. The two agents that were late most often, india and juliet, finished first and second. Three things in the game soften lateness:

  1. A late turn's orders aren't lost. They apply on the next tick, so a slow empire plays one tick behind rather than not at all.
  2. Most orders are standing orders. A fleet keeps flying to its destination, and a build queue keeps building, whether or not the agent shows up.
  3. Planets defend themselves. Defence grids and home fleets fight without new orders.

So a 36-second tick doesn't punish a 37-second model much. It would take a much shorter tick, or a game where one tick decides a battle, for speed to matter more.

But this run can't show that speed doesn't matter. The two winners were the only Codex agents on medium effort, and two of the three agents told to conquer. (The third, Fable's aggressive empire, was eliminated first.) Speed, model, effort and style all point the same way, and with one game there's no way to separate them. What the data does show is that being late on 85% of turns didn't stop an empire from winning.

Missed ticks cluster in the slowest agents. delta missed 9 ticks and juliet 7, and the number of their turns that ran over two tick boundaries is almost the same (8 and 7). A missed tick means the empire ran on standing orders for that tick. There's no sign in the results that it hurt either of them much, but that isn't measured here.

Compared with chapter 2

The same models took these mean times per turn in the turn-based games, where nothing waited on them:

Model Chapter 2 (CLI default or medium) Chapter 3
Haiku 18 to 24 s 17 s (low)
Sonnet 11 to 12 s 15 s (low), 16 s (medium)
Opus 22 to 25 s 16 to 17 s (low)
Codex 21 to 25 s (medium) 30 s (low), 35 to 38 s (medium)

The briefing in chapter 3 is far bigger: ten empires, 70 stars and up to 27 planets each, against three or four empires on much smaller maps. So turns had more to read. Low effort made Haiku and Opus faster than before even so. Codex on the same medium effort got 50 to 60% slower, which fits a bigger briefing. Claude's cost per turn barely moved: Opus was about $0.15 a turn in chapter 2 and $0.14 now, Sonnet $0.06 in both.

Codex's token counts aren't comparable between the rounds. In chapter 2 the runner read one total from Codex's text output; now it adds up the input and output of every step from Codex's JSON events.