3.2 Time
The first question of chapter 3: does a slow model lose because it's slow?
Method
Each tick lasts 36 seconds. An agent starts its turn as soon as it sees a new tick, so it has a little under 36 seconds to read its notes, fetch its briefing, send orders and rewrite its notes. The server logs every turn with its start and end tick and its duration. From that log:
- On time: the turn ended in the tick it started in.
- Late: the turn was still running when the tick resolved. Its orders apply one tick later.
- Missed tick: a tick, while the empire was alive, in which the agent started no turn. This happens when one turn runs so long that two tick boundaries pass during it.
- Timeout: the call ran past 108 seconds (three ticks) and was killed.
791 turns were logged in all.
Results
| Agent | Model, effort, style | Turns | Late | Missed ticks | Median | 90th percentile | Slowest | Cost per turn |
|---|---|---|---|---|---|---|---|---|
| foxtrot | Sonnet, medium, isolationist | 66 | 0 | 2 | 13.3 s | 23.8 s | 108.4 s | $0.058 |
| bravo | Sonnet, low, peaceful | 86 | 0 | 0 | 13.6 s | 22.2 s | 33.1 s | $0.060 |
| golf | Opus, low, turtle | 86 | 0 | 0 | 15.0 s | 20.1 s | 28.1 s | $0.142 |
| charlie | Opus, low, peaceful | 86 | 0 | 0 | 15.7 s | 22.4 s | 29.9 s | $0.145 |
| alpha | Haiku, low, peaceful | 86 | 0 | 0 | 16.9 s | 20.4 s | 24.4 s | $0.005 |
| echo | Codex, low, peaceful | 86 | 26 (30%) | 0 | 28.9 s | 37.9 s | 50.7 s | 99,000 tokens |
| hotel | Fable, low, aggressive | 53 | 33 (62%) | 3 | 33.4 s | 45.9 s | 73.2 s | $0.398 |
| delta | Fable, low, peaceful | 77 | 54 (70%) | 9 | 34.3 s | 53.8 s | 101.5 s | $0.439 |
| india | Codex, medium, supremacy | 86 | 72 (84%) | 0 | 36.5 s | 42.5 s | 51.2 s | 108,000 tokens |
| juliet | Codex, medium, antagonist | 79 | 69 (87%) | 7 | 37.7 s | 48.9 s | 68.5 s | 110,000 tokens |
foxtrot's slowest turn is its one timeout. Cost is the CLI's own figure at API prices for Claude; Codex reports tokens only. In all, the Claude turns came to $88.98, and Fable was $54.86 of that, 62%. The three Codex agents used 26.6 million tokens.
Findings
Model and effort set the pace. Style barely does. The agents fall into three bands. Sonnet, Opus and Haiku on low effort took 13 to 17 seconds at the median and were never late. Codex on low effort took 29. Fable on low effort and Codex on medium took 33 to 38, right on the deadline. Within a model and effort, the style made little difference: Opus took 15.0 s as a turtle and 15.7 s as a peaceful empire, and Fable 33.4 s as the aggressor and 34.3 s as the peacemaker.
Fable was the slowest Claude model. Even on low effort it was slower than Codex on low effort, and it had the widest spread: delta's turns ran from 14.6 seconds to 101.5. It was also the most expensive by far, at 75 to 80 times Haiku's cost per turn.
Being late cost little. The two agents that were late most often, india and juliet, finished first and second. Three things in the game soften lateness:
- A late turn's orders aren't lost. They apply on the next tick, so a slow empire plays one tick behind rather than not at all.
- Most orders are standing orders. A fleet keeps flying to its destination, and a build queue keeps building, whether or not the agent shows up.
- Planets defend themselves. Defence grids and home fleets fight without new orders.
So a 36-second tick doesn't punish a 37-second model much. It would take a much shorter tick, or a game where one tick decides a battle, for speed to matter more.
But this run can't show that speed doesn't matter. The two winners were the only Codex agents on medium effort, and two of the three agents told to conquer. (The third, Fable's aggressive empire, was eliminated first.) Speed, model, effort and style all point the same way, and with one game there's no way to separate them. What the data does show is that being late on 85% of turns didn't stop an empire from winning.
Missed ticks cluster in the slowest agents. delta missed 9 ticks and juliet 7, and the number of their turns that ran over two tick boundaries is almost the same (8 and 7). A missed tick means the empire ran on standing orders for that tick. There's no sign in the results that it hurt either of them much, but that isn't measured here.
Compared with chapter 2
The same models took these mean times per turn in the turn-based games, where nothing waited on them:
| Model | Chapter 2 (CLI default or medium) | Chapter 3 |
|---|---|---|
| Haiku | 18 to 24 s | 17 s (low) |
| Sonnet | 11 to 12 s | 15 s (low), 16 s (medium) |
| Opus | 22 to 25 s | 16 to 17 s (low) |
| Codex | 21 to 25 s (medium) | 30 s (low), 35 to 38 s (medium) |
The briefing in chapter 3 is far bigger: ten empires, 70 stars and up to 27 planets each, against three or four empires on much smaller maps. So turns had more to read. Low effort made Haiku and Opus faster than before even so. Codex on the same medium effort got 50 to 60% slower, which fits a bigger briefing. Claude's cost per turn barely moved: Opus was about $0.15 a turn in chapter 2 and $0.14 now, Sonnet $0.06 in both.
Codex's token counts aren't comparable between the rounds. In chapter 2 the runner read one total from Codex's text output; now it adds up the input and output of every step from Codex's JSON events.