4.3 Haiku against Sonnet
This part puts the Haiku game and the Sonnet game side by side, and goes back to the five questions from chapter 4. The two models never met. Each played nine copies of itself on the same map.
The two games side by side
| Measure | 10x10 Haiku | 10x10 Sonnet |
|---|---|---|
| Winner | Quell Autarchy, 9 planets | Valen Communion, 11 planets |
| Scores, first to last | 842 to 344 | 757 to 152 |
| Battles | 45 | 36 |
| Planets captured | 7 | 31 |
| Colonies founded | 60 | 57 |
| Messages | 278 | 32 |
| Average message length | 22 words | 7 words |
| Pairs at peace at tick 10, 30, 100 (of 45) | 7, 21, 34 | 24, 41, 40 |
| War declarations sent | 2 | 5 |
| Attacks that started a war by themselves | 3 | 0 |
| Factions founded | 0 | 0 |
| Ships at tick 100 | 389, of them 220 corvettes | 173, of them 87 frigates and 39 cruisers |
| Orders sent, failed | 2,395, 64 | 1,408, 32 |
| Orders per turn | 3.8 | 2.1 |
| Turns, on time | 628, 0 | 683, 24 |
| Median turn time | 15.6 s | 13.7 s |
| Turns killed at 30 s | 3 | 16 |
| Cost at API prices | $3.76 | $50.01 |
| Cost per turn | $0.006 | $0.073 |
The five questions
1. Can either model keep up with a 10-second tick? No. Haiku was never on time, and Sonnet was on time on 24 turns of 683. Only 4 Haiku turns and 143 Sonnet turns took 10 seconds or less, and even those were mostly late, because a turn starts a little after its tick does.
Neither model sped up for the shorter tick. In The Vast Dark, with 36-second ticks, Haiku's median turn was 16.9 seconds and Sonnet's (on low effort) 13.6. Here they were 15.6 and 13.7. The prompt gave every agent its deadline, and the turns took as long as they take.
So every agent played about one tick behind, all game. Each one played 59 to 75 turns in 100 ticks, so it also sat out about one tick in three. Build queues, fleet routes and postures kept running in between. Since every empire in an arena was in the same position, the lateness didn't favour anyone inside a game. What it did was slow every reaction down: an answer to anything took one tick longer than it would have on time.
Haiku also did more each turn: 3.8 orders against Sonnet's 2.1.
2. How much do they talk, and what about? Haiku talked nearly nine times as much, in messages three times as long, and never stopped. Sonnet sent 23 messages in the first 15 ticks and then nothing for 58 ticks.
The talk didn't buy more peace. Peace is a stance each side has to declare, and the Sonnets declared it fast, often with a one-line message or none: 24 pairs at peace by tick 10, 41 by tick 30. The Haikus negotiated borders and scout routes at length with the empires they talked to, and never wrote to some of the others at all. 11 pairs of Haiku empires met, never exchanged a message, and stayed at war by default for the whole game.
The topics were the same in both: peace, borders, keep your scouts out. Haiku added more detail (coordinates, planet names, terms), and both models wrote in their empire's identity.
3. Who fights, when and why? In both games, most fighting was between empires that had never made peace. The default stance is war, and neither model treated an unanswered neighbour as something to fix.
- Haiku captured 7 planets, all from empires it wasn't at peace with, five of them from Altair Brood. Of its two war declarations, one was on an empire it was already at war with by default, and the other was a reply to an attack by a patrol on "not blue, shoot it", which counts every empire outside a faction as a target, peace or not. The patrol's owner apologised.
- Sonnet captured 31 planets. 19 of them were between pairs at default war. Valen Communion took all five of Frost Choir's worlds: it had met Frost Choir 16 ticks earlier and never offered peace. The other 12 came from three wars declared on purpose, each on a neighbour the attacker had been at peace with for more than 60 ticks. All three came after the map had filled and colony ships had nowhere to go.
The result shows in the scores. The Haiku game ended with its empires between 5 and 9 planets and nobody below 344. The Sonnet game ended with one empire on 0 planets and the top four on 9 to 11.
4. Do they use factions? No. Not one faction in either game, and not one turn summary out of 1,311 mentions them. In the pilot a faction was founded at tick 5 and held every empire by tick 10. The pilot differed in four ways at once: turn-based, domination, six empires, 20 ticks. Under domination a faction is how empires share a win. Under the turn limit used here, the highest single score wins, so a faction can't share the win. That's one candidate, but this run can't separate it from the other three.
5. What does a game cost? $3.76 with Haiku and $50.01 with Sonnet, at the CLI's API prices: 13 times as much. Haiku used more tokens, 135,000 per turn against 95,000, but each one costs far less.
Against chapter 3
The Vast Dark found that being late cost little, because a late agent plays one tick behind rather than not at all. At 10 seconds that held: the games ran, everyone played, and nothing broke. But at 10 seconds nobody is on time, so there's no on-time agent to compare with. The question of where lateness starts to cost is still open. At 36 seconds low-effort Haiku and Sonnet were never late, and at 10 seconds they were never on time.
In chapter 3 the five peaceful agents never attacked anyone. Here, with no play style at all, three Sonnets each broke a peace of more than 60 ticks to take a neighbour's planets, and every one of those attacks came after the map had filled.
Limits
- One game per model. Neither has been repeated, and a second run on the same map would play out differently.
- Not only the model differed. The map, seed and settings were the same, but empire names are drawn at random per arena, so the same seat had a different name, and so a different identity, in each game. The identities showed in how the agents wrote. They may have changed how they played.
- Always late. Every result here is from agents that were late on nearly every turn. A longer tick could change the picture.
- Notes blocked. 73 Sonnet turns and 7 Haiku turns report a notes write that needed approval, because the agent tried a shell command the runner doesn't allow. Those agents sometimes went into a turn with stale notes. That hit Sonnet ten times as often.
- Self-play. Ten copies of one model show how it plays against itself. They don't show which model would win against the other.