Adam K Dean

Notes on software, systems, engineering, and things

4.3 Haiku against Sonnet

This part puts the Haiku game and the Sonnet game side by side, and goes back to the five questions from chapter 4. The two models never met. Each played nine copies of itself on the same map.

The two games side by side

Measure 10x10 Haiku 10x10 Sonnet
Winner Quell Autarchy, 9 planets Valen Communion, 11 planets
Scores, first to last 842 to 344 757 to 152
Battles 45 36
Planets captured 7 31
Colonies founded 60 57
Messages 278 32
Average message length 22 words 7 words
Pairs at peace at tick 10, 30, 100 (of 45) 7, 21, 34 24, 41, 40
War declarations sent 2 5
Attacks that started a war by themselves 3 0
Factions founded 0 0
Ships at tick 100 389, of them 220 corvettes 173, of them 87 frigates and 39 cruisers
Orders sent, failed 2,395, 64 1,408, 32
Orders per turn 3.8 2.1
Turns, on time 628, 0 683, 24
Median turn time 15.6 s 13.7 s
Turns killed at 30 s 3 16
Cost at API prices $3.76 $50.01
Cost per turn $0.006 $0.073

The five questions

1. Can either model keep up with a 10-second tick? No. Haiku was never on time, and Sonnet was on time on 24 turns of 683. Only 4 Haiku turns and 143 Sonnet turns took 10 seconds or less, and even those were mostly late, because a turn starts a little after its tick does.

Two histograms of turn times with a dashed line at the 10 s tick. Haiku's 628 turns peak at 15 to 16 seconds and almost none fall below 10. Sonnet's 683 turns are spread from 6 to 30 seconds, with 143 at 10 seconds or less

Neither model sped up for the shorter tick. In The Vast Dark, with 36-second ticks, Haiku's median turn was 16.9 seconds and Sonnet's (on low effort) 13.6. Here they were 15.6 and 13.7. The prompt gave every agent its deadline, and the turns took as long as they take.

So every agent played about one tick behind, all game. Each one played 59 to 75 turns in 100 ticks, so it also sat out about one tick in three. Build queues, fleet routes and postures kept running in between. Since every empire in an arena was in the same position, the lateness didn't favour anyone inside a game. What it did was slow every reaction down: an answer to anything took one tick longer than it would have on time.

Haiku also did more each turn: 3.8 orders against Sonnet's 2.1.

2. How much do they talk, and what about? Haiku talked nearly nine times as much, in messages three times as long, and never stopped. Sonnet sent 23 messages in the first 15 ticks and then nothing for 58 ticks.

Two bar charts of messages sent per tick. Haiku sends up to 12 a tick in the first 15 ticks and one to five a tick for the rest of the game. Sonnet sends up to 4 a tick until tick 14, nothing from tick 15 to tick 72, and a few from tick 73 on, most of them Frost Choir's pleas

The talk didn't buy more peace. Peace is a stance each side has to declare, and the Sonnets declared it fast, often with a one-line message or none: 24 pairs at peace by tick 10, 41 by tick 30. The Haikus negotiated borders and scout routes at length with the empires they talked to, and never wrote to some of the others at all. 11 pairs of Haiku empires met, never exchanged a message, and stayed at war by default for the whole game.

The topics were the same in both: peace, borders, keep your scouts out. Haiku added more detail (coordinates, planet names, terms), and both models wrote in their empire's identity.

3. Who fights, when and why? In both games, most fighting was between empires that had never made peace. The default stance is war, and neither model treated an unanswered neighbour as something to fix.

The result shows in the scores. The Haiku game ended with its empires between 5 and 9 planets and nobody below 344. The Sonnet game ended with one empire on 0 planets and the top four on 9 to 11.

4. Do they use factions? No. Not one faction in either game, and not one turn summary out of 1,311 mentions them. In the pilot a faction was founded at tick 5 and held every empire by tick 10. The pilot differed in four ways at once: turn-based, domination, six empires, 20 ticks. Under domination a faction is how empires share a win. Under the turn limit used here, the highest single score wins, so a faction can't share the win. That's one candidate, but this run can't separate it from the other three.

5. What does a game cost? $3.76 with Haiku and $50.01 with Sonnet, at the CLI's API prices: 13 times as much. Haiku used more tokens, 135,000 per turn against 95,000, but each one costs far less.

Against chapter 3

The Vast Dark found that being late cost little, because a late agent plays one tick behind rather than not at all. At 10 seconds that held: the games ran, everyone played, and nothing broke. But at 10 seconds nobody is on time, so there's no on-time agent to compare with. The question of where lateness starts to cost is still open. At 36 seconds low-effort Haiku and Sonnet were never late, and at 10 seconds they were never on time.

In chapter 3 the five peaceful agents never attacked anyone. Here, with no play style at all, three Sonnets each broke a peace of more than 60 ticks to take a neighbour's planets, and every one of those attacks came after the map had filled.

Limits