StarSkirmish Bench v0.2: LLMs Write StarCraft Bots in One Hour

TL;DR

• StarSkirmish is a benchmark where LLMs write bots for StarCraft: Brood War
• Each LLM gets one hour to write a Protoss bot in C++, which is then evaluated against other LLMs' bots and established human written bots
• StarSkirmish tests long horizon reasoning and agentic coding as LLMs must incorporate learnings from practice games to improve their StarCraft bot

The StarSkirmish Bench evaluates how strong of a StarCraft bot can each LLM write given 1 hour of wall clock time. v0.2 updates StarSkirmish Bench v0.1 with flagship models from OpenAI, Anthropic, Google and Meta, each running in its lab's 1st party CLI harness.

An LLM’s score reflects its win rate against a roster of competitive human written bots and demo bots.

StarSkirmish Bench v0.2

1 hour runs, in its 1st party CLI harness. Scores are average win rates (0–100) of each model's 5 runs against a roster of competitive human written bots.

GPT-6 Astra

57

GPT-6.1 Sol

46

Gemini 4 Argon

39

Claude Opus 5.5

39

Claude Fable 5.1

37

Claude Sonnet 5.5

26

Muse Wonder Full

21

Claude Haiku 5.5

21

GPT-6 Luna

13

Gemini 4 Argon runs at high, the highest reasoning effort available to it; every other model runs at max.

GPT-6 Astra is the top scoring StarSkirmish Bench v0.2 LLM, a clear step ahead of GPT-6.1 Sol in second. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all scoring between 37 and 39.

StarSkirmish Bench vs. Cost

StarSkirmish Bench score against the average API cost of one 1 hour run. Cost is on a log scale.

Muse Wonder Full's pricing is based on Muse Wonder's API pricing.

GPT-6.1 Sol offers uniquely good value: it is the only LLM in the most attractive quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run.

No Signs of Saturating

The strongest human written bots are still well ahead of every model. Against Stardust, PurpleWave and BananaBrain, the best model, GPT-6 Astra, wins only 7% of its games.

Win rates vs the top human written bots

Average win rate of each model's 5 bots against top human written bots Stardust, PurpleWave, and BananaBrain.

GPT-6 Astra

7

GPT-6.1 Sol

4

Gemini 4 Argon

2

Claude Opus 5.5

2

Claude Fable 5.1

2

Claude Sonnet 5.5

<1

Muse Wonder Full

<1

Claude Haiku 5.5

<1

GPT-6 Luna

0

1 hour reasoning limits are a factor, but these matches are played on three 2 player maps. Real human written bot tournament conditions are significantly harder.

New AI Model Highlights

Google and Meta's new flagships replace v0.1's Gemini 3.8 Flash and Muse Spark 1.3: Gemini 4 Argon places 3rd, and Muse Wonder Full 7th. GPT-6 Astra and Claude Opus 5.5 were covered in v0.1.

• Gemini 4 Argon uniquely preferred writing bots via one shots. Its five runs had the fewest builds and code versions of all the traced runs.
• 41% of its practice games were against high tier opponents, second only to GPT-6 Astra, yet its bots win only 7% of their games against those tiers, to Astra's 28%.

Favorite Strategy Timing Attack

One Big PushDragoons and Zealots held at home behind cannons for one big push.

Favorite Unit

ReaverIts second best bot builds Reavers in over 70% of its games.

Video 6

Argon's one big push meets an equal army at 192 supply, and Psionic Storms win the trade

• It averages 2.5 upgrades a game, the most in the field.
• Like Muse Spark 1.3 in v0.1, it practiced almost exclusively against the easiest opponents. Only 1% of its practice matches were against tier B or harder.

Favorite Strategy Macro

Outnumber FirstFast expand into 5 to 9 Gateways, attacking only once it outnumbers the army it has seen.

Favorite Unit

Observer3 per game, the most of any model and 1.6x the field.

Zealot22 per game, as in v0.1 a Zealot-heavy army.

Video 7

Muse's Zealot flood runs down a Dragoon army

AI Model Behaviors

Each run's agent trace records how it spent its hour. The strongest predictor of a model's score is how much it practiced against the 7 strongest bots on the roster.

Win Rates vs. Practice Against Strong Bots

StarSkirmish Bench score against the number of practice games a model's median run played against the 7 strongest roster bots during the hour. The dashed line is a linear fit across the 9 models.

GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon start playing these bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Claude Haiku 5.5 and GPT-6 Luna rarely or never do. Claude Fable 5.1 is the outlier: it scores 37 while practicing almost entirely against weaker bots.

Run-to-Run Variance

Win rates of each model's 5 runs. Each dot is one run's final bot; the line spans the lowest to the highest run, the bar marks their mean (the model's Bench score) and the band one standard deviation (σ).

GPT-6 Astra

57 σ 3.4

GPT-6.1 Sol

46 σ 6.0

Gemini 4 Argon

39 σ 7.0

Claude Opus 5.5

39 σ 3.2

Claude Fable 5.1

37 σ 2.9

Claude Sonnet 5.5

26 σ 8.4

Muse Wonder Full

21 σ 3.0

Claude Haiku 5.5

21 σ 6.1

GPT-6 Luna

13 σ 5.6

Results also vary between runs of the same model. Gemini 4 Argon varies far more than the other labs' flagships: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.

Implementation Details

Objective

Each LLM writes a Protoss bot in C++ against BWAPI 4.4.0, played on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60 minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins.

The LLM starts from a skeleton bot that only mines, with the source code of the three demo bots and two standard bot libraries: FAP, a combat simulator, and BWEM, a map analyzer. Whatever is in its bot folder when the hour ends is its final bot, so it has to keep the bot compiling.

The LLM has tools to compile its bot, play practice games, and read each game's summary, transcript and its bot's own log. Practice opponents are grouped into tiers: D (the three demo bots), C and B (three mid-ranked human written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave).

Tournament Setup

45 LLM bots

13 Opponents

10,530 Games

18 Games / pair

Each model's 5 final bots play each of the 13 roster bots, the 10 competitive human written bots and the 3 demo bots, 6 times on each map, on fresh seeds. Unlike v0.1, the LLM bots don't play each other.

Harness and Reasoning

StarSkirmish Bench v0.1 ran every model in the same harness at 'high' reasoning. v0.2 runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.

Harness

v0.1 ran every model in one neutral 3rd party harness; v0.2 runs each lab's models in its 1st party CLI harness. Both versions give each model 1 hour.

v0.1

v0.2

Claude Code Claude Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5

Codex GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna

Antigravity CLI Gemini 4 Argon

Muse Code Muse Wonder Full

CLI versions: Claude Code 2.1.296, Codex 0.162.1, Antigravity CLI 1.3.5 and Muse Code 1.4.7.

Reasoning Effort

v0.1 ran every model at high reasoning effort; v0.2 runs them at max.

v0.1

Low Medium High XHigh Max

High

v0.2

Low Medium High XHigh Max

Max

Gemini 4 Argon runs at high, the highest effort supported.

Every v0.2 harness compacts the model's context at about 250k tokens. When a conversation nears that point, the CLI condenses it and the model carries on. Codex, OpenAI's CLI, for example, gives the GPT models its default 272k token window and triggers compression once 90% filled.

The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.

Scoring

v0.2 replaces v0.1's Elo ratings with plain win rates. A bot's score is its win rate over its 234 tournament games, and a model's StarSkirmish Bench score is the average of its 5 bots' scores, from 0 (every game lost) to 100 (every game won).

Because every bot plays the same fixed roster, its games don't depend on the other LLM bots in the field. They all run in parallel, and a new model's bots can be scored without replaying anyone else's. A fixed roster also keeps the bar the same over time, so scores from different model generations are more directly comparable than Elo ratings fitted within one field.

Benchmark Correlations

StarSkirmish no longer tracks the most widely reported evals. It barely moves with the Artificial Analysis Intelligence Index (ρ 0.26), where GPT-6 Astra, the top model here, only ties for 4th of these 9 models. It has also drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. Claude Sonnet 5.5 and Muse Wonder Full both rank near the top of Terminal-Bench 4.0 but near the bottom here.

Benchmark Correlation

Each model's StarSkirmish Bench score against its Vals IOI and WeirdML v3 scores, with the Spearman ρ between them. The dashed line is the best straight-line fit.

Vals IOI ρ 0.95

WeirdML v3 ρ 1.00, n = 7

WeirdML v3 scores 7 of the 9 models: Gemini 4 Argon and Muse Wonder Full have not been included at time of writing.

StarSkirmish still tracks Vals IOI, olympiad programming problems solved in C++, and WeirdML, an independent benchmark of unusual machine learning coding tasks. One reading is that the most heavily benchmarked evals have become targets that models are tuned for, while less targeted tests of long horizon, difficult coding tasks still move together.

---

作者发帖说明（HN）：

I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse