AI Coding MPG · Report No. 1

What AI coding agents cost per solved task, on real code

56 real commits from 8 open-source projects, each redone 3 times by 5 agent setups in a sandbox, judged by the project's own hidden tests. Pre-registered and frozen before the main run (after a $1.79 pilot), and every record is public.

984agent runs
8 · 56repos · commits
$121total cost, at list prices
3 of 5comparisons with a finding

The findings

Every setup is compared with Claude Code on Sonnet 5.5, except the last chart, which compares a fresh session with continuing the session. The square shows the result overall with its 95% interval, and each dot is one repo. Left of the dashed line means lower (cheaper or fewer tokens), right means higher.

Haiku 5.5 vs Sonnet 5.5: cost per solved task findingAbout 6× cheaper per solved task, solving almost as much (49% vs 51%).
0.1×0.25×0.5×1×2×4×click: 0.11×lint-staged: 0.11×homebridge: 0.11×pyinfra: 0.15×axios: 0.16×litestar: 0.18×starlette: 0.19×recharts: 0.24×overall 0.17×
Opus 5.5 vs Sonnet 5.5: cost per solved task findingSolves more (61% vs 51%) but costs about 2.6× more per solved task.
0.1×0.25×0.5×1×2×4×click: 1.04×lint-staged: 1.38×homebridge: 2.31×axios: 2.40×starlette: 2.70×pyinfra: 2.71×recharts: 3.58×litestar: 3.93×overall 2.61×
Codex (gpt-6.1-sol) vs Sonnet 5.5: cost per solved task not shownSolved a bit more (57% vs 51%); cost per solved task about the same. No difference shown.
0.1×0.25×0.5×1×2×4×click: 0.47×lint-staged: 0.47×homebridge: 0.84×axios: 1.20×pyinfra: 1.37×starlette: 1.44×litestar: 1.44×recharts: 1.64×overall 1.09×
RTK (on Sonnet 5.5) vs Sonnet 5.5: tokens per attempt not shownNo token saving shown (+1%, −6% to +9%), and the same cost. RTK fired in 163 of 168 runs.
0.1×0.25×0.5×1×2×4×recharts: 0.90×starlette: 0.91×lint-staged: 0.96×pyinfra: 0.99×axios: 1.02×litestar: 1.03×click: 1.08×homebridge: 1.15×overall 1.01×
Fresh session for the next task vs continuing the session: cost per attempted task findingCost 13% more than continuing, despite 16% fewer tokens: the new session has to write its context to the cache again.
0.1×0.25×0.5×1×2×4×pyinfra: 1.03×recharts: 1.03×click: 1.09×axios: 1.15×litestar: 1.16×homebridge: 1.17×lint-staged: 1.36×starlette: 1.37×overall 1.13×

A result counts as a finding only if it was pre-registered as the comparison's primary measure, its Holm-adjusted p is below 0.05, and its direction holds in at least 5 of the 6+ repos. Everything else is reported as "not shown", with its interval.

All numbers

SetupPrimary measureSolved (95% CI)$ per solved task$ per attemptTokens per attemptRepos in the same directionHolm pResult
Haiku 5.5vs Sonnet 5.5cost per solved task49% (42%–57%)
vs 51%
-83%
-86% to -80%
-84%
-86% to -81%
+246%
+187% to +311%
8 of 8< 0.003Finding
Opus 5.5vs Sonnet 5.5cost per solved task61% (54%–68%)
vs 51%
+161%
+93% to +228%
+217%
+178% to +267%
+97%
+69% to +133%
8 of 8< 0.003Finding
Codex (gpt-6.1-sol)vs Sonnet 5.5cost per solved task57% (50%–64%)
vs 51%
+9%
-23% to +41%
+23%
+4% to +47%
+61%
+19% to +112%
5 of 81.000Not shown
RTK (on Sonnet 5.5)vs Sonnet 5.5tokens per attempt48% (40%–55%)
vs 51%
+6%
-6% to +26%
0%
-6% to +5%
+1%
-6% to +9%
4 of 81.000Not shown
Fresh session for the next taskvs continuing the sessioncost per attempted task50% (36%–64%)
vs 46%
+3%
-16% to +20%
+13%
+5% to +26%
-16%
-26% to -5%
8 of 8< 0.003Finding

Changes are relative to the baseline, with 95% intervals from a bootstrap over repos and then commits. Dollars are each run's tokens priced at Anthropic's and OpenAI's published prices. Solve-rate intervals are Wilson intervals over runs, which ignore that runs of one repo are alike, so they are too narrow.

How it was done

What went wrong, and what we changed

All of these are dated in the pre-registration's change and deviation lists.

Added after publication: results by task difficulty

Added 2026-11-14. This was not pre-registered, and the groups were chosen after seeing the data. A reader pointed out that cost per solved task can flatter a cheap model: if it only solves the tasks every model solves, it looks efficient while never solving a hard one. So here are the same runs grouped by how hard each of the 56 tasks turned out to be, counted as passing runs across four setups (3 runs each, so 0 to 12). The groups are made from the same runs they describe, so a hard task has few passes by definition. What the table adds is which setups those few passes came from.

TasksHow manyHaiku 5.5Sonnet 5.5Opus 5.5Codex (gpt-6.1-sol)
Easy10 to 12 of 12 runs passed2776 of 8180 of 8181 of 8179 of 81
In between3 to 977 of 215 of 2114 of 2113 of 21
Hard0 to 2220 of 660 of 668 of 664 of 66

Check it yourself

Limits

This site counts page visits with GoatCounter: no cookies and no personal data. The aimpg tool itself sends nothing.