What AI coding agents cost per solved task, on real code
56 real commits from 8 open-source projects, each redone 3 times by 5 agent setups in a sandbox, judged by the
project's own hidden tests. Pre-registered and frozen before the main run (after a $1.79 pilot), and every record is public.
984agent runs
8 · 56repos · commits
$121total cost, at list prices
3 of 5comparisons with a finding
The findings
Every setup is compared with Claude Code on Sonnet 5.5, except the last chart, which compares a fresh session with continuing
the session. The square shows the result overall with its 95% interval, and each dot is one repo. Left of the dashed line means lower
(cheaper or fewer tokens), right means higher.
Haiku 5.5 vs Sonnet 5.5: cost per solved task findingAbout 6× cheaper per solved task, solving almost as much (49% vs 51%).Opus 5.5 vs Sonnet 5.5: cost per solved task findingSolves more (61% vs 51%) but costs about 2.6× more per solved task.Codex (gpt-6.1-sol) vs Sonnet 5.5: cost per solved task not shownSolved a bit more (57% vs 51%); cost per solved task about the same. No difference shown.RTK (on Sonnet 5.5) vs Sonnet 5.5: tokens per attempt not shownNo token saving shown (+1%, −6% to +9%), and the same cost. RTK fired in 163 of 168 runs.Fresh session for the next task vs continuing the session: cost per attempted task findingCost 13% more than continuing, despite 16% fewer tokens: the new session has to write its context to the cache again.
A result counts as a finding only if it was pre-registered as the comparison's primary measure, its Holm-adjusted p is
below 0.05, and its direction holds in at least 5 of the 6+ repos. Everything else is reported as "not shown", with its interval.
All numbers
Setup
Primary measure
Solved (95% CI)
$ per solved task
$ per attempt
Tokens per attempt
Repos in the same direction
Holm p
Result
Haiku 5.5vs Sonnet 5.5
cost per solved task
49% (42%–57%) vs 51%
-83% -86% to -80%
-84% -86% to -81%
+246% +187% to +311%
8 of 8
< 0.003
Finding
Opus 5.5vs Sonnet 5.5
cost per solved task
61% (54%–68%) vs 51%
+161% +93% to +228%
+217% +178% to +267%
+97% +69% to +133%
8 of 8
< 0.003
Finding
Codex (gpt-6.1-sol)vs Sonnet 5.5
cost per solved task
57% (50%–64%) vs 51%
+9% -23% to +41%
+23% +4% to +47%
+61% +19% to +112%
5 of 8
1.000
Not shown
RTK (on Sonnet 5.5)vs Sonnet 5.5
tokens per attempt
48% (40%–55%) vs 51%
+6% -6% to +26%
0% -6% to +5%
+1% -6% to +9%
4 of 8
1.000
Not shown
Fresh session for the next taskvs continuing the session
cost per attempted task
50% (36%–64%) vs 46%
+3% -16% to +20%
+13% +5% to +26%
-16% -26% to -5%
8 of 8
< 0.003
Finding
Changes are relative to the baseline, with 95% intervals from a bootstrap over repos and then commits. Dollars are each run's
tokens priced at Anthropic's and OpenAI's published prices. Solve-rate intervals are Wilson intervals over runs, which ignore that runs of one repo are alike, so they are too narrow.
Commits: the first 7 per repo committed on or after every model's training cutoff (2026-07-01) whose tests fail before the change and pass after it, twice each. 10 of the 56 were written before that date; see below. Every candidate checked is listed, with its fingerprint.
Task: the commit message plus the names the hidden tests import. The commit's own tests decide, and any edits an agent makes to them are undone first.
Sandbox: each run starts from a fresh copy without git history. While the agent works, the only network is the model's API (plus localhost, for the project's own test servers).
Setups: Claude Code 2.1.287 on Sonnet 5.5, Haiku 5.5 or Opus 5.5; Codex 0.159.2 on gpt-6.1-sol; RTK 0.51.0 on Sonnet 5.5.
Fresh-session pairs: task 1 in a session, then task 2 either continued or started fresh with a one-line summary.
Statistics: 5 primary comparisons, Holm-corrected, with a headline only if the direction holds in at least 5 of the 6+ repos. The pre-registration is
frozen on GitHub.
What went wrong, and what we changed
Before any paid run, the harness was fixed so tests run fairly. Without these fixes, agents would have been marked as failing in 4 repos:
localhost networking for tests' own servers;
placeholder versions for builds that read git history;
coverage thresholds off;
imports from a folder;
dependencies pinned to GitHub commits.
The $1.79 pilot found that Claude Code shows Haiku 5.5 runs at about 27× the published price. Every run is priced from its tokens at the published prices. The author's Anthropic bill confirms those prices: billed at Claude Code's displayed rate, the 168 Haiku runs would have cost about $75 instead of $2.69, and the account balance shows no such charge. Claude Code's cost display for Haiku 5.5 (version 2.1.287) is wrong; the bill isn't.
During the run, both accounts ran out of credit once. Interrupted runs were redone. Codex's quota error was at first misread as aimpg's own failure, which wrongly excluded 2 commits; that was repaired before the analysis. The original file is published.
Found in a review after the analysis: 10 of the 56 commits were written before the cutoff. The rule says "authored on or after
2026-07-01", but the selection compared the date each commit was committed. All 7 homebridge commits and 3 click commits were written
in May or June and merged in July, so the models could have seen that code. Without those 10 commits, every result keeps its direction and its
finding status:
Setup
Primary measure
Change (95% CI)
Repos in the same direction
Result
Haiku 5.5vs Sonnet 5.5
cost per solved task
-82% -86% to -79%
7 of 7
Finding
Opus 5.5vs Sonnet 5.5
cost per solved task
+176% +93% to +249%
7 of 7
Finding
Codex (gpt-6.1-sol)vs Sonnet 5.5
cost per solved task
+17% -24% to +51%
5 of 7
Not shown
RTK (on Sonnet 5.5)vs Sonnet 5.5
tokens per attempt
-3% -10% to +4%
5 of 7
Not shown
Fresh session for the next taskvs continuing the session
cost per attempted task
+11% +2% to +28%
7 of 7
Finding
All of these are dated in the pre-registration's change and deviation lists.
Added after publication: results by task difficulty
Added 2026-11-14. This was not pre-registered, and the groups were chosen after seeing the data. A reader pointed out that cost per
solved task can flatter a cheap model: if it only solves the tasks every model solves, it looks efficient while never solving a hard one.
So here are the same runs grouped by how hard each of the 56 tasks turned out to be, counted as passing runs across four setups
(3 runs each, so 0 to 12). The groups are made from the same runs they describe, so a hard task has few passes by definition. What the table
adds is which setups those few passes came from.
Tasks
How many
Haiku 5.5
Sonnet 5.5
Opus 5.5
Codex (gpt-6.1-sol)
Easy10 to 12 of 12 runs passed
27
76 of 81
80 of 81
81 of 81
79 of 81
In between3 to 9
7
7 of 21
5 of 21
14 of 21
13 of 21
Hard0 to 2
22
0 of 66
0 of 66
8 of 66
4 of 66
Haiku 5.5 against Sonnet 5.5 stands. Sonnet solved 0 tasks that Haiku never solved, and Haiku solved 1 that Sonnet never did, so they solve almost the same set.
Haiku isn't cheap because it gives up early either. Its failed runs took more requests and more time than Sonnet's (13 requests and 120 seconds, against 8 and 60).
"Opus 5.5 costs 2.6× as much per solved task" hides something. On the hard tasks, Haiku solved 0 of 66 runs, Sonnet 0 of 66, Opus 8 of 66 and Codex 4 of 66.
For those tasks the choice isn't between a cheap and a dear solution. The cheaper models didn't solve them at all.
"Hard" here partly means thinly described (added later the same day). The task an agent gets is the commit message. The hard tasks'
messages had a median of 7.5 words, and the easy ones 93. Tasks described in 50 words or more were solved in
47 of 57 runs by Haiku, 51 of 57 by Sonnet and 53 of 57 by Opus. Tasks described in under 15 words were solved in
33 of 96, 33 of 96 and 47 of 96. So part of what the hard tasks test is working out what is wanted from very little, and that
is where Opus pulled ahead. Each task's description length and size are in data/tasks.json.
The task set separates the models poorly. 27 tasks were easy for every setup and 22 were too hard for most, which leaves 7 in between.
The counts on hard tasks are small, so read them as a pattern to test, not as a measured rate. Report No. 2 will report results by difficulty from the start.
Recompute every number, for free: download the data folder and run aimpg study report plan.json --runs . in it.
Rerun a comparison on your own machine: the Haiku-vs-Sonnet result for each of the 8 repos is a public record on the scoreboard; rerun one with aimpg verify --rerun <record> --repo <clone> (about $2–3 of API credit). Reruns by other people appear as Reproduced. The whole study reruns with aimpg study run plan.json.
Code:aimpg (MIT), with aimpg/replay/study.py for the statistics.
"Solved" means the project's own hidden tests pass. Code style, readability and maintainability weren't graded, so two passing solutions can still differ in quality. A cheaper model that passes the tests as often may still write code you'd rather not keep.
Haiku 5.5 was cheaper per solved task in all 8 repos, but it solved fewer tasks than Sonnet 5.5 in 3 of them (litestar 12 vs 15 of 21 runs, starlette 10 vs 12, homebridge 17 vs 18). Check it on your own code before switching: aimpg verify reruns your past commits with both models and lets your tests judge.
This covers 8 Python and TypeScript repos with good test suites, and tasks small enough to judge by tests. It may not hold for large refactors or other languages.
One configuration per tool, at default settings. RTK's maintainers were asked to review their setup and didn't reply by the deadline (discussion).
Prices change. The ratios hold only as long as the published prices behind them do.
aimpg is built by one developer using Claude Code. The author paid for every run and has no affiliation with any vendor tested.
This site counts page visits with GoatCounter: no cookies and no personal data. The aimpg tool itself sends nothing.