bundlebox · benchmarks
Two arms on the same checkout, the same issue text and the same estimator. Bare is what an agent does with a shell: grep the tree, read what came back, open the top files. bundlebox is one bb pinpoint brief. Neither arm called a model, so none of these numbers is a guess about one.
SWE-bench Verified · 100 instances · 2026-09-22 · 967.3s
bundlebox's own findings · 12 tasks · 2026-09-22 · 32.5s
context per task
67.5%
less than bare — 8.7k → 2.8k tokens
gold files located
64.3%
in the packed scope, against 24.6% bare
cost per 1k tasks
$29.42
saved on Opus 5 at list price
tasks per 1M tokens
352
against 114 bare — 3.1×
Localisation is a file-level score against the maintainer's own patch. It is not a resolve rate: no model was called and no test was run.
SWE-bench Verified · 100 instances
Same issue text, same checkout, same estimator. Bare greps the tree, reads what came back, then opens an 80-line range in each of the top 10 files; bundlebox answers with one bb pinpoint brief. Averaged over 100 instances and over this repository's own 12 open findings.
67.5% less on SWE-bench (3.1×) and 80.7% less here (5.2×). Both arms are token counts over text on disk. Neither called a model.
| suite | bare / task | bundlebox / task | saved | ratio |
|---|---|---|---|---|
| SWE-bench Verified | 8.7k | 2.8k | 67.5% | 3.1× |
| bundlebox findings | 23.1k | 4.5k | 80.7% | 5.2× |
derived · list prices as of 2026-09-13
The measured token counts above, multiplied by each model's published input price. Derived, not measured: no request was sent. One task, read once, no cache.
Per 1,000 tasks that is $43.59 against $14.17 on Opus 5 — $29.42 saved. Prices: vendor pricing pages (Anthropic, OpenAI, Google), see docs/prices.md.
| model | bare / task | bundlebox / task | bare / 1k tasks | bundlebox / 1k tasks |
|---|---|---|---|---|
| Opus 5 | 4.4¢ | 1.4¢ | $43.59 | $14.17 |
| Sonnet 5 | 1.7¢ | 0.57¢ | $17.43 | $5.67 |
| Haiku 4.5 | 0.87¢ | 0.28¢ | $8.72 | $2.83 |
SWE-bench Verified · 126 gold files
Scored against patch, the fix that ships with the dataset. In scope means the brief budgets the file to be read; named means it is also pointed at as a ranked candidate. This is a localisation score, not a resolve rate.
64.3% of gold files are inside the packed scope against 24.6% for the bare arm, and 83.3% are at least named. Whole instances: bundlebox has every gold file in scope on 61 of 100 against 26 bare — and the bare arm read 10 whole files to get there.
| measure | bare Claude | bundlebox Claude |
|---|---|---|
| gold files recalled | 24.6% (31/126) | 64.3% (81/126) |
| gold files named | — | 83.3% (105/126) |
| instances with every gold file | 26/100 | 61/100 |
| instances with at least one | 30/100 | 70/100 |
| files put in front of the model | 10 read whole | 6.8 in scope (avg) |
measured wall-clock · per arm
The one chart that does not favour bundlebox. Building a brief is real local work: 8.9s against 246ms to grep and read. That is the trade — seconds of your CPU for tokens of the model's window.
Most of that is a first visit: 6.6s of the 8.9s is building the symbol space for a repository bb has never scanned, which a workspace pays once rather than per task. Against it, the window the model then has to read: 8.7k tokens bare, 2.8k packed. Prefill is the model's time rather than yours and scales with the count — set a rate and the arithmetic follows.
| stage | bare Claude | bundlebox Claude |
|---|---|---|
| assemble the window — repo never scanned | 246ms | 8.9s |
| — of which: the first-visit symbol space | — | 6.6s |
| tokens the model must then read | 8.7k | 2.8k |
SWE-bench Verified · every instance
Not an average. Each x is one of the 100 instances, sorted by what the bare arm cost. The bundlebox curve stays flat because a brief is budgeted; the bare curve is whatever the tree happened to contain.
Widest gap: django__django-12663 at 10.8k against 2.1k. 0 instance(s) cost more packed than bare; they are counted in every total on this page.
derived from the measured averages
The same budget, divided by what one task costs to read.
352 against 114 — 3.1× the work from the same budget.
| budget | bare Claude | bundlebox Claude |
|---|---|---|
| 1M input tokens | 114 tasks | 352 tasks |
150 recorded runs
One measure per panel, because a single flattering run is not evidence and two suites are not two arms. The step on 1cdb60a is the baseline changing: the bare arm stopped reading matched files whole and started opening one 80-line range per hit. Everything left of it is measured against a more wasteful bare arm.
131 runs on this repository's findings, 19 on SWE-bench. Localisation is the line that moved on its own merits: 57.1% of gold files on the first recorded run, 63.3% on the last. The baseline change did not touch it.
DeepSWE
Everything above is a context measurement: what goes in the window, and whether the right files are in it. It is deliberately not a resolve rate. DeepSWE is a resolve rate, on 113 long-horizon tasks with program-based verifiers — the measurement this repository keeps saying it does not make.
So the board is here as published, the column bundlebox would act on is marked, and the size of run it would take to say anything is worked out. No bundlebox row is drawn, because none has been measured. The arm is written and the harness is validated; what is missing is the trials.
published · 113 tasks · 2026-09-22
A real resolve rate on 113 long-horizon tasks, pass@1, single attempt, all on mini-swe-agent, for consistency across models. These are Datacurve's measurements, not ours — read off the public board and reproduced nowhere in this repository. Anthropic models are marked.
The board's own reading is that leading coding benchmarks are saturating at the frontier: the top nine sit inside eight points of each other with intervals of ±1 to ±6, so most of that ordering is not separable. claude-sonnet-5 scores 54% ±4.
The hollow green row is not a result and not a prediction. bundlebox has never been run on this benchmark. It marks the bar: on all 113 tasks the smallest gain separable from 54% is 10.1-15.4 points depending on how often the two arms disagree, so anything below about 67% would be indistinguishable from changing nothing. It is drawn so the empty space is visible rather than implied.
| model | score | $/task | tokens/task | steps |
|---|---|---|---|---|
| gpt-6-astra | 74% ±3 | $4.43 | 30k | 29 |
| gemini-3.8-flash | 74% ±1 | $2.36 | 143k | 166 |
| claude-opus-5 | 74% ±4 | $11.84 | 118k | 99 |
| gpt-5.6-sol | 73% ±3 | $6.46 | 60k | 61 |
| claude-fable-5 | 70% ±3 | $13.41 | 80k | 68 |
| glm-5.3 | 69% ±3 | $3.99 | 80k | 124 |
| kimi-k3 | 69% ±5 | $4.65 | 81k | 98 |
| grok-4.6 | 67% ±2 | $3.45 | 50k | 70 |
| gpt-5.6-luna | 67% ±4 | $0.61 | 73k | 102 |
| gpt-5.5 | 67% ±6 | $7.23 | 46k | 82 |
| gemini-3.7-flash | 65% ±3 | $2.03 | 94k | 117 |
| glm-5.3-flash | 63% ±4 | $0.24 | 73k | 123 |
| deepseek-v4-pro | 63% ±6 | $1.67 | 106k | 155 |
| claude-opus-4.8 | 59% ±2 | $13.22 | 135k | 120 |
| qwen3.8-max | 57% ±3 | $3.73 | 95k | 111 |
| muse-spark-1.2 | 55% ±2 | $3.70 | 99k | 101 |
| claude-sonnet-5 | 54% ±4 | $26.40 | 214k | 268 |
| deepseek-v4-flash | 53% ±4 | $0.46 | 108k | 153 |
| gemini-3.6-flash | 47% ±4 | $2.21 | 96k | 117 |
| glm-5.2 | 44% ±2 | $3.92 | 78k | 129 |
| gemini-3.5-flash | 36% ±4 | $3.45 | 76k | 105 |
| claude-sonnet-5 (bundlebox) | ≥67% to be detectable — never run | — | — | — |
published · the axis bundlebox acts on
bundlebox changes neither the model, the scaffold nor the task. It changes what goes in the window. So this is the column where it could act at all — and it is the column claude-sonnet-5 is worst on.
claude-sonnet-5 spends 214k tokens and $26.40 across 268 steps per task — 2.6× the median of 81k, and the dearest row on the board — to land 54%. Nothing here says bundlebox would move that. It says this is where a change would have to show up.
derived · arithmetic over the board
Both arms run the same tasks, so the test is McNemar's on the tasks where they disagree. How many that is — the discordance rate — is not known until a run happens, so this is drawn as a band across three assumptions rather than one line. Derived, not measured.
On all 113 tasks the smallest gain separable from 54% is 10.1 to 15.4 points depending on how often the arms disagree — about 13 at the middle assumption. A 10-point gain needs 118 to 275 tasks, so this corpus is borderline for ten points and adequate for fifteen. Treating the two arms as independent samples instead of pairs would put that figure near 376 and understate the benchmark badly.
| tasks per arm | 15% discordance | 25% | 35% |
|---|---|---|---|
| 10 | 14.2 pts | 23.8 pts | 33.2 pts |
| 25 | 14.2 pts | 23.8 pts | 31.6 pts |
| 50 | 14.2 pts | 19.3 pts | 22.9 pts |
| 113 | 10.1 pts | 13 pts | 15.4 pts |
| 250 | 6.8 pts | 8.8 pts | 10.4 pts |
| 500 | 4.8 pts | 6.2 pts | 7.4 pts |
| 1000 | 3.4 pts | 4.4 pts | 5.2 pts |
Method
For each instance: clone the repository at the instance's own base_commit (blobless), hand bb only the issue text, and take two arms — BARE (search the tree, open the top N files whole) and PACKED (one bb pinpoint prompt). Score both file sets against the files the maintainer's own patch touched, which ships with the dataset. Both token counts come from bb's estimator over text on disk. Neither arm called a model.
Grep the checkout for the task's terms, read the search output, then open a 80-line range around the first hit in each of the top 10 files. This is the cheapest honest baseline: it is what a shell gives an agent that has no index.
One bb pinpoint prompt for the same task — the located regions, the scope it may edit, the evidence already on file, and a budget. At most 12 files enter scope.
patch. Wall-clock per arm.npm i -g bundlebox
bb bench swebench run --n 100 # the public instances above
bb bench run # this repository's open findings
node scripts/benchmark-page.mjs # rebuild this page from bothThe run writes every instance id it used, so the number can be reproduced or contradicted.