bundlebox · benchmarks
Two arms on the same checkout, the same issue text and the same estimator. Bare is what an agent does with a shell: grep the tree, read what came back, open the top files. bundlebox is one bb pinpoint brief. Neither arm called a model, so none of these numbers is a guess about one.
SWE-bench Verified · 100 instances · 2026-09-22 · 967.3s
bundlebox's own findings · 12 tasks · 2026-09-22 · 19.9s
context per task
67.5%
less than bare — 8.7k → 2.8k tokens
gold files located
64.3%
in the packed scope, against 24.6% bare
cost per 1k tasks
$29.42
saved on Opus 5 at list price
tasks per 1M tokens
352
against 114 bare — 3.1×
Localisation is a file-level score against the maintainer's own patch. It is not a resolve rate: no model was called and no test was run.
SWE-bench Verified · 100 instances
Same issue text, same checkout, same estimator. Bare greps the tree, reads what came back, then opens an 80-line range in each of the top 10 files; bundlebox answers with one bb pinpoint brief. Averaged over 100 instances and over this repository's own 12 open findings.
67.5% less on SWE-bench (3.1×) and 79.7% less here (4.9×). Both arms are token counts over text on disk. Neither called a model.
| suite | bare / task | bundlebox / task | saved | ratio |
|---|---|---|---|---|
| SWE-bench Verified | 8.7k | 2.8k | 67.5% | 3.1× |
| bundlebox findings | 23.7k | 4.8k | 79.7% | 4.9× |
derived · list prices as of 2026-09-13
The measured token counts above, multiplied by each model's published input price. Derived, not measured: no request was sent. One task, read once, no cache.
Per 1,000 tasks that is $43.59 against $14.17 on Opus 5 — $29.42 saved. Prices: vendor pricing pages (Anthropic, OpenAI, Google), see docs/prices.md.
| model | bare / task | bundlebox / task | bare / 1k tasks | bundlebox / 1k tasks |
|---|---|---|---|---|
| Opus 5 | 4.4¢ | 1.4¢ | $43.59 | $14.17 |
| Sonnet 5 | 1.7¢ | 0.57¢ | $17.43 | $5.67 |
| Haiku 4.5 | 0.87¢ | 0.28¢ | $8.72 | $2.83 |
SWE-bench Verified · 126 gold files
Scored against patch, the fix that ships with the dataset. In scope means the brief budgets the file to be read; named means it is also pointed at as a ranked candidate. This is a localisation score, not a resolve rate.
64.3% of gold files are inside the packed scope against 24.6% for the bare arm, and 83.3% are at least named. Whole instances: bundlebox has every gold file in scope on 61 of 100 against 26 bare — and the bare arm read 10 whole files to get there.
| measure | bare Claude | bundlebox Claude |
|---|---|---|
| gold files recalled | 24.6% (31/126) | 64.3% (81/126) |
| gold files named | — | 83.3% (105/126) |
| instances with every gold file | 26/100 | 61/100 |
| instances with at least one | 30/100 | 70/100 |
| files put in front of the model | 10 read whole | 6.8 in scope (avg) |
measured wall-clock · per arm
The one chart that does not favour bundlebox. Building a brief is real local work: 8.9s against 246ms to grep and read. That is the trade — seconds of your CPU for tokens of the model's window.
Most of that is a first visit: 6.6s of the 8.9s is building the symbol space for a repository bb has never scanned, which a workspace pays once rather than per task. Against it, the window the model then has to read: 8.7k tokens bare, 2.8k packed. Prefill is the model's time rather than yours and scales with the count — set a rate and the arithmetic follows.
| stage | bare Claude | bundlebox Claude |
|---|---|---|
| assemble the window — repo never scanned | 246ms | 8.9s |
| — of which: the first-visit symbol space | — | 6.6s |
| assemble the window — workspace already scanned | 38ms | 1.6s |
| tokens the model must then read | 8.7k | 2.8k |
SWE-bench Verified · every instance
Not an average. Each x is one of the 100 instances, sorted by what the bare arm cost. The bundlebox curve stays flat because a brief is budgeted; the bare curve is whatever the tree happened to contain.
Widest gap: django__django-12663 at 10.8k against 2.1k. 0 instance(s) cost more packed than bare; they are counted in every total on this page.
derived from the measured averages
The same budget, divided by what one task costs to read.
352 against 114 — 3.1× the work from the same budget.
| budget | bare Claude | bundlebox Claude |
|---|---|---|
| 1M input tokens | 114 tasks | 352 tasks |
142 recorded runs
One measure per panel, because a single flattering run is not evidence and two suites are not two arms. The step on 1cdb60a is the baseline changing: the bare arm stopped reading matched files whole and started opening one 80-line range per hit. Everything left of it is measured against a more wasteful bare arm.
125 runs on this repository's findings, 17 on SWE-bench. Localisation is the line that moved on its own merits: 57.1% of gold files on the first recorded run, 64.3% on the last. The baseline change did not touch it.
Method
For each instance: clone the repository at the instance's own base_commit (blobless), hand bb only the issue text, and take two arms — BARE (search the tree, open the top N files whole) and PACKED (one bb pinpoint prompt). Score both file sets against the files the maintainer's own patch touched, which ships with the dataset. Both token counts come from bb's estimator over text on disk. Neither arm called a model.
Grep the checkout for the task's terms, read the search output, then open a 80-line range around the first hit in each of the top 10 files. This is the cheapest honest baseline: it is what a shell gives an agent that has no index.
One bb pinpoint prompt for the same task — the located regions, the scope it may edit, the evidence already on file, and a budget. At most 12 files enter scope.
patch. Wall-clock per arm.npm i -g bundlebox
bb bench swebench run --n 100 # the public instances above
bb bench run # this repository's open findings
node scripts/benchmark-page.mjs # rebuild this page from bothThe run writes every instance id it used, so the number can be reproduced or contradicted.