Benchmark: A/B test with/without shards on real open-source issues

[jonathanpopham]

Goal

Measure whether agents make better decisions with graph shards than without. Timeline: 2 days.

First result

undb #2236 — Cross-Space IDOR in Webhook and Invitation Delete/Update

• Repo: undb-io/undb (1,391 TS files, 15,039 nodes, 25,134 relationships)
• Task: Add space_id authorization checks to webhook and invitation delete/update endpoints

Both agents produced the same correct fix: 5 methods across 2 files, adding space_id WHERE clause.

	Baseline	With shards	Delta
Duration	160s	145s	10% faster
Messages	53	37	30% fewer
Solution	correct	correct	same
Token cost	unavailable (subscription billing)	unavailable	—

Install

```bash

macOS/Linux

curl -fsSL https://supermodeltools.com/install.sh | sh

Or via Go

go install github.com/supermodeltools/cli@v0.4.2

Verify

supermodel version
```

Setup

```bash
cd /path/to/your/repo

Login (gets API key)

supermodel login

First run — generates graph + shards

supermodel analyze

Start the watcher (keeps shards fresh as you code)

supermodel

Install the Claude Code hook (auto-triggers incremental on every edit)

Add to .claude/settings.json:

```
```json
{
"hooks": {
"PostToolUse": [{
"matcher": "Write|Edit",
"hooks": [{"type": "command", "command": "supermodel hook"}]
}]
}
}
```

The agent discovers the `.graph.*` shards through its normal grep/rg workflow.

Benchmark methodology

Pick 5-10 real issues from open source repos. Issues where the fix requires understanding cross-file relationships — who calls what, what depends on what, what's the blast radius. Not typos or single-file bugs.

For each issue, run two sessions:

1. Without shards: Fresh clone, no `.graph.*` files, agent works with grep only
2. With shards: Same clone, run `supermodel analyze` first, agent has graph data in the filesystem

Measure:

• Tool calls: How many grep/read operations did the agent use to navigate?
• Files touched: Did the agent find the right files?
• Fix quality: Was the fix scoped correctly? Did it account for callers/dependents?
• Time to first correct edit: How long before the agent started writing the right code?
• Tokens consumed: Total input/output tokens

Control matters

Same model, same prompt, same issue, same repo state. The only variable is whether `.graph.*` files exist in the repo. Use `--no-shards` flag or `supermodel clean` to remove shards for the baseline run.

Candidate repos and issues

Need issues that are:

• Multi-file (fix spans 2+ files)
• Require understanding call chains or dependency relationships
• Have a clear "correct" fix that can be evaluated
• On repos with 500+ files (small repos don't benefit much from graph context)

Previous A/B tests showed the strongest signal on:

• Large repos (4000+ files): 2.5x faster, 42% cheaper, correct vs broken fix (Directus)
• Security fixes: Cross-file authorization checks (undb)
• Bug fixes requiring call chain understanding: Flow trigger cache key (Directus #26929)

[coderabbitai]

🔗 Related PRs

#57 - Rewrite README: outcome-led headline, how it works, works with any agent [merged]
#76 - Fix non-deterministic shard rendering [merged]
#99 - Fix incremental update: stale callee shards + false HEAD-change on startup [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jonathanpopham]

Benchmark in progress

Setup

• CLI: v0.4.2 (with previousDomains seeding)
• API: production v0.12.0
• Agent: Claude via claude --print --max-budget-usd 2.00
• Method: For each issue, fresh clone → run baseline (no shards) → fresh clone → supermodel analyze → run with shards. Same prompt, same model, same repo state.

Issues selected

#	Repo	Issue	Files in fix	Bug type
1	directus/directus	#26991	5	Stream resource leak across middleware
2	directus/directus	#26997	4	Auth/permission bug across services
3	strapi/strapi	#25118	5	Component structure regression
4	payloadcms/payload	#15980	6	Data loss across versioning + localization
5	medusajs/medusa	#14384	20	Workflow rollback across order system

Repo sizes

Repo	TS files
Directus	~3,700
Strapi	3,496
Payload	6,074
Medusa	9,586

Prior results for reference

undb #2236 (from earlier testing):

• Baseline: 160s, 53 messages
• Shards: 145s, 37 messages (10% faster, 30% fewer messages)

Abraham's A/B tests:

• Directus (large): 2.5x faster, 42% cheaper, correct vs broken fix
• Small repos: marginal benefit

Results will be posted as each benchmark completes.

[jonathanpopham]

Result 1: directus/directus#26991 — Asset download stream cleanup

Repo: 4,250 files, 17,908 nodes, 2,580 shards

	Baseline	With shards	Delta
Duration	782s	704s	10% faster
Tool calls	31	11	65% fewer
Files changed	5	5	same

Both agents produced 5-file fixes matching the known merged PR. The shard-guided agent used 65% fewer tool calls to navigate the codebase — it found the relevant files and call chains through the graph data instead of iterative grep searches.

[jonathanpopham]

Benchmark environment details

• Model: Claude Opus 4.6 (1M context) — opus[1m]
• Claude Code: v2.1.101
• CLI: supermodel v0.4.2
• API: production v0.12.0
• All issues created after May 2025 (post model training cutoff — agent cannot have memorized fixes)

Issue	Created
directus#26991	2026-03-27
directus#26997	2026-03-27
strapi#25118	2026-01-05
payload#15980	2026-03-17
medusa#14384	2025-12-22

[jonathanpopham]

Result 2: medusajs/medusa#14384 — revertLastVersion cleanup

Repo: 9,586 TS files, 68,109 nodes, 8,716 shards. Largest repo in the benchmark.

	Baseline	With shards	Delta
Duration	555s	502s	10% faster
Files changed	0	5	baseline failed
Budget	exceeded $2 limit	within budget	—
Fix produced	❌ none	✓ correct	—

The baseline agent burned its entire $2 budget navigating 9,586 files and produced no fix. It ran out of tokens before finding the right code.

The shard-guided agent found the fix in 502s: added tax line cleanup to `revertLastChange_` in `order-module-service.ts`, correctly identifying that `OrderLineItemTaxLine` and `OrderShippingMethodTaxLine` records need soft-deletion during rollback. It understood the version/item relationship pattern from the graph data.

This is the strongest result so far — on a large enough codebase, the baseline agent literally cannot complete the task within the same budget.

[jonathanpopham]

Interim results — $2 budget run

All 5 issues tested. $2 budget per run.

Summary

Repo	Issue	Files	Baseline ($2)	Shards ($2)
directus/directus	#26991	4,250	782s, 5 files ✓	704s, 5 files ✓
directus/directus	#26997	4,250	573s, budget exceeded ❌	324s, 5 files ✓
strapi/strapi	#25118	3,496	568s, budget exceeded ❌	410s, 3 files ✓
payloadcms/payload	#15980	6,074	563s, budget exceeded ❌	execution error ❌
medusajs/medusa	#14384	9,586	555s, budget exceeded ❌	502s, 5 files ✓

Observations

4 out of 5 baselines exceeded the $2 budget and produced no fix. The agents burned their entire budget navigating large codebases with grep alone. The one baseline that succeeded (directus#26991) used 31 tool calls vs 11 with shards — 65% fewer.

3 out of 4 shard runs produced fixes within $2. The graph data let the agent skip the exploration phase and go straight to the relevant files. Medusa (68K nodes, largest repo) was the strongest result — baseline produced nothing, shards produced a correct 5-file fix.

The $2 budget is too low for baselines on repos with 3,000+ files. Re-running baselines at $20 to get fair comparisons. The cost savings story is clear: shards-guided agents can do in $2 what baseline agents can't do in $2. The $20 re-run will tell us the actual baseline cost to complete the same fix.

What the directus#26991 comparison shows (only fair comparison so far)

Metric	Baseline	Shards	Delta
Duration	782s	704s	10% faster
Tool calls	31	11	65% fewer
Files changed	5	5	same fix

$20 budget re-run in progress for the remaining 4 issues.

[jonathanpopham]

Full metered results — $20 budget, sequential, stream-json

Model: Claude Opus 4.6 (1M context). Each run isolated — no parallelism.

Repo	Issue	Mode	Claude Time	Turns	Tool Uses	Total Tokens	Est. Cost	Files Changed
strapi/strapi	#25118	baseline	~1,146s	84	122	6,023,028	$14.93	3
strapi/strapi	#25118	shards	2,190s	87	315	8,922,917	$22.43	7
directus/directus	#26997	baseline	1,506s	67	169	4,180,619	$12.37	5
directus/directus	#26997	shards	1,824s	81	159	7,277,573	$19.53	5
medusajs/medusa	#14384	baseline	827s	44	190	1,943,422	$6.42	3
medusajs/medusa	#14384	shards	754s	51	142	2,186,680	$6.44	5
payloadcms/payload	#15980	baseline	1,341s	50	165	2,406,498	$6.81	0 ❌
payloadcms/payload	#15980	shards	—	—	—	—	—	API crash ❌

Plus directus#26991 from the earlier $2 text-mode run (no token counts):

| directus/directus | #26991 | baseline | 782s | — | 31 | — | — | 5 |
| directus/directus | #26991 | shards | 704s | — | 11 | — | — | 5 |

Analysis

Mixed results. Shards was faster only on Medusa (754s vs 827s, 25% fewer tool uses). On Strapi and Directus, shards was slower and consumed more tokens.

Why the $2 run looked different: The earlier $2 run used `--output-format text` (not stream-json) and ran in parallel. Baselines hit the $2 cap and died at ~568s. Shards finished at 324-502s. This looked like shards were faster, but it was actually the budget cap killing baselines before they could finish. With $20 budget, baselines complete — and are sometimes faster.

Why shards consumed more tokens: Every grep, glob, and file read now hits `.graph.*` shard files alongside source files. The agent reads more data per tool call. On Strapi, shards used 8.9M tokens vs 6.0M baseline — the extra 2.9M tokens were mostly shard file content being ingested during searches.

Where shards helped:

• Medusa (68K nodes, largest repo): 25% fewer tool uses (142 vs 190), 9% faster, correct 5-file fix vs baseline's 3-file partial fix. Same cost.
• Directus #26991 (earlier run): 65% fewer tool uses (11 vs 31), 10% faster
• Directus #26997: 6% fewer tool uses (159 vs 169) but slower overall due to token overhead

Where shards didn't help:

• Strapi: More tool uses (315 vs 122), slower, more expensive. The agent may have been distracted by shard content.

Key takeaway

Shards reduce navigation effort (fewer tool uses on 2/3 metered comparisons) but increase token consumption because graph files are read alongside source files. The net effect depends on repo size and how much navigation the task requires. Largest repo (Medusa) showed the clearest benefit.

Methodological notes

• `--output-format stream-json --verbose` was used for token counting
• The earlier $2 text-mode run and the $20 stream-json run may behave differently — stream-json could affect agent execution
• Cost estimated at Opus rates: input $15/MTok, cache read $1.50/MTok, cache create $18.75/MTok, output $75/MTok

[jonathanpopham]

Deep dive: tool usage breakdown and fix quality

Tool usage by type

Repo	Mode	Read	Grep	Bash	Glob	Edit	Agent	Total
strapi#25118	baseline	59	30	24	6	2	1	122
strapi#25118	shards	137	58	91	22	2	5	315
directus#26997	baseline	67	23	69	3	4	3	169
directus#26997	shards	66	30	48	11	2	2	159
medusa#14384	baseline	101	28	44	10	1	3	190
medusa#14384	shards	57	38	22	11	1	3	142

What happened on Strapi (shards worse)

The shards agent made 66 grep/find calls via Bash on top of 58 Grep tool calls — 91 total Bash invocations. It was hitting `.graph.*` files in search results and spending extra calls trying to filter them out or follow the graph data through additional searches. The shards polluted the search results instead of helping navigation.

What happened on Medusa (shards better)

The shards agent used 44% fewer Read calls (57 vs 101) and 50% fewer Bash calls (22 vs 44). It read 4 graph files explicitly and used the dependency/call data to navigate directly to `order-module-service.ts` without the iterative grep-read-grep cycle the baseline went through.

Fix quality vs known merged PRs

Repo	Known PR	Baseline	Shards
strapi#25118	5 files	1 file (InputRenderer.tsx) — partial	3 files (Form.tsx, InputRenderer.tsx) — wider scope
directus#26997	4 files	2 files (controller + service) ✓	1 file (service only) — missed controller
medusa#14384	20 files	1 file, 80 lines — correct targeted fix	1 file, 78 lines — same correct fix
payload#15980	6 files	0 files ❌	API crash ❌

Key findings

1. Shards help most on the largest repo (Medusa, 68K nodes): 25% fewer tools, 44% fewer reads, same cost, same correct fix
2. Shards hurt on medium repos (Strapi, 25K nodes): search result pollution caused 2.5x more tool calls
3. Fix quality is comparable — neither mode consistently produces better or worse fixes
4. The pollution problem is real: `.graph.*` files match grep patterns meant for source files, causing the agent to process graph data as if it were source code, then make additional calls to disambiguate