Migrating our code explorer to a cheaper model: 32% cheaper, no accuracy lost

We assumed the risk of using a cheaper model in our codebase-mapping agent was hallucination: that it would cite functions and files that are not in the code. Then we checked 4,222 code references against the repository itself. None were invented files.

What we found instead was subtler. It changed which model we run in production. This post is the full experiment: the pass bar we fixed in advance, where every cheap model broke, and what grading the output alone could not catch.

Claude Sonnet 4.6 runs that agent today at about $1.15 per run. Ten models are cheaper, some ten times cheaper. Can any of them do the job?

Motivation

We build Glassray. It watches AI agents in production and answers one question: did the agent do what it was supposed to do? Answering that takes two sources. What the agent was supposed to do lives in the code, in the prompts and rules that define it. What it actually did lives in the traces. Glassray reads both and flags where they disagree.

This post is about the first source. Our code explorer reads the repository and maps the agent flows inside it: what each flow does, which rules govern it, and, for every rule, a {file, symbol, line} anchor pointing at the code that enforces it. Everything downstream hangs off that map, so it has to be right. That is why swapping in a cheaper model is worth measuring carefully.

Results

Every model got the same repo and the same prompts, and we ran each one three times. Ten cheaper models were the candidates; our two Claude models were controls. To pass, a candidate had to roughly match the model we run today (Claude Sonnet 4.6) on every column, and that bar was fixed before any candidate ran.

Reading the columns:

• Successful runs counts the runs that delivered a usable map. The failed runs explored the repo, then failed while producing the final answer.
• Citations valid is the share of the map's {file, symbol, line} anchors that resolve to real code where the map says they are, checked by script (the three-grep check below), not by a model.
• Flows found is how many of the repository's 14 real flows the map covered.
• Rule accuracy is how much of the reference map's rule content the candidate's rules actually carried. This is the one column a script cannot produce; it needs an LLM judge, calibrated against human labels first (below).

Swap the model and change nothing else, and all ten cheaper models fail. The two Claude rows are controls. Sonnet sets the bar, and Opus shows it is reachable: Opus clears every quality column too, at a price above Sonnet's.

The last three rows take three of those failed models and wrap each in a custom harness: the model plus a small engine-side guard for the specific way it failed. One passes and beats Sonnet; it is what we run now. The other two miss by one criterion each. The rest of the post is how we got there, and every table reports medians.

How our code explorer works

Our code explorer reads the repository and returns a map: every agent flow, the rules that govern it, and for each rule a {file, symbol, line} anchor pointing at the code that enforces it.

Take one flow, the trace-search flow behind our "Ask" panel:

flow:   Find traces
does:   a user asks in plain language ("which runs failed silently last week?");
        the finder searches the project's stored traces and returns the matching
        ones as a structured result set, not prose
agent:  finder (the FINDER_SYSTEM prompt)
rules:  [...]

Every rule is a claim about behavior, and each one comes with a receipt. Take one of those rules, with its anchor:

rule:   "Find the matching traces and return them as a result set, not an essay"
anchor: packages/shared/src/server/agent-tools/ask-shared.ts :: FINDER_SYSTEM (line 150)

The anchor is what makes a rule checkable. Without one, a rule is just a claim, and nobody hand-checks a thousand of them. With one, a script can verify it: whether the file exists, whether the symbol occurs in it, and whether it sits near the line the map claims. That is three greps per rule, each taking milliseconds, with no judge model involved.

We expected hallucinations, but barely found any

So we ran that check across every anchor the models produced. We braced for hallucination, meaning invented functions and files. The package-security world calls this slopsquatting: models invent plausible package names, so you check the names against the registry, and the famous number there says roughly one in five AI-generated references point at packages that do not exist (source). A code map is the same problem, with the repo itself as the registry. Across 4,222 anchors from the twelve models, here is what they actually were:

The invented code we feared is only that last 0.3%. One DeepSeek run is the clearest example. It reported a Deep Search flow (a flow that scans a batch of traces for matching examples), entry point runDeepSearch in apps/worker/src/routes/deep-search.ts, governed by the rule "stop scanning early once ~2-5 confirmed examples are found," anchored at line 184. The file exists, the rule matches the code word for word, and the constant is there too. But the function is called runDeepSearchJob, and the constant lives at line 367. A human reviewer would nod along at all of it. The grep caught both in milliseconds.

Four models, from Haiku to Opus, invented the same kind of near-miss name: runSlackAgent for the real runSlackAgentJob, runGithubPoll for runGithubPollJob, and the inverse, runCodeExploreJob for the real runCodeExplore. Real names, give or take the word "Job." We spot-checked a sample of these by hand against the source before trusting the script's verdict.

So the failure we feared is the rarest row in the table. What dominates is wrong lines and, worse, missing content. Silent omission is the real threat to this task. A map with 6 real flows looks exactly as trustworthy as a map with 14, and the greps cannot help, because you can only check a receipt that was written. Catching omission needs a reference to compare against: a known-complete map of this repo. So we built one: an answer key and a pass bar.

The experiment: ten cheaper models against the same bar

The pass bar

Everything that could vary was frozen before the first candidate ran: the repo pinned to one commit, the file-search seed computed once and reused by every run, the production prompts untouched, and three runs per model with no cherry-picking. Then the pass bar itself went into the repo, before any candidate produced a number. This is a pre-registration, the same discipline drug trials use, so no one can move the goalposts after seeing the data.

The bar is set the way Hamel Husain keeps telling people to build evals: measure the model you already run, then hold candidates to within noise of it. Sonnet's three calibration runs produced the bar. To pass, a candidate needs:

• flows found within 5 points of Sonnet's average (bar: 71%)
• citations valid at 98% or better
• rule accuracy within 5 points of Sonnet's (bar: 76%)
• zero confirmed wrong rules
• no disaster runs: one collapsed run out of three fails the candidate
• at least 25% cheaper than Sonnet's $1.15, or the swap is not worth the risk (bar: $0.86)

One of those numbers cannot come from a script. Rule accuracy measures whether a candidate's rule means the same thing as the reference's rule. Answering that takes an LLM judge. An LLM judge can be unreliable, so we checked ours against human labels before using it. A human scored 60 blind examples; the judge matched those labels 90% of the time, with a documented bias toward under-crediting candidates.

Round 1: swap the model, change nothing else

Round 1 changed only the model id and left everything else alone. Ten cheaper candidates and the two controls, three runs each. Every round's table carries the same columns.

All ten candidates failed, in five distinct ways. Almost every model showed a different one:

• Failed final write. These runs explored the repo, then failed at the last step, producing the map as one large JSON object. DeepSeek v4 Pro read 23 files over 14 steps, then handed back a 63,728-character payload that wasn't an object at all: it had wrapped the whole map inside a single string. These come back in Round 2.
• Missing content. Haiku, GPT-5.4-mini, and Grok returned small maps in under 90 seconds. Nothing in them looked wrong. Most of the codebase simply was not there.
• Wrong line numbers. GPT-5.6-luna found real files and real symbols, then cited line numbers a median of 50-70 lines off.
• Slow and unreliable per run. Kimi's tokens cost a third of Sonnet's, but its runs took 13-17 minutes against Sonnet's 6, and one of the three runs failed.
• A gap every model shared. Every model, controls included, missed the same three flows (the semantic digest, rule linking, and the merge gate), because those flows' code sits far from anything the file search surfaces.

Round 2: keep the models, adjust the settings

Look again at the runs that failed at the final write, where the model had to produce the entire map as one large JSON payload in a single tool call. Nothing about the task requires that format. It was our design choice, made because Sonnet handled it fine. So before concluding anything about the models, we fixed the hand-off we had built. Round 2 changed two settings:

• Incremental submission. Instead of one giant payload at the end, the model submits each flow as it finishes it: one small submit_flow call per flow, a few thousand tokens each, then a tiny closing finalize_map call. The exploration and final output are unchanged; only the hand-off is smaller.
• High reasoning effort. GPT-5.6-luna never crashed at delivery; its failures were wrong lines and missed flows. So its change was the one setting OpenAI exposes: reasoning effort turned to high, which buys more thinking per step.

The table shows every combination we tried, including Sonnet on the new protocol as the control: if incremental submission had changed the reference's own numbers, comparisons across rounds would break. It did not.

The protocol fix worked completely, for what it targets: every run on the new protocol delivered a valid map. Qwen went from zero successful runs to three, DeepSeek Flash from one to three. Those failures came from our hand-off format. With delivery fixed, each model hit its real limit:

↑ / ↓ mark each candidate's move from its Round 1 run. (Qwen had no Round 1 numbers: it produced 0 runs.)

Two rows deserve a second look.

GPT-5.6-luna with both settings on.

It scored the highest rule accuracy in the experiment, 86.8%, above Opus. It covered every prompt file. One of its runs found 13 of 14 flows, the best exploration any model recorded. And its citations scored 0% in one run. Those citations were not invented: the files and symbols were all real; GPT-5.6-luna just filed them in the wrong fields, putting filenames in the symbol slot 153 times in a single run. The model mapped the codebase better than Sonnet but could not reliably fill in two schema fields. The cause was incremental submission, the same change that fixed DeepSeek and Qwen.

DeepSeek v4 Pro with incremental submission came the closest of any candidate.

It matched Sonnet's flow coverage. Its rule accuracy sat inside Sonnet's band, with zero contradictions. It cost about a quarter of Sonnet's price. It failed one check: citations, at 96.7% against the 98% bar, every miss the same right-file, right-symbol, wrong-line mistake. The deeper problem is reliability. Two of its three runs were strong; the third collapsed to 82%. Its best runs clear every bar we set, but the one-in-three damaged run is what fails it.

Round 3: redesign the task around each model's weakness

Round 2 ended with each model stuck on one specific, well-measured failure: Pro on line numbers, GPT-5.6-luna on two schema fields, Flash on rule depth. Failures that specific point to a different question. Round 3 asked what each model needs around it to do this task, and made the unit of comparison a custom harness: the model plus a small engine-side compensation for its one defect. We built three compensations, about 150 lines of plain engineering:

• Line repair, for Pro's wrong lines. A citation's line number is mechanically derivable: grep the symbol in the file it names, take the occurrence closest to the model's claim. The engine now computes it deterministically. It cannot invent, because a file or symbol it cannot find is left untouched, so fabrications still get caught.
• Strict anchors, for GPT-5.6-luna's mangled fields. The submit tool now rejects a filename in the symbol slot or a fused path-and-symbol, says exactly what is wrong, and lets the model resubmit that flow. A format error now triggers a retry, so it no longer fails the run.
• A rules pass, for Flash's shallow rules. After the map lands, one focused follow-up call re-reads each flow's anchor files and extracts the complete rule set, replacing the thin one.

↑ / ↓ and the number mark each custom harness's move, in points, from its Round 2 setup (GPT-5.6-luna vs "both"; Pro and Flash vs "+ incremental").

The GPT-5.6-luna custom harness passes. It does not scrape by: it finds more of the codebase than the model we run in production (12 flows median against Sonnet's 11, with a 13-flow best), cites it near-perfectly, and matches the reference's rule accuracy, at about a third less. With two mechanical guards around it, the model whose citations scored 8% two tables ago is the best mapper we measured. Nothing about the model changed. Everything about the system around it did.

The DeepSeek v4 Pro custom harness came close and failed on the same criterion as before. Every median clears the bar: 10 of 14 flows, 99% valid citations, 78% rule accuracy, at $0.60 a run. It fails on reliability: one run in three comes back damaged, and it's a different problem each time. In Round 2 it was wrong line numbers; this time it was 13 symbol names that are not in the files they cite, the one anchor failure line repair cannot touch, because repairing a location only helps when the symbol is real.

The DeepSeek v4 Flash custom harness is the second near-miss, short by two points. The rules pass did most of its job: Flash's rule accuracy went from 59% bare to 74% with it, against a 76% bar. Everything else passed. It found 11 flows in every single run, cited them almost perfectly, and produced zero wrong rules. At $0.09 a run it is thirteen times cheaper than Sonnet. It is also four times slower, a median of 22 minutes a run against Sonnet's six.

Round 3 showed that the model swap by itself did not improve the result. What beat the incumbent was a model plus a system to catch what it misses, built from the failure data the first two rounds produced. The compensations cost almost nothing to run. Most of the effort went into the measurement that told us which compensation each model needed.

Why measuring the output isn't enough

Every number so far grades the map: the output. None of them checks how the model got there. But grading the output has a blind spot here: an agent can ignore its instructions or skip a guardrail and still hand back a map that looks complete and cites perfectly. Nothing errors, so this kind of failure never surfaces on its own.

We built and tuned the whole harness, every tool and prompt, around Sonnet, and the winning model is not Sonnet. It is capable, but a capable model on a hard task may not follow the approach we built, and without watching how it works you cannot tell a good shortcut from a bad one. A map that scores well but was produced by ignoring the harness is a map that will drift the next time the repo changes or the model updates, and no amount of grading the output will surface it.

So we did to our own experiment what Glassray does to any agent. We pointed Glassray Coach, our local behavior debugger, at the experiment's own traces. First it reads the explorer's code and derives the rules the flow is supposed to follow, each tied to the source it came from.

Then it checks every run against those rules, quoting its evidence.

The winning model does well even though it doesn't read the files in the order we told it to.

Whether that is a bug or a discovery is genuinely open. You can only ask because we watched how the model worked, not just what it produced. That check is what made the swap safe to ship. The numbers said GPT-5.6-luna's map was good; Glassray showed us how GPT-5.6-luna produced it, so a harness we tuned for one model does not silently drift on the next.

What we shipped

We are moving the explorer to the GPT-5.6-luna custom harness: high reasoning effort, incremental submission, strict anchors, line repair. It costs about a third less, with no drop in accuracy, and it's a behavior we have actually looked at. The measurement stays in the repo, so the next model release gets the same frozen protocol and a verdict in an afternoon, and Coach stays pointed at the flow, so the next harness change gets checked the way this one did.

We are also still working on the DeepSeek v4 Pro and Flash custom harnesses, because the price gap is large: Pro cleared every median at $0.60, Flash at $0.09 is nearly a tenth of the winner's price, and each missed the bar on a single criterion.