We found a 1.5x-cheaper trace judge that catches more real failures

Companies looking to scale AI often rely on AI-as-a-judge to determine whether their agent is following the right steps and giving the correct output. These evaluators often run over every trace an agent produces, and each check answers one narrow question: does this trace show a pre-defined failure or not? The main issue with this approach is usually cost, and many teams respond by evaluating a small subset of traces or by switching to smaller models as a temporary solution.

Motivation

At Glassray we are building an AI monitoring platform that evaluates production traces and creates evals per flow and per customer. To evaluate each agentic flow we analyze the agent's prompts and use them to create an initial rubric, which then runs against hundreds of thousands to millions of traces per client per month. This AI-as-a-judge currently represents about 30% of our total AI spend.

In this post we measure the difference in quality and cost between models used as that judge, and walk through how we benchmark them.

Results

For this experiment we tested an agent called "chatbot supervisor", which answers security questionnaires from a company knowledge base. Its one non-negotiable rule is that answers must come from the customer's documents. We look for one failure. When the sources do not cover a question, the agent sometimes fills the gap from the model's own general knowledge and states it as confirmed fact. (Case details are paraphrased throughout; the traces contain customer content.)

So we asked whether a cheaper model could run these checks without missing real failures. We treated a missed real failure as disqualifying, whatever the price. We tested four models on the same fixed set of traces, three runs each, and scored every verdict against labels written before any model ran. The result up front:

Two things in that table surprised us. The cheaper model caught a confirmed case (an invented regulation cited as fact) that our reference model missed in all three runs. And the reference itself caught only one of the three real cases. On this evidence, gpt-4.1 is the candidate to run.

The rest of the post covers how the judge works, the setup, the golden set, the runs, and what this run does not show. If you want to run this on your own agent, the checklist at the end is self-contained.

What the judge actually does

Evaluating a whole agent trace with a broad prompt such as "review this run" is expensive and vague, and you cannot tell whether two models disagree because one is wrong or because the question was mushy. We split the work into two steps instead.

A strong model writes a detection recipe once per failure type. The recipe defines what counts as a match, what does not, common false positives, the evidence the checker must quote, and what to return when the evidence is unclear. We write recipes on Opus.

A per-trace check then applies one recipe to one trace and answers present or absent, quoting its evidence. This is the repeated step, and it runs today on Sonnet 4.6.

The check sees the full trace, and its job is to compare the answer against what the trace actually contains.

This is the condensed recipe the experiment ran (case details paraphrased):

DEFINITION
  Answers must come from retrieved sources. When the sources do not cover
  the question, the gap must not be filled from the model's own knowledge
  and stated as fact.

COUNTS AS A MATCH
  The final response states a specific factual claim (a process, an
  obligation, a regulation cite, a registration fact) that no retrieval
  or tool output in the trace contains, unhedged.

DOES NOT COUNT
  The claim is supported by any retrieval, even partially or in different
  wording. The response hedges, attributes, or declines. The claim only
  restates something the user said earlier.

EVIDENCE TO QUOTE
  The claim itself, and the retrievals checked that lack it.

IF UNCLEAR
  Return absent.

Everything in this post tests only the repeated step. The recipe is fixed. We are testing which model can apply it without missing real failures.

The setup

The agent's traces already stream into Glassray, which had already flagged the two confirmed instances of this failure as deviations (a deviation is a run where the agent departs from its intended behavior), each with its evidence. The benchmark work was mostly plumbing on top of that store.

We pinned a fixed trace set: 24 traces, the 2 confirmed instances plus the 22 most recent traces from the agent at the moment we froze the set. Every run, on every model, scanned this exact list; the only thing that changed between runs was the model.

From each trace we extracted the same four inputs:

• the user's question
• the prior conversation
• every retrieval and tool output the agent made
• the final response

Each model ran the recipe over these identical extracts three times.

The golden set

We built the golden set first: the expected verdict for every one of the 24 traces, written down before any candidate model ran.

The labels were also model-written, by a stronger model that was not one of the four candidates. It applied the recipe to each of the 24 traces, checked every specific claim in the final response against every retrieval and tool output in that trace, and quoted the evidence behind each verdict. Each trace was labeled once. 3 of the 24 traces contained the failure; 21 were clean.

The three positive cases, paraphrased:

• Case A, an invented process step. The agent described a document-amendment step that appears in no retrieved source.
• Case B, an invented regulation. The agent described a regulatory process and cited a regulation the sources never mention.
• Case C, an answer that contradicts its own retrieval. The agent stated a company registration fact, and the retrieved document says otherwise.

Cases A and B were the two confirmed instances we started from. We did not know about Case C; the labeling pass found it. It is also the hardest of the three to detect, because the answer does not lack support, it disagrees with the support.

Labels have to come before verdicts: if you label after seeing a model's output, the model's opinions leak into the labels, and you end up grading the judge against itself.

We scored each model on agreement with the golden set. False negatives (real failures called absent) mattered most, then false positives, then cost.

What the runs showed, model by model

gpt-4.1 was the strictest reliable checker. It returned the identical 24 verdicts in all three runs: Cases A and B caught, one false positive (the same trace every time), and Case C missed. Determinism matters in a checker because a flagged trace should stay flagged when you re-run to confirm.

The hardest catch was Case C. Catching it requires comparing the answer's claims against the content of the retrieval. Sonnet, gpt-4.1, and gpt-5.4-mini missed it in every run; Haiku caught it in two of three. Whatever model you pick, this shape of failure deserves its own recipe with the contradiction spelled out as a match condition.

Haiku 4.5 was the only model to catch Case C. In two of its three runs it caught two of the three real cases, at 2.5x below the reference cost. It also raised two false positives, on the same two traces, in every run.

gpt-5.4-mini raised no false positives, but its verdicts were unstable. Its catches went 0, then 2, then 1 across identical runs. A checker whose verdicts change on unchanged inputs is unusable regardless of price.

Sonnet 4.6, the model we actually run, caught the fewest real failures. It caught Case A in every run and nothing else, with at most one false positive. Trace by trace its verdicts look reasonable; only scoring against the golden set showed the two misses.

What this run does not show

• Three real positives is a small target set. The corpus is a fixed, pre-registered slice of real traffic, but 3 present cases cannot rank models finely; the catch counts show how each model behaves, and precise rates need a bigger set.
• Costs are estimated metered prices. Input tokens were counted once on the identical prompts and applied to every model, with each model's own output tokens, priced from one price book; the Anthropic side runs on a subscription locally, and prompt caching (which favors the incumbent) is excluded. Before shipping a cross-provider switch, the candidate has to beat the incumbent's net cost with caching enabled.
• One agent, one failure type. The ranking could change on a different deviation or a different trace shape. The protocol should transfer to other agents and failure types, but do not assume this ranking will.
• We could not validate the judge without labels. We tried corrupting passing traces so they must fail (deleting their retrievals, swapping in retrievals from another trace) and checking that the judge notices. Neither corruption produced a trace that both looks complete and genuinely cannot support the answer, so every claim in this post rests on the labeled golden set.

Other ways to do this

Not every trace rule needs an LLM judge. When the expected behavior is precise, a deterministic check is better: if retrieval must happen before the final answer, inspect the tool-call sequence directly. Tools such as Promptfoo can assert that expected tools were called, and you can test routing with pytest and a mocked model.

The limitation is flexibility. Agents can reach the same valid result through different paths, and rigid trajectory checks reject behavior that is unusual but still correct. Model-based evaluation handles that case: tools such as agentevals, LangSmith, and DeepEval evaluate full trajectories with room for variation.

Our approach sits in that second category but distributes the cost differently: the expensive reasoning is done once when the recipe is written, and the cheapest model that passes a pre-set bar runs the repeated checks. All it requires is access to the agent trace.

Try it on one failure

Take one known failure and collect a small set of traces containing both positive and negative examples. Then:

1. Label your traces before testing any judge, with the evidence quoted per trace. The golden set is what the comparison is scored against. Labeling may also surface cases you did not know about; our third positive came from the labeling pass.
2. Ask your strongest model to turn the failure into a precise detection recipe: what counts, what does not, the evidence to quote, what to return when unclear.
3. Run the recipe with your current judge, more than once. Verdict stability on unchanged inputs is part of the bar.
4. Test cheaper models against the identical traces. Different inputs per model will manufacture differences that are not real.
5. Count false negatives first, false positives second, and cost third, and compare on net cost with caching included.
6. Add a health check so broken traces and broken verdicts (a present with no evidence) never reach a reviewer.

Glassray was already monitoring the agent and storing its traces; Glassray Coach is our open-source local bench for exactly this kind of comparison. Nothing in the method depends on either. The next post explains why we show the judge the trace and what output-only evaluation misses.