Judge the trajectory, not just the output

Teams evaluate LLM systems mostly by grading the system's output. How the output is graded depends on the amount of time and resources invested, from small deterministic checks when there is a clear right or wrong answer to expensive and time-consuming human reviews.

LLM-as-a-judge sits in between: a model reads the final answer and returns a verdict, ideally a binary pass or fail calibrated against a domain expert. All three grade the answer, not the process by which the agent got to its output.

For a single completion, that is fine, because the output is all there is. But as we move towards multi-agent and long-horizon workflows, the ways in which our agents can fail and the opportunity for slight deviations in trajectory grow significantly, which can reduce the likelihood of the output being accurate and consistent.

This article expands on the need for evals that measure agent trajectory, some of the agentic mistakes we have observed, and the system we have created to monitor the behavior of some of our agents.

Why this matters

We are building Glassray, a platform that monitors agents in production to catch silent failures and generate evals from production data. Most of the failures it catches look like this: the output is fine, and the trajectory behind it is wrong.

We evaluated an agent that answers security questionnaires from a company's knowledge base. A correct-sounding answer that does not come from the customer's documents is worth nothing on a security questionnaire.

During one of our evaluations, the agent answered an ISO 27001 certification question confidently and correctly. Every output check passed it.

Then we opened the trace.

The agent had never searched the knowledge base. Earlier in the conversation, the customer had asked about SOC 2, and the agent had answered correctly. When the ISO 27001 question arrived, it assumed a company with SOC 2 probably has ISO 27001 too. The knowledge base had nothing to confirm that.

Judged on the final response, there was nothing to catch. Open the trace and you get a question about one certification, a conversation history about a different one, no retrieval call, and no source check before the agent answered.

Its rules told it to ground every answer in the knowledge base; this answer was implied from the conversation instead. That shortcut holds only while nothing changes. When the customer replaces a policy or lets a certification lapse, the same shortcut produces an outdated answer that sounds no less confident, and a final-answer check still sees a plausible response.

The gap between accuracy and reliability

The problem is not unique to one agent. In the last two years, model capability and accuracy have improved rapidly. What has not kept pace is reliability: whether an agent that can solve a task actually solves it consistently, every time, without silent shortcuts or missed steps.

Princeton's reliability study (Rabanser et al., ICML 2026) makes the gap concrete. They ran 15 agents over two benchmarks, GAIA's 165 tasks and the verified 26-task subset of τ-bench airline, five times each. Accuracy across the agents ranged from 17% to 80%. Their overall reliability score, a composite of consistency, predictability, and robustness, sat in a narrow band from 0.69 to 0.87, and the most accurate agent was not the most reliable one. Two years of capability gains produced only small improvements in reliability. One detail matters for what comes later: across runs, agents were consistent about which actions they took and inconsistent about the order they took them in. The study's own summary of the shift is to stop asking "How often does the agent succeed?" and start asking "How predictably, consistently, robustly, and safely does it behave?" Its verdict on why that matters: "an agent that succeeds on 90% of tasks but fails unpredictably on the rest may be a useful assistant yet an unacceptable autonomous system."

If agents fail unpredictably, the next question is what those failures look like. MAST, a taxonomy built from over 1,600 annotated multi-agent traces across seven frameworks, catalogs 14 recurring failure modes (Why Do Multi-Agent LLM Systems Fail?). The systems it studied failed on 41% to 87% of their tasks. The most common failures are trajectory-shaped: repeating a step that already ran (16% of failures), stating one plan and taking a different action (13%), failing to notice the task is done (12%). None of these show up in the final answer. An output check sees a wrong answer or a right one; it cannot see a right answer that was reached by repeating work, contradicting its own reasoning, or overshooting the task.

A wrong output tells you the run failed. A trajectory tells you where it failed, and whether the same failure is hiding in runs that happened to produce a correct output. The longer the trajectory and the more agents involved, the more places a silent failure can sit while the final answer still looks right.

The failures we found

On the questionnaire agent, we went through the traces with Glassray and flagged the runs that deviated from the intended trajectory. Three failure shapes kept coming back:

Wrong-source retrieval. Retrieval happened, so a "did it retrieve?" check passes. But the agent searched the wrong place: an internal wiki for a question about the current API, or an old document where a newer version exists. The answer looks reasonable and rests on the wrong evidence. You can only spot it by comparing which source the answer needed against which source the trace actually consulted.

Phantom grounding. The answer presents a claim as something it looked up, and the trace contains no retrieval behind it. The ISO 27001 answer is this one. The agent never retrieved a source and framed the claim as if it had. To catch it, take each claim framed as looked-up and search the trace for the lookup. The failure is an absence, so the output never shows it.

Silent fill-in. The agent states a fact confidently without attempting a lookup, and unlike phantom grounding, nothing in the answer claims a lookup happened. A policy, document, or value just appears in the answer as if it had always been known. Because the assumption is often right, nothing surfaces the behavior until the assumed value goes stale.

All three produce plausible answers, and some produce correct ones. Output evals would mark all as passing.

The first row is why output evals stay in the stack. The other five rows are the gap.

How to evaluate the trajectory

Every agent follows instructions. A security-questionnaire agent's system prompt says "always cite sources" and "never guess configuration values." A customer-support agent's says "verify the account before making changes." Every one of these converts directly to a check you can run against a trace.

Does the trace contain the behavior in the agent's instructions? If the answer cites a source, that retrieval must exist in the trace. If the answer states a certification or policy fact, a source lookup must precede it. If a required value was not provided, the trace must show where it came from.

A useful formalization of this is a behavioral spec: a document that lists what the agent must and must not do, written in natural language, decoupled from any specific tool-call sequence. "Ground every answer in the knowledge base" is a behavioral spec clause. "Call search_documents at step 3" is a golden path. One describes conduct; the other prescribes a route.

A behavioral spec turns into two kinds of checks.

• Some clauses reduce to deterministic predicates: Did a retrieval tool call occur before the final answer? Is the cited document ID present in the trace? Does the count of search calls fall within bounds? These are string matches and sequence assertions, free to run, and they cover most of the spec.
• The remaining clauses need judgment: does the answer's claim actually match the content of the retrieved source? Is the agent's summary faithful to the document it cited? These become scoped LLM calls, one clause at a time, each with a binary verdict and quoted evidence.

That split controls cost and reliability. Deterministic checks are free and perfectly consistent. LLM checks are expensive and stochastic, so you want as few of them as possible, each scoped as narrowly as possible.

behavior-judge is an open-source project that demonstrates this architecture end to end. It compiles a behavioral spec into an intermediate representation of deterministic predicates and scoped semantic checks, runs the predicates first, and calls the LLM only where the predicate layer cannot reach.

How we evaluate agentic trajectories

At Glassray, we generate a behavior spec for each agent and flow (a group of agents). These files contain the goal, instructions, and guardrails set for that process. For each one of those rules, we generate a small rubric with a combination of deterministic checks, such as a regex or an identifier to look for, and non-deterministic checks for an AI judge to evaluate.

We then pass the entire trace to a model that goes through those evaluations. The deterministic checks run on the full trace, whereas the AI judge has access to a skeleton of the full trace (the steps and agents involved) and a summary of each step, with the ability to investigate the full context of the relevant sections.

Through this process, we discover deviations, which you can think of as errors. The three failures above are examples of deviations.

Once we suspect a deviation, we create a detection recipe for a small model to find it across a larger cohort of traces. The recipe defines what counts as a match, what does not, common false positives, the evidence the checker must quote, and what to return when the evidence is unclear.

Run the recipe against one trace at a time, with a constrained verdict (present or absent) plus the quoted evidence.

A condensed version of our phantom-grounding recipe:

match:      the answer states a fact as retrieved or looked up, AND the trace
            contains no retrieval or tool call that returned that fact
not match:  the fact is framed as an assumption, or as a question back to the
            user; retrieval happened but returned a superseded document (that
            is wrong-source retrieval, a separate recipe)
evidence:   quote the answer's claim AND name the retrieval that is absent
if unclear: return absent (false alarms erode reviewer trust faster than a
            rare miss, and misses are backstopped by sampling)

We ran this end to end on the questionnaire agent's own traffic. Glassray had recorded the confirmed cases as deviations; a strong model turned one into a recipe like the block above; we labeled 24 of the agent's real traces as an answer key, with the evidence quoted per trace; and each candidate model then ran the recipe over the identical extracted traces, three times each.

What's hard about it

Trusting the judge. LLM judges are inconsistent. They need calibration before you can trust their verdicts, and the calibration has to be redone whenever the model changes. For open-ended quality scoring (is this answer good?), that problem is hard to solve.

Trace checks are narrower. They check whether a specific relation holds inside one document: does the cited source appear in the trace, does a retrieval precede the claim? The checks are close to binary, which makes them easier to validate.

You can validate without labeled data. Take a trace that passed, break it deterministically (delete the retrieval span or swap the cited document for a different one), and check that the judge flags the broken copy while still passing the original. Because the break is deterministic, you know the right answer without labeling anything. When we do calibrate against labels, the bar is agreement on a small adjudicated set before the judge's verdicts count, and we weight misses most heavily.

Cost. Running an LLM over every trace is expensive if you send the full trajectory in a single call. Most of the work does not need an LLM. Checking "does this retrieval exist in the trace" is a string match. Checking "did the agent call the search tool before answering?" is a sequence assertion. Reserve the LLM for what only an LLM can check (does the answer's claim match the content of the retrieved source), scope it to one claim at a time, and use a cheaper model.

We benchmarked four models and cut the cost of the per-trace check by 1.5x while catching more deviations. The full comparison is in We found a 1.5x-cheaper trace judge that catches more real failures.

Trajectory evals are not there to replace output evaluation; they complement it. You still want to know whether answers are good, and a final-answer eval is the right tool for that question. It cannot tell you whether a good-looking answer came from the customer's documents or from a guess that happened to land. The trace is where that gets answered.

Start smaller than any of that. Pull your last 20 traces and read them against the three failure types.