Skip to main content
Coach’s real value is finding the silent ways your agent misbehaves - wrong answers that aren’t errors, so ordinary monitoring shows everything green. This page walks the loop with a concrete example.
The setup. Say you’ve captured ~30 runs of a support agent. Every run “succeeded” - 0% errors - but some are quietly wrong: for order-status questions the agent sometimes answers with an invented status instead of calling its lookup_order tool.

Run discovery

Go to Deviations → Run discovery (CLI: glassray-coach deviations discover). Coach reads a sample of your traces, judges each one open-ended (“what went wrong here?”), and clusters the findings into recurring deviation types - each with a plain-language rule and the exact traces that match it.On our corpus it surfaces something like:None of these threw an error - they’re semantic deviations. A run shows live scanned/total progress and can be cancelled; a run that finds nothing recurring says so honestly.

Generate a fix

Open the Ungrounded order status deviation and click Generate fix. Coach reads the rule and every example trace and writes one concrete fix as a markdown instruction doc addressed to your coding agent, with fixed sections: a Goal, a repo search plan (the exact grep/ripgrep commands to find the code), the likely files (each labelled a guess to confirm), the implementation steps across prompt / tools / guardrails / orchestration / code, example coverage, and acceptance criteria.Copy it into Claude Code or Cursor and let it apply the change in your repo. Because Coach only ever sees traces, the fix tells the agent what to search for rather than asserting file paths - so it works without Coach touching your source.
Prefer the smallest change that addresses the root cause. For AI-agent deviations the fix is often not application code - it’s a tightened system prompt, a tool definition, or a guardrail. The generated plan considers that whole space.

Scope it as a flow

A deviation is about a behaviour - order-status questions, refunds, card updates. Give that behaviour a durable home: a flow is a named scope with a membership definition, and it persists across sessions. Define it with a deterministic selector (agent, name/intent substrings, status, explicit trace-id pins, and a limit for eval sampling) and/or a plain-language rule the background classifier matches by intent:
From here on, Coach classifies new traffic into your flows automatically - selector matches at ingest, rule matches in a background sweep. glassray-coach flows audit <id> shows you whether the definition holds up (member samples, low-confidence assignments, unclassified counts), so vague definitions get tightened instead of silently drifting. Prefer to bootstrap? Discover flows reads your agent’s own source (code discovery, not traffic) and infers candidate flows for you to tighten.

Lock the rule in as a flow-scoped eval

Open a deviation you care about, click Save as eval, and bind it to its flow. That freezes its rule into a repeatable pass/fail check that scores only that flow’s traces - the order-status eval never wastes a judge call on small talk.
You don’t need a full discovery run to start. Looking at one obviously-wrong trace? Hit Save as eval right from its detail view and write the rule there - the loop is reachable straight from the debugger.

Run the eval

From the eval, click Run eval. Coach scores the flow’s newest traces against the rule and gives you a pass/fail rollup - your baseline. Each verdict cites the specific evidence from the trace.

Prove it - hands-free

With the generated fix applied, send fresh traffic and don’t touch anything: new traces classify into the flow in the background, and once enough land (the eval’s autorun threshold, default 10) the eval reruns on its own. The pass rate climbs - and if a change reintroduced a case that used to pass, the eval flags it as a regression (▲). Once the traces pass, click Mark resolved on the deviation to close the loop.
Discover → scope → fix → codify → and from then on it runs itself. Every flow-scoped eval you keep is a standing, self-rerunning guard against that deviation coming back - including when you swap your agent’s model and want to know if the cheaper one holds up.

​
Alongside the loop

Discover flows

Don’t want to define flows by hand? flows discover scans your agent’s own source (its codeRoot) and proposes rule-defined flows inferred from the code - name-deduped against what already exists, so re-running it only ever adds. Tighten what it finds into selectors.

Replay

Open any LLM span and re-issue it with an edited model / system / prompt, with the fresh output beside the original - the viewer becomes a debugger.

​
Watch your spend

Discovery, evals, background classification, and replay all call a model. Coach meters every call against a budget - GLASSRAY_LLM_BUDGET_USD, default $50 - so a metered API key can’t drain while you experiment. The free mock and claude-subscription paths accrue $0. The Overview → LLM usage card shows spend vs. budget per model.

​
Next steps

CLI & coding agents

Every command, plus the agent skill (Claude Code, Codex, Copilot) that runs this whole loop for you.

Send traces

Instrument your agent so discovery has real signal to cluster.