> ## Documentation Index
> Fetch the complete documentation index at: https://glassray.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Find & fix deviations

> The self-healing loop, worked end to end - discover recurring failures, generate a fix for your coding agent, lock the rule in as an eval, and prove you didn't regress.

Coach's real value is finding the **silent** ways your agent misbehaves - wrong answers that aren't errors, so ordinary monitoring shows everything green. This page walks the loop with a concrete example.

<Note>
  **The setup.** Say you've captured \~30 runs of a support agent. Every run "succeeded" - **0% errors** - but some are quietly wrong: for order-status questions the agent sometimes answers with an invented status instead of calling its `lookup_order` tool.
</Note>

<Steps>
  <Step title="Run discovery" icon="ghost">
    Go to **Deviations → Run discovery** (CLI: `glassray-coach deviations discover`). Coach reads a sample of your traces, judges each one open-ended ("what went wrong here?"), and clusters the findings into recurring **deviation types** - each with a plain-language rule and the exact traces that match it.

    On our corpus it surfaces something like:

    | Deviation                   | Rule the agent broke                                        | Examples |
    | --------------------------- | ----------------------------------------------------------- | -------- |
    | **Ungrounded order status** | Must call `lookup_order` before stating an order's status   | 7        |
    | **Unauthorized refund**     | Refunds over the policy limit must escalate, not auto-issue | 4        |
    | **PII leak**                | Never echo a full card number back to the customer          | 3        |

    None of these threw an error - they're *semantic* deviations. A run shows live `scanned/total` progress and can be cancelled; a run that finds nothing recurring says so honestly.
  </Step>

  <Step title="Generate a fix" icon="wand-magic-sparkles">
    Open the **Ungrounded order status** deviation and click **Generate fix**. Coach reads the rule and every example trace and writes one concrete fix as a markdown instruction doc addressed to your coding agent, with fixed sections: a **Goal**, a **repo search plan** (the exact grep/ripgrep commands to find the code), the **likely files** (each labelled a guess to confirm), the **implementation steps** across prompt / tools / guardrails / orchestration / code, **example coverage**, and **acceptance criteria**.

    Copy it into **Claude Code or Cursor** and let it apply the change in your repo. Because Coach only ever sees traces, the fix tells the agent what to *search for* rather than asserting file paths - so it works without Coach touching your source.

    <Tip>
      Prefer the smallest change that addresses the root cause. For AI-agent deviations the fix is often **not** application code - it's a tightened system prompt, a tool definition, or a guardrail. The generated plan considers that whole space.
    </Tip>
  </Step>

  <Step title="Scope it as a flow" icon="diagram-project">
    A deviation is *about* a behaviour - order-status questions, refunds, card updates. Give that behaviour a durable home: a **flow** is a named scope with a membership definition, and it persists across sessions. Define it with a deterministic **selector** (`agent`, name/intent substrings, `status`, explicit trace-id pins, and a `limit` for eval sampling) and/or a plain-language **rule** the background classifier matches by intent:

    ```bash theme={null}
    glassray-coach flows create --name "Order status" \
      --selector '{"agent":"support-bot","q":"order","limit":20}'
    ```

    From here on, Coach classifies **new traffic into your flows automatically** - selector matches at ingest, rule matches in a background sweep. `glassray-coach flows audit <id>` shows you whether the definition holds up (member samples, low-confidence assignments, unclassified counts), so vague definitions get tightened instead of silently drifting. Prefer to bootstrap? **Discover flows** reads your agent's own **source** (code discovery, not traffic) and infers candidate flows for you to tighten.
  </Step>

  <Step title="Lock the rule in as a flow-scoped eval" icon="circle-check">
    Open a deviation you care about, click **Save as eval**, and bind it to its flow. That freezes its rule into a repeatable pass/fail check that scores **only that flow's traces** - the order-status eval never wastes a judge call on small talk.

    <Tip>
      You don't need a full discovery run to start. Looking at one obviously-wrong trace? Hit **Save as eval** right from its detail view and write the rule there - the loop is reachable straight from the debugger.
    </Tip>
  </Step>

  <Step title="Run the eval" icon="list-check">
    From the eval, click **Run eval**. Coach scores the flow's newest traces against the rule and gives you a pass/fail rollup - your baseline. Each verdict cites the specific evidence from the trace.
  </Step>

  <Step title="Prove it - hands-free" icon="wand-magic-sparkles">
    With the generated fix applied, send fresh traffic and *don't* touch anything: new traces classify into the flow in the background, and once enough land (the eval's **autorun threshold**, default 10) the eval **reruns on its own**. The pass rate climbs - and if a change *reintroduced* a case that used to pass, the eval flags it as a **regression** (▲). Once the traces pass, click **Mark resolved** on the deviation to close the loop.

    <Check>
      Discover → scope → fix → codify → and from then on it runs itself. Every flow-scoped eval you keep is a standing, self-rerunning guard against that deviation coming back - including when you swap your agent's model and want to know if the cheaper one holds up.
    </Check>
  </Step>
</Steps>

## Alongside the loop

<CardGroup cols={2}>
  <Card title="Discover flows" icon="diagram-project">
    Don't want to define flows by hand? `flows discover` scans your agent's own **source** (its `codeRoot`) and **proposes** rule-defined flows inferred from the code - name-deduped against what already exists, so re-running it only ever adds. Tighten what it finds into selectors.
  </Card>

  <Card title="Replay" icon="rotate">
    Open any LLM span and re-issue it with an edited model / system / prompt, with the fresh output beside the original - the viewer becomes a debugger.
  </Card>
</CardGroup>

## Watch your spend

Discovery, evals, background classification, and replay all call a model. Coach meters every call against a budget - **`GLASSRAY_LLM_BUDGET_USD`, default `$50`** - so a metered API key can't drain while you experiment. The free `mock` and `claude-subscription` paths accrue `$0`. The **Overview → LLM usage** card shows spend vs. budget per model.

## Next steps

<CardGroup cols={2}>
  <Card title="CLI & coding agents" icon="terminal" href="/coach/cli">
    Every command, plus the agent skill (Claude Code, Codex, Copilot) that runs this whole loop for you.
  </Card>

  <Card title="Send traces" icon="plug" href="/coach/instrument">
    Instrument your agent so discovery has real signal to cluster.
  </Card>
</CardGroup>