Find every run where your agent went wrong, prove each fix on your own traces,
and monitor production so every fix stays fixed.
Case studySuper44 replaced the agent monitoring it built in-house with Glassray and increased agent reliability 2.2x2.2xagent reliability55custom failure patterns736deviation investigations“The statefulness of Glassray is the biggest win: it reduces the noise and helps us figure out what's important over time.”
Henry George, Co-Founder at Super44
Case studyArchitect caught 15x more agent failures and cut investigation time by 19 hours a month with Glassray15xmore failures caught5xmore fixes verified~19hsaved per month“The quality of the answers we give is an extension to our business. It's absolutely essential and core.”
Luke Ramsden, CPTO at Architect
& more1/5Trace
Bring the traces you already have from Langfuse, LangSmith or OpenTelemetry. Every step and tool call in one place.
Every trace ingested across your connected sources. Filter, sort, and search to narrow the stream.
Oct 22 9:03 am
Summary: Customer asks why they keep getting logged out on other devices; agent explains the session policy and suggests signing in again.
User
Why do I keep getting logged out on my other devices?
Routed to → account_support
For security, every session expires after 24 hours on all of your devices. Signing in again will sort it out.
A deviation is a recurring way your agents stray from their intended behaviour. Glassray watches your synced traces and surfaces new ones automatically.
GlassrayAPP9:04 AMFabricated policy when the knowledge base has no answer · major
Suspected
When search_kb finds nothing, the agent answers anyway and states an account policy as fact.
Answer an account question
41 instances in 1,280 traces scanned · 41 major · last seen 41s ago
3 repliesLast reply just now
When the knowledge base has no answer, say so and offer a handoff to a person.
After search_kbreturns no documents, the agent states an account policy as fact instead of admitting it doesn't know.
Measured across all 1,280 traces evaluated against this rule.
41 examples over Oct 9 - Oct 22, against the 1,280 traces scanned in this flow.
trc_01M5…F8A41s agoView tracesearch_kb found nothing for "logout other devices", and the agent told the customer every session expires after 24 hours.
Why does the support agent make up account policies?
Deviation · Fabricated policy when the knowledge base has no answerAll 41 runs start with an empty retrieval. search_kb returns no documents, and the system prompt tells the agent to always give a definitive answer, so it fills the gap with a policy that sounds right.
search_kb("logout other devices") → 0 documents
prompt: "Always give the customer a definitive answer."Suggested fix: when retrieval comes back empty, say so and offer a handoff to a person.
acme / support-agentmain from fix/kb-empty-handoffReplayed 48 production traces from Fabricated policy when the knowledge base has no answer against fix/kb-empty-handoff.
mainInvents a policyThis PROffers a handoffmainAnswers from the docsThis PRHands off instead| Same request | main | This PR |
|---|---|---|
| Logged out on my other devices | Invents a policy | Offers a handoff |
| Refunds on annual plans | Invents a policy | Offers a handoff |
| How do I reset my password? | Answers from the docs | Hands off instead |
41 fixed · 1 regression · 6 unchanged · View the replay in Glassray
Measured across the 1,412 traces evaluated since #218 merged.
3 examples since the merge, against 1,412 traces scanned in this flow.
Super44 briefs local merchants on their own business every morning: an agent running up to 20 tools compiles sales, staffing, reviews, and the weather that feeds the revenue estimate. One morning the weather tool returned empty, and the agent wrote the briefing anyway, reusing yesterday's forecast as if it were confirmed.
No error, no complaint, nothing for an eval to fail. Glassray compared the reply against the tool outputs in the trace, found a forecast with no data behind it, and flagged the run with the evidence attached. The pattern stays monitored, so a made-up number never gets to become a habit the merchant plans around.
“Do merchants trust that when the agent tells them something about their data, it's true? If it says ‘you've sold more Aperol Spritzes’ and they think, I don't believe you, our value diminishes.”
app.super44.aiGood morning! Today's sunny and mild, so expect the terrace to fill up. Projected revenue: €18,400.
Trace viewmorning briefingThe statefulness of Glassray is the biggest win: it helps developer experience, reduces the noise, and helps us figure out what's important over time.
We follow a "price should be a no-brainer" strategy. If it's not, tell us why.
Every trace, in one place. Plug in your stack and see what your agents are doing.
Glassray watches production for you and flags the runs that quietly went wrong.
Slack / webhooksA custom solution for teams running agents at scale.
Limited alpha spots. Join the waitlist and we'll reach out when your stack is ready to plug in.