Architect caught 15x more agent failures and cut investigation time by 19 hours a month with Glassray
About Architect
Architect builds an AI sales agent for ecommerce websites. The agent answers buyer questions, assembles personalized product pages, and drives purchases in real time, all in under four seconds. The company raised $4.75M from Project A and Concept Ventures and operates out of London and San Francisco with a team of eleven.
The Challenge
Architect's AI output goes directly to shoppers. If the agent gets something wrong, the shopper sees it. But the brand running the agent on their website usually doesn't. That makes failures invisible to the people paying for the product and visible to the people who matter most.
The only quality signal was customer complaints, about five a month. After a fix went out, there was no regression monitoring. The team would run some manual tests, confirm it looked fine, and move on. There was no way to know if the fix had actually worked or introduced a new problem.
The team had already lost customers to quality issues compounding over time. Small failures would stack up unnoticed until they added up to a serious problem.
“The quality of the answers we give is an extension to our business. It's absolutely essential and core.”
On top of that, the team ships fast. With a small engineering team running multiple workstreams in parallel, there's no room to stop and manually sift through production traces looking for problems nobody has reported yet.
Choosing Glassray
Before Glassray, Architect had tried a third-party observability tool. But it couldn't represent their telemetry properly, so it would flag confidence where there shouldn't have been any and miss failures that were actually there.
- It assumed a simple tool-calling loop, but Architect's agents are multi-step workflows, so the traces never mapped correctly
- It didn't understand Architect's codebase or requirements, so the results weren't actionable
- No separate environments for staging and production
- Too much magic in the instrumentation and not enough developer control over how tracing was set up
“It just did a bit too much “magic.” As an engineer, I just want something I can have a lot of control over and use in a way that makes sense for me.”
Glassray fits because it matches how the team actually works. It can be used entirely through MCP, supports separate environments, and automatically maps traces into flows without requiring anyone to define what failures to look for upfront.
Getting Started
After connecting, Glassray scanned Architect's codebase and started evaluating production traces automatically. No one had to define what “good” looked like upfront. In the first month:
- 2,470 traces evaluated across production traffic
- 28 flows mapped, each one a distinct path from a shopper's question to a final answer
- 77 failure patterns generated, checking for things the team never had to specify
- 20 fixes tracked, including one that triggered a full pipeline redesign
The team checks Glassray almost daily. When a failure is worth investigating, they pull it into their coding agent through MCP and let it validate or disprove the finding against the codebase. What used to take 15 minutes per issue now starts with the analysis already done.
“Having that first-order analysis makes it easier in terms of what my coding agent needs to do. It's all through MCP. It's already got a good idea, and it can validate or disprove the claim.”
An Example of a Silent Failure
Architect's agent reads its customers' websites, extracts facts like prices, and stores them in a database. When a shopper asks a question, the agent is designed to answer from that database.
Glassray found the database had a wrong price stored with high confidence, but the agent was answering from the website directly instead. Shoppers got the right answer by coincidence, so nobody knew anything was broken.
Glassray could see this because it looks at every step in the trace, not just the final answer. In this case:
- Scraper: read the customer's website and saved the wrong price to the database
- Validator: confidence scores and conflict checks said the data was fine, but the checks were broken and the wrong price passed through
- Answer generator: skipped the database and pulled from the website directly, which isn't the intended path
- Final output: correct, because it happened to use the right source by accident
If you only check the output, this looks fine. Glassray caught it because the data at the retrieval step didn't match the data at the extraction step. If the agent had followed its intended path, every shopper would have seen the wrong price.
The fix took the team multiple weeks. They redesigned how the system stores, validates, and serves facts, and built a review UI for disputed data. All from one failure Glassray found.
“This is the failure class we can't catch with evals or user reports. The system looked healthy because a downstream component masked the defect.”
Results
First month on the platform
“I definitely think that's really important to have that continuous monitoring and making sure that it doesn't come back.”
What Today Looks Like
Architect is preparing for 40x volume growth over the next three to four months. They're launching self-serve at $79/month, going broad to find their best customer segments. At that volume, a single undetected failure doesn't just risk churn. It mis-educates shoppers who never signed up for Architect's product but interact with it on every site visit.
More volume also means more customer websites to ingest, each with its own contradictions and bad data. The same class of knowledge-pipeline failure that Glassray caught once will keep appearing in new forms as the input surface grows.
After shipping a fix, the team can now check that the problem doesn't come back. Glassray automatically matches new pull requests to known failures, confirming that a code change actually addressed the issue. The team plans to expand Glassray access beyond Conrad as the engineering team and customer base grow.
“We need to do more with less. Part of that is being very efficient with tools and using a lot of AI.”
Find the failures your evals miss
Glassray monitors your agent's production traces and surfaces the runs where it looked like it worked but didn't.