Content from the team building Glassray.
- Chamath Palihawadana
Judge the trajectory, not just the output
Every output check passed a certification answer an agent produced without ever searching the knowledge base. Why agent failures hide in the trajectory, what they look like, and how we evaluate trajectories at Glassray without golden-path matching.
Read more - Chamath Palihawadana
We found a 1.5x-cheaper trace judge that catches more real failures
Our per-trace LLM judge is about 30% of our total AI spend. We benchmarked four models on the same 24 real traces, three runs each, against labels written in advance. gpt-4.1 is 1.5x cheaper and caught more real failures than the model we run today.
Read more - Toma Rares
Trace digests for LLM monitoring, at 1/30th the price of Sonnet
The digest is our highest-frequency LLM call. We put four cheaper models against Claude Sonnet 4.6 on a frozen trace set, and the one we shipped costs about a thirtieth as much, with search returning the same traces.
Read more - Manuel Martinez Chamorro
Migrating our code explorer to a cheaper model: 32% cheaper, no accuracy lost
We assumed the risk of a cheaper codebase-mapping model was hallucination. After checking 4,222 citations against the repo, zero were invented files, and the real failures changed which model we run in production.
Read more