AI observability braintrust workflow explained

Tracing alone does not improve your agent; reading every trace by hand does not scale either. AI observability in Braintrust takes the middle path: automated scores and a topics pipeline attach signal to traces so a coding agent can query only the failures worth investigating. Guthrie showed the loop running on 968 traces and ending in a pull request.

What AI observability with Braintrust covers

AI observability with Braintrust, an AI evaluation and observability platform, means capturing agent traces, scoring them automatically, clustering them into topics, and feeding the findings back into development. Doug Guthrie, a solutions engineer at Braintrust, demonstrated the full workflow at an AI Engineer workshop in October 2026 using a Python support agent built with the OpenAI Agents SDK. The agent itself was a stand-in; the point was the loop around it.

Braintrust positions evals and observability as two sides of one coin. Evals tell you whether a change improved or regressed the agent before deployment. Observability tells you what is actually failing in production. Guthrie described the company's direction as "active observability": instead of waiting for a human to pull and interpret trace data, the platform attaches insights to the traces themselves. This framing is the vendor's own positioning and should be read as such.

The workshop repo, published under the GitHub username dpguthrie, seeded a free-tier Braintrust organization with a project, an API provider configuration, and a small evaluation dataset. Everything shown ran on Braintrust's SaaS offering, though hybrid deployments exist, covered below.

Tracing: the foundation everything else sits on

Without a trace, an agent is a black box. Tracing records every step the agent takes, the inputs to each tool, and the outputs, which is what makes it possible to locate where quality deteriorates. Guthrie's integration was deliberately small: pull in the Braintrust SDK, set a trace processor on the OpenAI Agents SDK, and every span the agent emits flows to Braintrust. A one-line auto-instrument option also exists.

Braintrust supports multiple programming languages (Python, TypeScript, Java, Ruby, Go), major AI providers, agent frameworks, and OpenTelemetry, according to the workshop material. Integration breadth is a vendor claim, not an independently tested one. The workshop used the SDK path because it also routed model calls through Braintrust's gateway, which exposes all configured providers through one interface.

Each conversation turn appeared as its own trace in the logs view. A thread view shows the user-assistant back and forth, and a timeline view shows latency and cache-hit information per LLM call. For multi-turn conversations whose turns span separate traces, a grouping scope ties them together by a shared metadata field, in this case a conversation ID.

Scores: the first kind of signal

Scores measure agent output and come in several flavors. Guthrie's rule of thumb, drawn from customer conversations: start binary and be harsh. If you score 100% on every eval, your test cases are probably too weak. He also recommends including examples in judge prompts and often using a stronger model for judging than for generation, with the caveat that online scoring changes the cost calculus.

Scores run in two modes. Offline scores power evals, can be wired into CI, and answer "did this change help or hurt?" Online scores run as automations against incoming production traces. In the workshop, three LLM judges (communication quality, support resolution, tool-use quality) ran at 100% sampling against the trace scope, with the explicit warning that sampling rate directly drives model cost, since judges run on your own configured models.

The workshop used four score types, each with a different tradeoff. Code-based scores invoke no model at all.

Score type	What it does	Cost profile
Code-based score	Validates schemas, tool order, branching logic in code	Fast, cheap, no LLM call
LLM judge	Applies subjective criteria to a trace or span	Model compute per scored trace
Judge with human review	Calibrates LLM judges against human labels	Adds review overhead
Code wrapper around an LLM	Encodes branching logic, then calls a model	Mixed

One detail from the Q&A: scores cannot be chained in the UI. If score B should depend on score A, you write a single code-based score that calls the model conditionally and returns an array of score objects. Pushed scores are version-controlled in your repo and reusable in the playground.

Topics: catching the unknown unknowns

Topics is Braintrust's answer to failure modes your evals never anticipated. Evals test the failures you already know about; Topics clusters patterns across traces, including silent ones that factuality and moderation scores miss. The workshop's 968 imported traces surfaced an order record retrieval failure that no built-in score had flagged.

The pipeline works in steps: a default preprocessor compresses the trace into the thread's back-and-forth, a facet (the taxonomy prompt) summarizes it, the summary is embedded, and the embeddings form a topic map used to label and classify traces. Built-in facets cover task, sentiment, and issues. Custom facets, like the workshop's support-workflow-issues prompt, let you target specific failure families and exclude "no problem" outputs with a regex.

Two operational details matter. First, the summarization and embedding steps run only on Braintrust-hosted models; the company says alternative models produced poor topics, so the choice is enforced. Second, Topics requires at least 100 matched facets before it generates a cluster; the workshop's built-in issues facet never crossed that threshold on the demo data, which Guthrie presented as a feature rather than a bug. Vendor-reported pricing for the hosted pipeline is 6 cents per million input tokens and 40 cents per million output tokens. Dropbox, per Guthrie, has experimented with replacing a set of LLM-judge classification scores with Topics to cover far more of its trace volume at lower cost; that is a customer example reported by the vendor, not an independent study.

From signal to pull request: the automated flywheel

The core claim of the workshop is that production findings should drive development. Guthrie's loop: filtered traces and topics identify representative failures, those examples return to eval datasets, the agent code and scores change, evals run against the updated dataset, and results are compared to the previous run. The Braintrust CLI, released in late February 2026 per Guthrie, makes the loop scriptable: it runs SQL against trace data, creates datasets, and runs evals from a terminal.

The workshop extended this to a coding agent. With the CLI installed and a Braintrust-improvement skill attached, Codex queried the trace data with SQL, inspected a cluster of escalate-to-human tool errors, proposed code and scorer changes, added regression examples to a new dataset, and ran evaluations. Guthrie also demonstrated a GitHub Action that runs Claude Code, Anthropic terminal-based agentic coding tool, with the same CLI and skill, producing a pull request that packaged the SQL queries, the reasoning, and links to the supporting eval experiments for human review.

Scaling claims here belong to the vendor: one customer, per Guthrie, generates five to ten additional pull requests per day with this workflow. The structural insight stands on its own, though. Once scores and topic labels are attached to traces, a coding agent can query relevant failures instead of scanning raw traces, which is the difference between an agent that has context and one that has a firehose.

Backend, deployment, and cost boundaries

Braintrust's answer to trace storage at scale is BrainStore, a purpose-built backend the company says it built in late 2024 after outgrowing ClickHouse, the database it previously used behind the scenes. Trace data is queryable with SQL directly, which is what lets coding agents pull narrow slices of production data. This history and the competitor comparison (Guthrie claimed LangSmith and Arize have since built their own databases) come from the speaker, not from independent verification.

Deployment comes in three shapes: full SaaS, BYOC, and a hybrid model restricted to the enterprise plan. In hybrid, the control plane (web app, metadata) stays with Braintrust while the data plane (traces, datasets, experiments) runs in the customer's VPC, with Terraform or Helm deployment per cloud. Guthrie estimated that about 90% of Braintrust's large enterprise customers are self-hosted; again, a vendor figure.

Guthrie's recommendation on sampling rates: start high on a new agent, then scale back as known failure modes stabilize. For classification-style LLM judges, he suggested considering Topics as a cheaper substitute when judge costs stop scaling. Access control has one notable pattern from the Q&A: write traces to a locked-down project, run the automated flywheel under a service account, and share only the resulting analysis and pull requests, so reviewers see conclusions without raw trace access.

FAQ

• Can I use Topics on coding-agent sessions like Claude Code or Codex? Braintrust lists integrations for Claude Code, Codex, Cursor, and Open Code, and Guthrie said multiple customers are layering Topics on those traces to understand how the tools are actually used. He noted the surrounding machinery is still being built out.

• Do I have to use Braintrust's models for everything? No. Online scores and LLM judges run on the AI providers you configure. Only the Topics pipeline's summarization and embedding steps are restricted to Braintrust-hosted models.

• Can traces from another observability provider be imported? Yes. The workshop imported 968 traces and 5,260 spans from an S3 bucket via the CLI's sync command, and Guthrie said he has helped customers migrate traces from other providers.

• How do topics handle long-running conversations? Two settings help: a group scope that ties turns across separate traces by a shared ID, and a configurable topic window that controls how much time the pipeline waits before generating a topic map.

• Can this loop propose code changes automatically? Yes, with the CLI and skill attached to a coding agent. Guthrie demonstrated sessions where the agent wrote SQL, proposed code and score changes, and opened a pull request, with a human reviewing before merge.

Turn your own walkthroughs into written guides

The most useful part of this workshop was not the demo agent but the loop: trace, score, cluster, feed back into evals, repeat. That same loop applies to knowledge work of any kind, including the talks and walkthroughs engineers record on video but never publish as text.

If you have a workshop, tutorial, or deep-dive sitting on YouTube, Skala Blog turns it into a structured, searchable article. Paste the video URL, get a transcription, and generate a publishable draft. The signal is already in the video; this just makes it retrievable.

Source video