Self-Learning Agent Loops with Arize

Most production agent failures stay invisible because the evals you wrote only cover cases you already knew about. Self-learning agent loops with Arize close that gap by treating trace data as the source of truth and letting a coding agent investigate the code path behind a failed run. Detection, investigation, a fix, and verification against real failures form one loop.

Self-Learning Agent Loops with Arize in Four Steps

Self-learning agent loops with Arize follow four steps: detect, investigate, fix, and verify. Detection surfaces a failure before a user reports it, investigation marries the trace to the code path that produced it, the fix lands as a pull request, and verification replays real failure cases against a development endpoint.

The four steps separate cleanly. Detection runs on production traces. Investigation needs the repository as well as the trace, because a trace alone shows what happened without showing which line caused it. Verification needs a dataset of cases the agent actually failed on, not synthetic examples written in advance.

• Detect: surface the failure from live traces before a support ticket arrives.
• Investigate: read the trace alongside the agent's repository to find the code path.
• Fix: let a coding agent patch the fault and OpenAI pull request.
• Verify: replay the failing dataset against a development endpoint and compare.

The loop is what Arize, an AI observability and evaluation platform, presented at an AI Engineer workshop on 11 October 2026. The presenter was Fuad Ali, who works in product at Arize, and the example application was the Wonder Toys shopping assistant.

Why the Failures That Cost You Are Invisible at Launch

The failures that cost money are invisible at launch by definition, because a failure you caught would already be fixed. An agent passes the evaluations its team thought to write, then breaks on an edge case nobody imagined testing.

The Wonder Toys example makes the shape concrete. A shopper asks for a birthday gift under $30. The assistant returns items priced at $30 and $50, then purchases one confidently, because a price gate in the code never fires. Nothing in the response signals an error, and the failure does not appear in staging.

That pattern repeats across agent deployments. Staging exercises the paths the team knows about. Production exercises user phrasing, unusual budgets, partial tool failures, and combinations of arguments that never appeared in a test fixture. A fixed evaluation file cannot cover a case the team has not seen.

Arize reports that it processed over one trillion spans and ran more than 60 million evaluations per month as of the October 2026 workshop. Those figures come from the vendor and describe its own platform traffic, not an independent audit of agent reliability.

How Arize Signal Clusters Production Failures

Arize Signal is an always-on agent that runs inside the Arize platform and reads incoming production traces continuously. It clusters failures by root cause, removes duplicates across runs, and uses external tools such as a connected repository to write an investigation and OpenAI pull request.

Signal is enabled with a toggle in project settings. Once traces start flowing, it begins reading them in the background and builds a table of clustered issues that trends over time. Each cluster links to the individual traces that produced it, so a reviewer can open the evidence behind a category rather than trusting the label.

Fabricated results, Python serialization errors, and broken filter logic all appear as separate clusters in the workshop demonstration. For one cluster, Signal had already opened a pull request. The investigation view showed a plain-language overview of what the agent was doing, a hypothesis about the cause, supporting evidence from a single trace, and a proposed fix.

Signal can also be triggered by conditions other than a new cluster. Ali described configuring it to fire from an alert on a specific evaluator, a latency spike, or a cost spike, so the loop responds to operational thresholds the team already monitors.

Four trigger conditions surfaced in the session, each responding to a different operational threshold:

1. A new cluster forming from incoming production traces.
2. An alert firing on a specific evaluator.
3. A latency spike.
4. A cost spike.

The four evidence surfaces available to a reviewer differ in how close they sit to raw production data:

Surface	What it shows	Scope
Cluster table	Issues grouped by root cause, trending over time	All production traces
Trace link	The individual traces behind a cluster	One cluster
Supporting evidence	Evidence drawn for a single trace	One trace
Investigation view	Overview, hypothesis, proposed fix	One issue

The platform itself, including Signal's clustering behavior, is documented in the Arize docs, and the OpenAI models behind the investigation and pull request steps are described in the OpenAI developer documentation. The workshop session that demonstrated these clusters and the resulting pull request was hosted by Arize AI.

Arize Skills: Trace Analysis On Demand in Your Coding Agent

Arize skills let a coding agent query Arize traces directly from inside your repository, so the same session can see what the agent did in production and which code produced it. Skills cover trace analysis, dataset curation, and evaluator execution.

Installation runs through the skills command line, which opens an interactive list or installs every skill at once. Once installed, a prompt like "check my traces for errors and cross-reference with the code for this agent" returns a diagnosis grounded in both the trace tree and the source file.

Skills and Signal are two routes to the same outcome. Skills are the on-demand path for an engineer who is already editing the product and wants an answer now. Signal is the continuous path for teams that want the analysis to arrive without anyone opening a terminal.

The workshop used Claude Code, Anthropic agentic coding tool, as the demonstration client for the skills path. Arize states that the skills work with Cursor, Cursor's AI-first code editor, and other coding agents, though the recording only shows the Claude Code flow.

Harness as a Judge vs LLM as a Judge

Harness as a judge replaces a fixed evaluation rubric with an agentic evaluator that can call tools, read the full trace tree, and check outcomes against external context. LLM as a judge scores an input and an output against criteria you wrote beforehand, which works well for issues you already know about.

Dimension	LLM as a judge	Harness as a judge
What it reads	Input and output only	Full trace tree plus tool results
Criteria source	Rubric written in advance	Goals checked against live context
Tool access	None	Can query external systems
Unknown failure modes	Cannot cover them	Surfaces them from the trajectory
Best fit	Known issues	Access, budget and handoff checks

The limitation of a fixed rubric is timing. You cannot write a test for a bug you have not seen, and the price-filter failure was exactly that kind of unknown. A harness-based judge reads the trajectory the agent actually took, asks whether the right tool was selected, whether the arguments were valid, and whether the context handed between agents was correct. For background on the underlying evaluation pattern, see the OpenAI evals guidance and Anthropic writing on building effective agents.

Access checks show where the two approaches diverge. A judge that reads only the input and output cannot know what a given user is permitted to do unless every user record is pasted into its prompt. A harness that can call external systems can query permissions directly and confirm whether the trace reflects the access that user actually had.

For the budget failure, the rubric reduced to one sentence: pass if every returned product price falls inside the requested minimum and maximum range, fail if any result exceeds the budget. That rubric runs on every subsequent evaluation rather than covering a single hand-written case.

One ordered way to decide which judge to reach for:

1. Start with LLM as a judge whenever the criteria are already known and the input-output pair carries enough signal.
2. Move to a harness-based judge when the failure depends on tool selection, argument validity, or context handed between agents.
3. Add access and permission checks to the harness once the trace needs to be confirmed against external state rather than the prompt alone.

Arize's evaluation documentation covers how traces and rubrics are wired into a running loop.

Agent Experiments Test a Fix Before It Ships

Agent experiments in Arize let you replay a dataset of real failure traces against a development deployment and compare it with the production version before merging a fix. The development agent stands behind an endpoint, receives the dataset inputs, and is scored by the same judge rubric used in production.

The dataset comes from actual failures rather than invented examples. In the workshop flow, roughly 25 budget-constraint searches that failed were collected into a dataset, either through the skills interface or through Signal, the monitoring surface that flags production regressions. The experiment then ran the fixed version against those inputs and compared quality, latency, and cost side by side with production.

The comparison is deliberately multi-dimensional. A fix that improves the price-correctness judge while doubling latency or token cost is a trade-off a team should see before shipping, and the experiment view exposes all three measures on the same run. Judging follows the same evaluator pattern described in the OpenAI evals documentation, so a rubric that scores well offline is expected to behave the same way in production.

Measure	What it answers	Direction that matters
Judge score	Did the fix resolve the failure mode?	Higher
Latency	How long does the development agent take per input?	Lower
Token cost	What does each additional run cost to serve?	Lower
Dataset coverage	Do the replayed traces reflect real failures?	Broader

Ali described the outcome of one such validation as a 36% improvement on the specific evaluator for that fix. That number is a vendor-reported result from a single demonstration run on the Wonder Toys dataset, not a general benchmark for agent quality improvements.

To reproduce the workflow without shipping anything, the experiment loop can be followed in the same order every time:

1. Collect failing traces from production into a dataset, using either the skills interface or Signal.
2. Point the experiment at the development endpoint that hosts the candidate fix.
3. Score the development agent with the production judge rubric, alongside latency and token cost.
4. Compare the run against the production baseline before merging, then re-run the same dataset after any further change.

Keeping the dataset fixed across iterations is what makes the 36% figure meaningful: it describes one evaluator on one set of real inputs, and a different dataset can legitimately move it. Teams that want a second opinion on evaluator design can consult the guidance in Anthropic evaluation cookbook, which treats rubric Stable Diffusion a prerequisite for trusting any single-run number.

What a Full Autonomous Loop Actually Produces

A complete autonomous run produces a pull request containing the diagnosis, the proposed patch, and the validation results, with a human still holding the merge decision. Arize's stated position is that code review stays with a person.

The sequence runs without intervention once Signal is enabled. Signal reads a failing trace where a user asked for items under $30 and the response priced higher, investigates the code path behind the price filter, opens a fix as a pull request, then uses the experiments API to redeploy that branch to a development endpoint and rerun the failure dataset.

Because the verification stage is exposed through skills and an API, the agent can test itself against real failures before notifying anyone. The pull request then carries the evidence, the diff, and the measured change on the evaluator, which is what a reviewer needs to make a decision.

The loop depends on tracing being on. Without traces there is no source of truth about what the agent did in production, and no failure dataset to replay. Every later stage in the loop reads from that data.

Where the Loop Still Needs a Human

The loop still needs a human at code review, and the vendor says so directly. Automatic detection and repair produce a reviewable proposal, not a deployed change.

Two further boundaries matter for anyone planning this architecture. First, the demonstration relies on a connected repository and, in some configurations, downstream data sources such as Datadog, which means the investigating agent needs read access to your code. Second, the 36% evaluator improvement is a single vendor-reported run on one demonstration dataset, and it does not establish that the same loop improves every agent task.

The approach also assumes failures are observable in traces. An agent that produces a plausible but subtly wrong answer without any trace-level signal of an error will be harder to cluster, because Signal groups failures from what the traces contain.

The workshop code is the Wonder Toys assistant with a deliberate price-filter bug, traced to Arize and run on the OpenAI Agents SDK, which is OpenAI's open-source framework for building agent applications.

Frequently Asked Questions

• What is a self-learning agent loop? A self-learning agent loop is a cycle in which an agent reads its own production telemetry, investigates the cause of a failure in its own code, proposes a fix, and validates that fix against the real cases that failed. Arize presented one implementation in October 2026 based on detect, investigate, fix, and verify.

• How is harness as a judge different from LLM as a judge? LLM as a judge scores an input and output against a rubric written before deployment. Harness as a judge runs an agentic evaluator that can call tools, read the whole trace tree, and fetch external context such as a user's permissions, so it can assess cases the team never wrote a test for.

• Does Arize Signal open pull requests automatically? Signal can OpenAI pull request on a connected repository after clustering a failure and writing an investigation. Arize's guidance keeps code review and the merge decision with a human.

• What does an agent experiment measure? An agent experiment replays a dataset of failure traces against a development endpoint and compares the fixed version with production on quality, latency, and cost. The same judge rubric scores both sides.

• Why can't staging catch these failures? Staging exercises the paths a team already anticipated. Production surfaces unexpected phrasing, unusual constraints, and tool combinations that no fixture covered, which is why the failure dataset has to come from live traces.

Closing the Loop on Your Own Agent

The underlying lesson is that trace data and source code only become useful together. A trace shows what an agent did; the repository shows what it was supposed to do. Connecting the two is what turns a dashboard nobody reads into a proposal someone can review.

If you have recorded a walkthrough of a system like this on video, whether it is an agent pipeline, a debugging session, or a talk about how something works, that material already contains the explanation. Skala Blog turns a YouTube URL into a transcribed, structured article you can review and publish.

Source video