How to Build AI Agent Memory From Traces

AI agent memory is the durable context an agent references on future runs, and a trace only becomes memory when a lesson from it is written back into that context. Observability alone tells you what happened; memory changes what the agent does next.

Why Observability Alone Does Not Improve AI Agent Memory

Observability tells you what an agent did; it does not change what the agent does next. Jake Broekhuizen, who leads the labs team at LangChain, argued at the AI Engineer World's Fair 2026 in San Francisco that most teams have gotten good at watching agents but far fewer turn those observations into lessons. If the context an agent references never changes, a misguided instruction or skill repeats the same mistake in every future run.

His running example is a real project (speaker first-hand account): LangChain's labs team worked with a financial services company building a spending and budgeting agent that gives non-investment advice. In a regulated sector, tone matters. The agent's tone would slip from advisory to pointed and directive, saying things like "users in your similar situation would generally set a cap" or telling users to cancel subscriptions and move money into savings.

The company's fix was manual: a person reads the traces, finds where the tone slipped, edits the context the agent references, and checks for regressions. Broekhuizen's point is that this does not scale, and the system around the agent should absorb that work. Every run already produces a trace showing tool calls, reasoning, and retrieved artifacts. The gap is what happens after the trace is stored: the trace must stop being a log and become a source of signal that updates memory.

Semantic, Episodic, and Procedural Memory Types

Broekhuizen defines AI agent memory as the durable context an agent references to guide future runs, and he borrows a three-part taxonomy from cognitive science to organize it. Facts, preferences, patterns, examples, skills, and instructions all qualify as memory, but they do different work, and knowing which type you are updating changes how you should treat the update.

Procedural memory is where most visible gains come from, according to Broekhuizen. In the financial assistant example, the tone correction was not a new fact or a banned word; it was a rule to follow, something like "be measured and not pointed or directive." That rule changed behavior on every subsequent turn.

Memory type	What it holds	Example from the talk
Semantic	What the agent knows	Facts and user preferences
Episodic	What the agent has experienced	Past interactions, learned patterns, examples
Procedural	How the agent should behave	Instructions, skills, tone and policy rules

Working Memory Versus Long-Term Memory

Working memory is what the agent references during its current run: scratch pads, tool results, retrieved files. Long-term memory lives in a store outside the run and holds the instructions, skills, and rules the agent should reference across future runs. The distinction matters because a trace is a record of how working memory changed over the run, and only part of that record deserves to become durable context.

Different agent harnesses inject long-term memory differently. Broekhuizen lists several patterns: some inject it directly into the prompt, some retrieve it with tools, some pull it from a store, some load it through files, and some change runtime state. The implementation varies; the separation between temporary and durable context is what stays constant.

This is also where ideas like sleep-time compute, sometimes described as "dreaming," fit: memory is updated or pulled from long-term into short-term storage either while the agent works or while it is idle. The mechanism is flexible, but the read-write structure underneath it is not.

The Read-Write Cycle That Turns Traces Into Memory

The read-write cycle is the core loop of AI agent memory: memory informs the run, the run produces evidence, the evidence is filtered into signal, and the signal updates memory. Broekhuizen reduces the cycle to three verbs: capture what happened, analyze what might matter, and update the context so future runs can use it.

1. Read. The agent loads relevant context from long-term memory into working memory: skills, instructions, policies, and reference files. Broekhuizen connects this to progressive disclosure, where an agent pulls in only the skills relevant to the current task.
2. Run. The agent executes and produces a trace: tool calls, sub-agent calls, retrieved context, decisions.
3. Filter. This is the step he calls most interesting. You decide which evidence is useful signal and which stays as history. Most evidence should stay as history; converting everything into long-term memory would make the agent harder to reason over future tasks.
4. Write. The chosen signal is written back into durable long-term context, such as the "be measured, not directive" rule for the financial assistant.

The filter step is where teams will differ, and Broekhuizen is explicit that implementations across frameworks are still evolving. The three-part shape of capture, analyze, and update is the stable part of the pattern.

How LangSmith and Context Hub Implement the Loop

LangChain's implementation of the cycle pairs LangSmith, the company's observability platform, with a capability Broekhuizen called Context Hub. The capture step maps to LangSmith's tracing, where you can see tool calls, sub-agent calls, and retrieved context. The analyze step is a background process that extracts signal from traces based on criteria you define, and the store step promotes that signal into the memory the agent pulls down on its next run. (This description is vendor-reported; it reflects how LangChain presented the product in October 2026.)

In the financial assistant, the loop ran as follows: the agent loaded its instructions, skills, markdown files, and policies from Context Hub; the invocation produced a trace flagging where the tone strayed; the analysis engine looked for patterns and updated or promoted the relevant static skill markdown files stored in Context Hub; the next run showed the corrected tone. Broekhuizen is careful to note this is one implementation of the pattern, not the only one.

Keep the ownership straight when evaluating this: the memory taxonomy is standard cognitive science, the read-write cycle is the talk's framing, and Context Hub is LangChain's product layer for the analyze-and-store steps. A benchmark or behavior change observed in that stack is evidence about that stack, not about every agent framework.

Three Lessons From Building Agent Memory in Production

Broekhuizen closed with three design lessons drawn from his team's work, and each one constrains a decision you will otherwise get wrong.

Most trace data should stay history

Agents produce what he calls exhaust: far more feedback than any system should promote. Deciding what matters to you is the hard part. In his experience, most trace data remains referenceable history, some becomes datasets for offline evaluation, and only a small subset should change the agent's memory.

Watch caching in long-running agents

A gotcha he hit repeatedly: if a memory update lands at the start of a long-running background agent but cached runtime state never reloads from the store, the update is invisible to future runs. He spent time debugging why behavior was not changing before realizing the caching layer was the problem. Know what you can cache and what you cannot, so memory updates actually reach the agent.

Put human review on procedural changes

Broekhuizen notes the rise of self-updating agents and open-core harnesses in this space, then draws a line: procedural memory such as instructions, policies, and tone guidelines drives what he estimates at around 75% of an agent's behavior (his estimate, not a measured benchmark). Changes to that layer should surface for human review before they enter the agent's hot-reload path, while lower-stakes memory can update automatically.

FAQ

• What is AI agent memory? It is the durable context an agent references to guide future runs. A trace or log is evidence of what happened; it becomes memory only when a lesson from that evidence is converted into context the agent reads again.

• Which memory type changes agent behavior the most? Procedural memory, the instructions, skills, and rules layer. In Broekhuizen's financial assistant example, tone fixes came from updating procedural rules rather than adding facts, and he says most visible gains come from this type.

• Do I need LangSmith to build agent memory? No. The read-write cycle of read, run, filter, and write is implementation-independent. LangSmith and Context Hub are one concrete implementation of capture, analyze, and update; you can build the same loop with your own store and analysis step.

• Should agent memory updates be fully automatic? Not for everything. Broekhuizen recommends human review for procedural memory changes, since instructions and policies drive most agent behavior, while most trace data should simply remain history or feed offline evaluation datasets.

Turn Your Own Conference Talks Into Written Memory

A talk is a trace: seventeen minutes of evidence that only helps future runs if someone converts it into durable context. Written articles are that conversion step, and they are exactly what a trace becomes when it is filtered for signal instead of left as a log.

If you have knowledge sitting in YouTube videos, whether conference talks, tutorials, or interviews, Skala Blog turns that recording into a written article: paste the URL, transcribe the video, and generate a structured draft you can review before publishing. The flow is the same one Broekhuizen preaches: capture the experience, filter for what matters, and write it into a form people can reference later. Skala Blog was created by Gustavo Dev Doido, who also teaches the Crazystack Typescript stack and runs the Bootcamp do Dev Doido for developers building projects like this one.

Source video