agentic search vs vector search: 3 eval lessons

26 searches, still no answer: that is one vector-search trace from Braintrust's agentic search vs vector search eval. The agent retrieved semantically close chunks of code, missed the imports and calling code around them, and searched again. The agentic-search arm, running plain grep and file reads through Claude Code, solved the same tasks at the same accuracy for roughly a quarter of the cost.

Agentic Search vs Vector Search: Which Won?

Neither arm won on accuracy — both solved the same share of tasks — but agentic search won decisively on cost. In Braintrust's October 2026 eval, vector search ran roughly four times as expensive because its agent kept re-retrieving incomplete chunks, while the agentic-search agent found what it needed on the first pass.

In Braintrust's October 2026 eval, agentic search and vector search reached the same accuracy, but vector search cost about four times as much. Both arms ran inside Claude Code, Anthropic agentic coding tool for the terminal, on 20 bug-location tasks built from merged fix PRs in Microsoft's open-source TypeScript Go repository. Scoring was binary: pass the repo's test suite and the run scores 100%, fail and it scores 0%.

The cost gap came from behavior, not raw retrieval quality. The vector-search agent retrieved semantically similar chunks, found them incomplete, and searched again and again, so its LLM calls piled up. The agentic-search agent grepped a function name, read the full function, spotted the flaw, and followed the call into another file.

Dimension	Agentic search	Vector search
Accuracy	Same	Same
Relative cost	~1x	~4x
Typical behavior	Grep, read full function, follow calls	Re-retrieve incomplete chunks repeatedly
LLM call volume	Lower	Higher

All figures here are Braintrust's own reported results from a single eval run by one practitioner, not an independent reproduction. Treat them as a worked example of eval methodology as much as a retrieval benchmark.

What Vector Search and Agentic Search Actually Are

Vector search turns code or text into embeddings, numerical vectors that represent semantic meaning, and stores them in a vector database such as Qdrant or Pinecone. A query like "where does the database logic live?" returns the chunks whose embeddings sit closest to the query's meaning. The failure mode is structural: a chunk is a fragment, and fragments often lack imports, tool calls, or the calling code above them.

Agentic search gives the model tools instead of pre-computed vectors. The LLM runs grep, find, ls, cat, and bash commands, opens a file, reads a function in full, and follows references across files, roughly the way a human developer explores an unfamiliar codebase.

The two approaches answer different questions. Vector search asks "which fragments look semantically similar to this query?" Agentic search asks "can the agent navigate the repository until it understands the logic?"

How Braintrust Built the Eval Dataset and Task

Braintrust built its eval dataset from roughly 20 real merged pull requests in the microsoft/typescript-go repository, then ran two search arms — agentic and vector — against those bug-fix tasks, after solving a tracing problem that hid what Claude Code did inside its subprocess.

The dataset came from real merged pull requests. Wang and her team collected PRs titled "fix" from the microsoft/typescript-go repository, checked out the parent commit — the state where the bug still existed — and used Claude to generate a bug task from the diff between buggy and fixed code. About 20 PRs made it into the dataset.

Implementing the two arms required different effort:

1. Agentic search was easy because Claude Code uses agentic search by default; the team simply pointed the harness at the tasks.
2. Vector search needed constraints: the prompt explicitly forbade agentic search, and the disallowed tools flag blocked the terminal commands Claude Code would otherwise call.
3. Prompting alone was not enough, in Wang's experience — the tool-level flag was what actually enforced the constraint.

One infrastructure problem surfaced during tracing. Claude Code runs as a subprocess, which orphaned the traces: the parent logged that a Claude process started but recorded nothing inside it. The fix was passing parent span IDs as environment variables, which restored visibility into every turn, LLM call, and terminal command inside the subprocess.

Why Vector Search Cost Four Times More

The traces explained the cost gap. In one run, the vector-search agent made 26 searches and still could not piece together how three functions interacted across files. Each retrieved chunk lacked surrounding context, so the agent kept issuing new queries, and every query meant more LLM calls and more tokens.

Wang's summary of the pattern: vector search gave proximity to the correct code, while agentic search supplied the connective tissue between chunks. The agentic agent grepped a function name, read the whole function, identified the flawed logic, and followed the call chain into another file, finishing with fewer searches.

This is a component-level cost result for one eval configuration, not a universal pricing law. The 4x figure reflects this dataset, this embedding setup, which Wang herself described as a relatively weak implementation, and Claude Code as the shared harness for both arms.

The Four Parts of an Eval, and Why They Matter

An eval is built from four components — a dataset, a task, a scorer, and experiments — that together turn "it seems better" into measurable evidence. The dataset defines what you test, the task defines what the system does, the scorer judges quality, and experiments let you compare runs to spot improvements or regressions.

• Dataset — holds golden cases, edge cases, and known failure modes.
• Task — defines the system's behavior, usually a prompt plus a model.
• Scorer — judges output quality via deterministic checks, LLM-as-judge, or human review.
• Experiment — one specific dataset-task-scorer combination; comparing experiments shows improvements or regressions.

Wang framed evals as the alternative to shipping on vibes: instead of "engineering said it looked ready," you can say "200 test cases ran, 94% passed," or report trade-offs like a 5% accuracy gain that cost 5% on tone.

She also argued evals are a team sport, with each role contributing a distinct piece:

1. AI engineers write code and move data.
2. Product managers define success hypotheses.
3. Subject matter experts label ground truth in domains like medicine or law.
4. Data analysts interpret results.

As a cautionary example, she cited OpenAI's April 2025 model update, which aimed to be more helpful and instead made the model sycophantic and less truthful — a regression a systematic eval could catch.

Limitations of This Eval

Wang listed the holes herself. The eval ran single trials per task on about 20 rows, too few to trust against LLM non-determinism; best practice is multiple trials per task. The dataset covered one repository. And the vector-search implementation was admittedly weak, so a stronger retrieval setup, better chunking, or reranking could narrow the cost gap.

Cursor has publicly claimed that semantic retrieval improved its coding agent, which is part of why the topic drew discussion online, so the honest reading is narrower than "agentic search wins." The defensible conclusion: in this setup, embeddings added cost without adding accuracy, and any team choosing a retrieval strategy should run its own eval on its own repos before deciding.

Frequently Asked Questions

• Did vector search find the wrong code? No. It found code near the bug but often missed imports, tool calls, and calling context, which left the agent unable to act without more searches. Accuracy still matched agentic search; cost was the difference.

• Can I trust the 4x cost figure? It is Braintrust's own measurement from one eval on 20 tasks in a single repo, with a weak vector implementation by the author's own admission. Treat it as directional evidence, not a benchmark for all codebases.

• What scored the eval? The repository's own test suite. A run that made the TypeScript Go tests pass scored 100%; a failing run scored 0%. This binary scoring avoided subjective judgments entirely.

• How do I force Claude Code into vector-only search? Braintrust combined explicit prompt instructions with the disallowed-tools flag to block grep, find, and similar commands. The team found prompting alone insufficient.

• What is the practical takeaway for coding agents? Default agentic search was cheaper at equal accuracy here, but the stronger lesson is methodological: build an eval from your real fix PRs, run both retrieval strategies under the same harness, and let your own traces decide.

Turn Your Own Talks and Walkthroughs Into Articles

This article exists because one 18-minute conference talk held a complete eval: dataset design, tooling workarounds, trace debugging, and honest limitations. That kind of knowledge usually stays locked inside videos, watchable only by people who find them.

If you have talks, interviews, or technical walkthroughs on YouTube worth reading, Skala Blog turns a video URL into a transcription and then into a structured, publishable article. Paste the link, and the reasoning that lived in the recording becomes text that search engines and readers can actually find.

Source video