MCP sandbox stress testing means giving a coding agent an MCP server that exposes your sandbox, then having it generate and run 1,000 tests while logging results to a database. It found image encoding corruption, null-byte handling bugs, and files written in a sandbox that never came back.
What MCP sandbox stress testing actually measures
MCP sandbox stress testing measures whether a hosted code environment can complete the arbitrary workflows a coding agent invents at runtime, not whether a fixed REST endpoint returns the expected shape. The setup is an MCP server that wraps the sandbox, a coding agent on top, and a results store underneath.
The workflow comes from a talk by Kevin Orellana, a software engineer on Amazon Bedrock AgentCore, a cloud platform that hosts agent runtimes, identity, memory, gateways and code sandboxes. He wrapped the sandbox and a cloud browser in an MCP server so a coding agent could drive them through a protocol frontier models already handle well. The agent then generated 1,000 tests across 10 domains and ran them overnight.
The number 1,000 is arbitrary. Orellana says it was chosen to cast a wide net, and each test represents a workflow a customer might run: an end-to-end machine learning training job, a generated PDF report, or simply opening a web page. That framing matters because the failure that matters is rarely the failure you predicted.
Two hosted applications were exercised side by side: a code sandbox and a cloud browser. Both sit on the same platform, and their failures diverged sharply, which is the first sign that the sandbox, not the agent, decides what finishes.
Why results belong in SQLite, not markdown files
Results belong in a database because an agent's habit of writing markdown summaries stops scaling once you run 1,000 tests. The talk calls this out directly: markdown files do not hold a thousand test outcomes, and they cannot be queried.
Orellana's team stored findings in SQLite so multiple sub-agents could run locally, execute their own tests concurrently, and write outcomes into one store. That choice changes the shape of the experiment. With flat files you get a narrative; with a relational store you can group failures by domain, count them, and compare two hosted applications against each other.
The concurrency detail is easy to skip and important. If a single agent runs the suite serially, an overnight run is a tolerance test. If several sub-agents write into the same database, the sandbox is under contention, and contention is where lifecycle and memory failures surface.
The same principle applies outside this platform. Any agent-driven test campaign that only produces prose lies about its own coverage, because nothing can be recounted, filtered, or rerun by query.
Three failures that show the real pattern
Three failures from the run show the pattern better than a failure count does: an image that changed between upload and download, a null byte that derailed tests, and files written into a sandbox directory that never came back. Each one was mundane until it was reproduced.
The image that came back different
An image was uploaded, then downloaded, and compared against the original. It did not match. The cause sat in the encoding layer of the MCP server, not in the sandbox storage, which means the defect lived in the interface the agent used to reach the platform and would have survived any amount of sandbox tuning.
This class of bug is invisible in unit tests written from documentation. It only appears when something round-trips real bytes through the same path an agent uses.
The null byte that broke the test
A null byte, the zero-valued byte that terminates strings in some languages, threw off the tests. Whether the byte arrived from a file, a payload, or a string conversion, the effect was the same: a failure that looked like a service problem until it was traced.
Edge-case bytes are exactly what a broad suite catches and a hand-written happy path does not. Orellana notes the tests were unforgiving by design, and this is the least glamorous and most reusable finding in the talk.
Files written where nothing can read them
The third failure belongs to the sandbox layer. Files were written to a specific directory in the sandbox and were not available when the agent asked for them. Nothing about the write failed loudly; the retrieval did.
That single case is why the talk separates interface failures from sandbox failures. A code interpreter that cannot see its own output is not slow or unreliable. It is incomplete for the task, and the agent will retry, guess, or give up.
Interface-layer and sandbox-layer failures are different problems
Interface-layer failures live in the MCP server or client between the agent and the platform, while sandbox-layer failures live in memory, filesystem permissions, process limits, persistence, and kernel controls. Mixing the two sends debugging to the wrong team.
The distinction matters because the old mental model no longer fits. A long-running API service exposes one endpoint, and you know what the backend does. A sandbox hosts an arbitrary workload the agent writes at runtime, and no one knows in advance which system calls, packages, or file paths that workload will need.
Orellana lists the dependencies that decide the outcome: available memory and CPU, process and filesystem permissions, whether storage is persistent or ephemeral, and whether a process may spawn another process at all. A machine learning package that needs to fork, or a compiled binary that must land in a specific directory, either finds the environment ready or fails.
| Failure layer | Where it lives | Example from the run | Who fixes it |
|---|---|---|---|
| Interface | MCP server or client between agent and sandbox | Image corrupted by the encoding layer on round-trip | MCP server maintainers |
| Interface | Test harness and payload handling | Null byte derailing a test run | Agent and harness authors |
| Sandbox | Filesystem view and mount permissions | Files written to a directory, unavailable on read | Platform and sandbox config |
| Sandbox | Memory and session limits | Cloud browser tabs exhausting memory | Platform and workload design |
What the code sandbox and cloud browser failed on
The code sandbox and the cloud browser failed in different places because each demands different guarantees from the sandbox hosting it. Same platform, same agent, different primitives under pressure.
In the browser, the agent opened many tabs, scraped and summarized them, took screenshots, downloaded files, filled out forms, and ran a login workflow. The dominant failure was memory. Tab-heavy browsing exhausted sessions and sandboxes, and the run recorded those as complete failures rather than degraded results.
The code interpreter did not show the same memory fault. Its failures clustered around filesystem behavior: permissions, mounted filesystems, read-write-execute access, and whether storage persisted between steps. When the faults were mapped out, they gathered around a small set of sandbox primitives instead of spreading evenly.
The practical lesson is that a single benchmark number for a sandbox is close to meaningless. A workload that opens tabs and a workload that compiles a package stress different parts of the same container, and a platform can be strong on one and weak on the other.
Sandbox primitives that decide whether an agent finishes
Sandbox primitives decide completion before the model's reasoning quality matters. Kernel capabilities, seccomp filters, and extended BPF programs can allow or block the operations a workload needs, and the agent cannot negotiate around them.
Orellana walks down that stack: kernel interceptors, eBPF programs that inspect or control process behavior, egress rules that decide whether an agent can reach an internal company API, and the system calls a container permits. A build step that must spawn a process needs that permission. A Python package that must install into a specific module path needs that path writable and importable.
None of these look like agent problems. They look like infrastructure configuration until an agent's task fails at minute forty of an overnight run, and then they look like the only problem that mattered.
The talk's framing is that these low-level controls can hamper an agent's ability to reach its stated goal more often than the interface between the agent and the model does. That is a testable claim, and the run produced evidence for it by grouping failures around filesystem and memory primitives rather than around protocol errors alone.
Moving coding agents from your laptop to a remote sandbox
Remote sandboxes rarely reproduce your local workstation, and the mismatch is a tooling gap rather than a performance gap. Locally you have packages, MCP tools, and binaries that the remote container may not carry.
Developers already run coding agents such as Claude Code, Anthropic terminal-based agentic coding tool, on their own machines. As the number of workspaces grows, the laptop becomes the constraint, and teams push those agents into remote sandboxes to work through more pull requests in parallel.
That move changes the environment without changing the agent's expectations. Orellana points to one mitigation his team shipped: the ability for customers to SSH into their sandbox to inspect what is happening. He notes the user on the other end is often not a human but a coding agent, which is the whole point of the feature.
If your agent works locally and stalls remotely, the failure is probably not the model. Compare the two environments feature by feature before rewriting prompts.
What benchmark and broken-tool studies say about agent scores
Agent benchmark scores move with the environment, in both directions, and two studies cited in the talk make that concrete. One showed an agent beating benchmark tasks when given the right environment controls; the other showed performance dropping when tools were broken.
Orellana cites a study by researchers at UC Berkeley in which an exploit agent with control over the environment reached scores near 100% on benchmark tasks. He also cites work where a wide set of business problems was given to an agent whose tools were deliberately broken, and effectiveness fell sharply.
Those two results are useful together because they bracket the problem rather than settling it. A score published for a coding agent is a measurement of the agent plus its environment plus the harness, and none of the three can be read off the number alone.
Treat any single agent benchmark figure as configuration-specific until you can see the tool set, the sandbox limits, and the task source. The run described here did not produce a leaderboard score. It produced a list of defects.
How to fuzz your own agent sandbox overnight
You can run this experiment on your own platform by wrapping the sandbox in an MCP server, asking an agent for a broad suite, and storing outcomes in a database you can query. The talk's method is reproducible without the vendor's internal tooling.
- Wrap the sandbox and every hosted application in an MCP server, then smoke-test a few hand-written tasks such as writing and executing a Python file or converting a Python file to Java before trusting the harness.
2. Ask the coding agent to write a thousand tests across ten domains. Choose breadth over depth for the first pass, and accept that the count is a net, not a target.
3. Solve credential injection and failure triage before the long run. Decide in advance whether a failure is a service issue, an MCP-layer issue, or a test issue, and let the agent classify it.
4. Store results in SQLite so sub-agents can write concurrently and so you can group failures afterward instead of reading a narrative summary.
5. Map the failures onto sandbox primitives, then fix the interface bugs in the wrapper and the limits in the environment separately.
Orellana's closing challenge is to test locally and then fuzz remotely with a wide set of domain tests. The remote run is the one that finds the sharp edges, because that is where the environment differs.
FAQ
- What is MCP sandbox stress testing? It is the practice of exposing a hosted code sandbox through an MCP server and letting a coding agent generate and run a large suite of realistic tasks against it. The goal is to surface environment and interface defects rather than to score the model.
- Why did the agent write 1,000 tests instead of a focused suite? The count was arbitrary and chosen to cast a wide net across 10 domains. Breadth surfaced failure classes such as image encoding corruption and null-byte handling that a narrow suite would have missed.
- Why store test results in SQLite instead of markdown? Markdown does not scale to a thousand outcomes and cannot be queried. SQLite let multiple sub-agents run tests concurrently and write results into one store for later grouping.
- How do interface-layer and sandbox-layer failures differ? Interface failures live in the MCP server or client, such as an image corrupted during round-trip encoding. Sandbox-layer failures live in memory limits, filesystem permissions, persistence, and kernel controls, such as files written but never readable.
- Which sandbox limits broke the cloud browser tests? Memory. Opening many browser tabs exhausted sessions and sandboxes, and the run recorded those as complete failures. The code interpreter did not show the same memory fault.
- Does a remote sandbox match a local development machine? Usually not. Local MCP tools, installed packages, and binaries may be absent remotely, which is why an agent that works on a laptop can stall after being moved to a hosted environment.
- Can an agent beat a benchmark by controlling its environment? A study by researchers at UC Berkeley cited in the talk reported an exploit agent with environment control reaching scores near 100% on benchmark tasks, which is a reason to read agent scores with the configuration attached.
- What should you check first when an agent fails in a sandbox? Check the interface before the sandbox. Confirm that bytes, files, and payloads survive the round trip through the MCP server, then examine memory, permissions, and process limits only after the wrapper is ruled out.
- Is a coding agent reliable enough to run tests unattended? For this experiment, yes, with credential injection and failure triage set up beforehand and results written to a database. The run produced usable defect classes, and the agent also had to decide whether each failure came from the service, the MCP layer, or the test itself.
Turning the experiment into a habit
The value in Orellana's run is not the failure count or the platform it targeted. It is the habit: give the agent a protocol it already knows, give it a database to write into, and let it build workflows you would never have written by hand.
If you host agents in sandboxes, the sharp edges are already there. The only question is whether you find them during an overnight run or during a customer's first afternoon.
Where this goes next
If the knowledge from this talk lives in a recorded session rather than a written page, that recording can still become an article. Skalablog takes a YouTube URL, transcribes the video, and generates a draft you can review and publish.
That workflow fits exactly the problem this article describes: important findings trapped in a format that search engines and answer engines cannot index. Paste the link and the transcript becomes a structured piece with headings, sources, and a clear answer at the top.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits