Computer use agents are AI systems that operate a full desktop through screenshots, accessibility trees, keyboard and mouse instead of APIs. Simular, the company Ang Li co-founded in 2023 and now runs, pairs a vision-language agent for exploration with a symbolic coding layer that replays proven steps as deterministic code, and sells that combination through its Sai product, which is still in closed beta.
Replay is the reliability question, and the answer is: not unconditionally. A vision-language agent explores and finds a path; the symbolic coding layer replays that path as deterministic code. So a replay succeeds only as far as the recorded steps stay valid against the live interface.
| Layer | How it acts | Replay behaviour |
|---|---|---|
| Vision-language agent | Screenshots, accessibility trees, keyboard and mouse | Re-derives the path each run; flexible but non-deterministic |
| Symbolic coding layer | Replays proven steps as code | Deterministic where the recorded steps still match the interface |
| Sai product | Ships both layers together | Closed beta; exploration plus code replay in one combination |
Computer use agents are software agents that operate a full desktop environment — reading screenshots or accessibility trees and issuing mouse and keyboard input — instead of calling an application programming interface. That pixel-and-pointer loop is what lets them drive software that was never built for automation.
What are computer use agents?
Simular, the company Ang Li co-founded in 2023 and leads as CEO, builds them around a vision-language model for exploration plus a symbolic code layer, not around API calls alone.
The distinction matters in practice. An API-based agent asks a service for data and receives structured output. A computer use agent has to look at pixels or a window tree, decide which control to hit, move the pointer and then confirm the result. That extra loop is what lets an agent work on tools that were never designed for automation, including legacy Windows ERP software and Windows-only line-of-business applications.
| Dimension | API-based agent | Computer use agent |
|---|---|---|
| Input | Structured response from a service | Screenshots, accessibility trees |
| Action | Function or endpoint call | Mouse and keyboard input |
| Confirmation | Return value | Re-inspection of pixels or window tree |
| Reach | Tools with a published API | Legacy Windows ERP, Windows-only line-of-business apps |
| Core loop | Request, parse, respond | Look, decide, act, verify |
Simular's framing splits the problem into two words. Autonomy covers context understanding, planning, recovery from mistakes and the absence of human supervision. Computer covers the machine the agent drives, whether a personal desktop or a cloud virtual machine. Put together as ordered steps, an agent run looks like this:
- Autonomy: understand the context of the task.
- Autonomy: plan the sequence of actions.
- Computer: act on the target machine via pointer or keyboard.
- Autonomy: recover from mistakes when a step fails.
- Autonomy: continue without human supervision.
The company built its own product around that split. Sai is the interface where the user configures agents and algorithms, and it is still in closed beta according to Li's workshop. The autonomy and computing halves converge there, which is why the company describes Sai as the entry point for both local and cloud machines.
Simular's answer to reliable computer use is redundancy rather than a single modality: Agent S reads the accessibility tree, Agent S2 reads pixels, and Li argues no one modality covers every case. The generations differ in grounding method, benchmark trajectory and model choice.
How Simular moves from Agent S accessibility grounding to Agent S2 vision
Simular's first generation agent reads the accessibility tree rather than the pixels, and Li says that choice produced about 20% success on the OSWorld benchmark in late 2024, when early 2024 runs sat near 5%. The accessibility tree is the structured description an operating system uses to render windows and controls, so it is text, not an image.
Grounding in that generation was ID-based. The harness linearizes the on-screen controls, gives each one an identifier, and asks a language model which identifier to click. This works because the tree is a lossless-enough description of the screen for many tasks, and because text matching is exact.
The second generation dropped the tree entirely and went vision-only, with a layered manager and worker structure that mirrors a human organization. Li reports that combining two foundation models inside one harness beat using one, and the second model was a 7-billion-parameter open-source model rather than a frontier system, which is why small teams took interest.
| Generation | Grounding input | Reported OSWorld success | Model choice |
|---|---|---|---|
| Agent S (early 2024) | Accessibility tree | near 5% | Language model over linearized control IDs |
| Agent S (late 2024) | Accessibility tree | about 20% | Language model over linearized control IDs |
| Agent S2 | Vision only | Not stated | Two foundation models, one a 7-billion-parameter open-source model |
The trade-off is unresolved. Li argues there is no single modality that covers every case: some applications deliberately withhold parts of the tree, while some screen states only appear visually. His comparison is a self-driving stack with lidar, radar and camera, where redundancy exists to prevent the fatal error rather than to win a benchmark.
What pass^k reliability means for repeat runs
Pass^k reliability measures whether the same agent completes the same task every time across repeated attempts, and Simular reports that OSWorld's best agents lose roughly 10 percentage points when measured this way. That is a different question from pass@k, which counts a task as solved if any single attempt succeeds.
The gap shows up in production conversations. Li says customers ask whether an agent can run a workflow daily on a server without supervision, and that is exactly the promise a pass@k score does not test. His team measured three attempts on the same tasks and saw degradation even for the strongest agents.
Prompt perturbation makes the numbers worse. When the phrasing of the request or the environment shifts, success falls further, which is the operational version of the same problem: a model that samples a probability distribution each run will not produce identical behaviour by default.
Simular's published answer is a clarification prompt. The agent is instructed to ask questions about ambiguous parts of the instruction while minimizing how many questions it needs, and Li reports that reliability improves across several agents, with the largest gain on one configuration using an OpenAI model.
How Simulan turns a workflow into replayable code
Simulan is an open-source language from Simular for expressing desktop workflows as deterministic code that an agent can write and later replay. Li compares it to the RPA industry's approach of having engineers script a process, with the difference that Simulan is meant to be a shared protocol rather than a vendor-specific format.
The practical flow is short. Install the package, then let a coding agent such as Claude Code write Simulan code from a natural-language instruction. In Li's demonstration, Claude Code opened the AI Engineer website, chose an accessibility snapshot over a screenshot for a summarization task, extracted the event list and revised the script after errors.
Replay is where the economics change. Once a workflow is stored as code, a repeat run performs no model call for the steps the code covers, so token consumption drops. Li describes this as stateful programming: each line executes against the live environment and checks the current state before proceeding.
The limitation is honest. A script that matched yesterday's interface can break when the application changes, which is why the code path carries verification steps and why a model-driven recovery loop exists beside it.
Neuro-symbolic execution: code first, model when code fails
Neuro-symbolic execution runs the deterministic code path first and only escalates to a vision-language model when the environment no longer matches the script. Simular calls this its current research direction, and Li describes the target as a single framework that is stable when Stable Diffusion required and adaptive when the environment changes.
The stability-plasticity trade-off explains why both layers exist. A learning system has to retain what it already does well and also adapt to new conditions, and Li says after years of continual-learning research at DeepMind he accepts there is no configuration that maximizes both at once.
In a live demonstration, an agent saved a workflow for finding the first speaker on a Tuesday conference schedule, then replayed it for Wednesday. Replay changed the day parameter, hit a filtering snag, and handed control back to the neural agent, which resolved it before the loop finished.
The failure mode is the normal case, not the exception. Li told the audience the codebase for this kind of agent will accumulate branch and loop logic that no human will maintain by hand, and that the agent is expected to keep the code working as conditions change.
Sai, SaiBox and the API for cloud machines
SaiBox is Simular's cloud virtual computer, and Sai is the interface and agent layer that operates it. Li says he runs about 20 machines in the background himself, and that the company released an API for controlling cloud machines during the workshop.
The interface and the compute are sold together but structured separately. On Simular's diagram, Sai handles planning and decisions while SaiBox supplies the desktop, and the same interface can attach to a machine the customer already owns. Simular markets one product where local and cloud computers are interchangeable from the agent's point of view.
Platform coverage is uneven and dated. Li says Windows and macOS are supported, Linux and Android are described as coming soon, and Android was shown in internal evaluation clips doing a drag action in a game and completing a multi-factor authentication flow in a virtual phone.
The legacy-software case is the commercial wedge. Li names insurance data retrieval and healthcare data entry into old Windows ERP systems as customer examples, and describes a collaboration with Microsoft around Windows 365 virtual machines for form entry. He also guesses that roughly 80% of applications live in legacy systems, which is an estimate rather than a measured figure.
Guardrails, irreversibility and what local execution does not prove
Simular's guardrail design stops an agent before an irreversible action and asks for human approval, and the same code-first principle governs it. Rules decide what they can, a model handles what the rules miss, and a person confirms the remainder.
Audience questions pushed on operating-system-level restrictions, where an organization wants a specific application blocked outright. Li's answer was that this requires control below the agent, and that a cloud Windows image is the natural place to enforce allowlists because the image can be modified before the agent runs.
Running the agent on your own machine or on a managed virtual machine does not by itself establish compliance with healthcare, financial or other regulated requirements. The architecture makes it possible to restrict applications and require approval on sensitive steps; the controls, logging and policy that a regulated deployment needs remain the responsibility of the application and the organization that operates it.
The two-factor authentication demonstration carries the same nuance. Filling an MFA challenge from a virtual phone works technically, and Li explicitly told the audience to use that capability with caution rather than to treat it as a general capability to enable.
FAQ
- What are computer use agents? Computer use agents are AI systems that operate a full desktop through screenshots, accessibility trees, keyboard and mouse input instead of APIs. Simular builds them by combining a vision-language model for exploration with a symbolic code layer for replay, which lets the agent work on software that exposes no API.
- Is Sai generally available? No. Ang Li described Sai as still in closed beta during the October 2026 workshop and handed the audience an invitation code, so access depends on getting in through the beta. Simulan, the workflow language, is published as open source separately.
- What is the difference between Agent S and Agent S2? Agent S, the first generation, reads the operating system accessibility tree and grounds actions by control identifier. Agent S2 discards the tree and works from vision only, with a layered manager and worker structure that combines two foundation models in one harness.
- Why does pass^k matter more than pass@k for production? Pass@k counts a task as solved if any attempt succeeds, which suits code generation where you keep the output that compiles. Pass^k requires every repeated run to succeed, which is what a daily unattended workflow needs, and Simular reports that strong agents lose about 10 percentage points under that stricter measure.
- Does replaying code remove the model entirely? Not completely. Code handles the steps it already encodes and performs no model call for them, but verification runs at each step and failures escalate to a vision-language agent that explores and repairs the script before the code path is used again.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits