Computer use & browser agents

The next frontier of agentic AI: instead of giving the model APIs, give it a screenshot, a mouse, and a keyboard. The model uses your computer the way a human does.

What computer use actually does

Computer-use agents drive software the way a person would — by looking at screenshots, clicking buttons, and typing on the keyboard. No API, no integration, no "let's add this to our roadmap." If you can do it, the model can attempt it.

That's a big deal for the long tail of software that never had decent APIs: legacy enterprise tools, browser-only services, anti-bot-protected portals, anything where "automate this" used to mean either fragile scraping or a contract negotiation. Computer use bypasses all of that — at the cost of reliability (a 20-step task with 95% per-step accuracy succeeds end-to-end about a third of the time).

That's the whole concept. Below: the loop, the major products, where it works today, and the brand-new attack surface it opens up.

When you'd reach for it

Computer use earns its keep when the alternative is "no automation":

• Software with no API — legacy enterprise tools, internal SaaS, vendors that won't give you access.
• Cross-app workflows — pulling from Slack, processing in Notion, sending via email.
• Browser-only tasks — booking, shopping, research with logged-in sessions.
• One-shot personal tasks — "find me a flight under £300, book it" — where building an integration would be absurd.

You wouldn't reach for it when a real API exists (always faster, cheaper, more reliable), or when the task is high-stakes-irreversible without human approval gates. Computer use is also bad at CAPTCHAs, drag-and-drop, and unusual UI components.

How it's actually built

The agent loop, adapted for computer use:

Loop:
  1. Take a screenshot of the screen (or a specific window/browser).
  2. Send the screenshot + the user's goal + the conversation so far to the model.
  3. The model produces one of: click(x,y), type(text), key(combo), scroll, wait, done.
  4. The harness executes that action on the real machine.
  5. Take another screenshot. Loop.

Three things make this hard:

• Vision quality. The model has to read the screen accurately — small text, low contrast, dropdown menus, partial state. OCR-grade reading inside a multimodal model.
• Spatial precision. Click coordinates need to be exactly on the right button. A 5-pixel miss = wrong action.
• Sequence reliability. A 20-step task fails if any step is wrong. Multiplicative.

Modern computer-use models are trained specifically on screenshot-action pairs to get this right. They've gone from "barely works" in 2024 to "useful for many tasks" in 2026, with steady progress on accuracy and speed.

The current landscape (mid-2026)

• Anthropic Computer Use (Oct 2024) — Claude variant trained to operate a computer via screenshots. Open-source reference; runs in your Docker container.
• OpenAI Operator — hosted browser agent in OpenAI's cloud sandbox. Strong fit for one-shot tasks like booking; doesn't reach your machine.
• Browser-Use, Stagehand — open-source browser-only frameworks using a multimodal LLM + Playwright. Active ecosystem, cheaper for browser-only work.
• Plus various startup products (Multi-on, Replit Agent) doing flavours of the same.

Cloud-hosted is zero-setup but the vendor sees everything; self-hosted runs on your machine with your sessions but you own the sandbox, security, and recovery logic. Production deployments often combine: hosted model, locally-controlled execution.

Strong / weak today

Where it bites in real life

Under the hood (optional)

A skeleton call to Anthropic's computer-use model: declare a computer tool with screen dimensions, send a goal, the model returns tool_use blocks describing actions (screenshot, click, type) which your harness executes on the real machine. Skip if you don't code.

import anthropic
 
client = anthropic.Anthropic()
 
response = client.beta.messages.create(
    model="claude-3-5-sonnet-20241022",   # or current computer-use model
    max_tokens=4096,
    tools=[
        {"type": "computer_20241022", "name": "computer",
         "display_width_px": 1920, "display_height_px": 1080},
    ],
    messages=[{"role": "user",
               "content": "Open Firefox, navigate to claude.ai, take a screenshot."}],
)
 
# response.content includes tool_use blocks like:
#   {"type": "tool_use", "name": "computer",
#    "input": {"action": "screenshot"}}
# Your harness executes that on the real machine and feeds back the result.

Try it yourself

If you want to see this for real, the easiest paths:

1. OpenAI Operator (paid product, US first): give it a research / booking task.
2. Browser-Use (free, open-source): clone the repo, install, give it a browser task. Bring your own LLM API key.
3. Anthropic's reference Computer Use repo (github.com/anthropics/anthropic-quickstarts): runs in Docker. Fully local.

Try a simple task — "find the price of organic almond butter on Amazon" — and watch the agent navigate. Then try something harder. You'll quickly feel the boundary of what's reliable.

Further reading

• Anthropic — Introducing computer use — original Computer Use announcement and design discussion.
• OpenAI — Operator launch — design and product announcement.
• Browser-Use — popular open-source browser agent framework.
• Stagehand (BrowserBase) — TypeScript-native browser AI framework.
• WebArena benchmark — independent benchmark for measuring web-task agents.