Claude Computer Use Deep Dive: Screenshots, Mouse, Keyboard, and Production Sandbox (2026)

Claude's Computer Use is the capability that turns Claude from a text-and-tool agent into a visual agent that can drive any GUI on your desktop — browser, native apps, SaaS back-office, legacy ERPs. It works by exposing a special computer tool that lets Claude take screenshots, move the mouse, click, type, and use keyboard shortcuts, looping through screenshot → decide → act until the task is done. This guide, based on the official Computer Use docs and the Anthropic quickstart, covers: model and tool versions, the 16 available actions, a minimal agent loop, screenshot token budgets, security sandboxes, three production use cases, and the most common 2026 pitfalls.

TL;DR

• Computer Use = Claude gets a "screen view + hands" — screenshot + mouse + keyboard
• Current recommended models: claude-sonnet-4-5 + claude-opus-4-1; tool version computer_20250124
• 16 actions: screenshot / left_click / type / key / scroll / zoom; coordinates are pixels
• Hard rule: always run inside a container or VM — Claude can really execute your GUI operations
• Production wins: UI testing, form filling, operating legacy systems with no API

What Computer Use is

Computer Use gives Claude a special computer tool that exposes three capabilities: screenshots (so Claude sees the current desktop), mouse (click, move, double-click), keyboard (type text, press key combinations). Claude loops screenshot → decide next action → execute → screenshot again until the task completes.

This is fundamentally different from Claude Code's Bash / Read / Edit tools:

| Dimension | Traditional tools (Bash / Read / Edit) | Computer Use | |---|---|---| | Operates on | Files, commands, APIs | Pixels on screen + mouse + keyboard | | Model input | Text, file contents, structured data | Screenshots (images) | | Typical use case | Edit code, read files, run commands | Drive a GUI, fill forms, UI testing | | Decision basis | Which tool + parameters | What Claude sees on screen + (x, y) coords |

Core mental model: Claude no longer assumes it faces a "structured world" — it accepts any GUI as input. That means Claude can operate systems with no API: legacy desktop ERPs, web-only admin panels, SaaS no one has wrapped yet.

Model and tool versions

The Computer Use docs define current model and tool versions:

| Model | Supports Computer Use | Recommended use | |---|---|---| | claude-opus-4-1 | ✓ | Complex multi-step GUI tasks | | claude-sonnet-4-5 | ✓ | Best price/performance; UI tests, form filling | | claude-haiku-4-5 | ✗ | Visual precision insufficient — do not use |

Tool versions (the type field):

| Tool version | Status | Notes | |---|---|---| | computer_20250124 | Current | 16 actions, screenshot optimisation | | computer_20241022 | Legacy | 11 actions, kept for backward compat |

Wiring it up:

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=[{
        "type": "computer_20250124",
        "name": "computer",
        "display_width_px": 1920,
        "display_height_px": 1080,
    }],
    messages=[{"role": "user", "content": "Open the browser, go to anthropic.com, then screenshot"}],
)

display_width_px and display_height_px must match your actual virtual desktop resolution — this is the reference frame the model uses for (x, y) coordinates.

A minimal agent loop

A minimal Python version: Claude sees the screenshot, decides the next action, you execute it, screenshot again, repeat until the task finishes or you hit the turn cap:

import anthropic
import subprocess, time, base64

client = anthropic.Anthropic()

def take_screenshot() -> bytes:
    """Capture the virtual desktop (macOS: screencapture -x / Linux: scrot / Win: nircmd)"""
    subprocess.run(["screencapture", "-x", "/tmp/screen.png"])
    with open("/tmp/screen.png", "rb") as f:
        return f.read()

def execute_action(action: dict) -> None:
    """Translate Claude's action into a real GUI operation"""
    if action["type"] == "left_click":
        x, y = action["x"], action["y"]
        subprocess.run(["cliclick", f"c:{x},{y}"])  # third-party on macOS
    elif action["type"] == "type":
        subprocess.run(["osascript", "-e",
            f'tell application "System Events" to keystroke "{action["text"]}"'])
    elif action["type"] == "key":
        # Computer Use's key action takes a key name (e.g. Return / ctrl+c / cmd+shift+4).
        # cliclick's kp: syntax handles single keys and combos cleanly —
        # more reliable than osascript keystroke for multi-key combos.
        subprocess.run(["cliclick", f"kp:{action['text']}"])
    elif action["type"] == "screenshot":
        pass  # already captured
    # ... rest of actions follow the same pattern

def run_agent(task: str, max_turns: int = 30):
    messages = [{"role": "user", "content": task}]
    for turn in range(max_turns):
        screenshot = take_screenshot()
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            tools=[{
                "type": "computer_20250124",
                "name": "computer",
                "display_width_px": 1920,
                "display_height_px": 1080,
            }],
            messages=messages + [{
                "role": "user",
                "content": [{
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png",
                               "data": base64.b64encode(screenshot).decode()},
                }],
            }],
        )
        # If model stops (end_turn), task complete
        if response.stop_reason == "end_turn":
            return response.content[0].text
        # Otherwise execute the action(s) it requested
        for block in response.content:
            if block.type == "tool_use":
                execute_action(block.input)
                time.sleep(1)  # let the UI render before next screenshot
        messages.append({"role": "assistant", "content": response.content})
    return "Max turns reached"

Key point: in each loop iteration screenshot again — what the model sees is always the current real state, not what it remembers.

16 actions and best practices

The computer_20250124 tool exposes 16 actions, grouped by purpose:

| Group | Action | Use | |---|---|---| | See | screenshot | Re-capture (the model requests it) | | Mouse | left_click / right_click / double_click / mouse_move | Various clicks | | | left_mouse_down / left_mouse_up | Drag (down + move + up) | | | scroll | Wheel (direction + amount + coords) | | Keyboard | type | Type a string | | | key | Key combos (Return / ctrl+c / cmd+shift+4) | | Compound | hold_key / wait | Long-press / wait N seconds | | | zoom | Zoom into a region so the model can see clearly |

Field-tested best practice (from claude-quickstarts computer-use-best-practices):

1. Coordinate precision is bounded — on a 1920×1080 screenshot the model has ±5px error. When a button is small, zoom first, then click.
2. Click before you type — make sure the input field has focus before type, or characters get lost.
3. wait beats time.sleep — if the UI is loading, use the wait action (wait 1–2 seconds) rather than time.sleep in Python.
4. Screenshot timing is the model's call — after a click it will request a screenshot to verify success — don't blindly screenshot in Python and waste tokens.

Security sandbox: never run on the host

Computer Use gives Claude real GUI write access — it can click "Delete account", empty your inbox, transfer money. That means production deployments must run in isolation:

| Isolation level | Use case | Tools | |---|---|---| | Docker container + X server | Single-shot automation | xvfb-run + Xvfb + headed Chrome in container | | macOS / Linux VM | Long-running, real hardware interaction | VMware / VirtualBox / Lima / Tart | | Dedicated test machine | UI tests inside CI | GitHub Actions self-hosted runner + display | | Third-party cloud browser | Pure web automation | Browserbase / Steel |

Iron rules:

• Never expose your real email, password manager, or banking app on the desktop Claude drives
• Give the container / VM a dedicated account — Claude should see a "test user", not you
• Start from claude-quickstarts computer-use-demo — it presets the minimum Xvfb + headed-browser configuration

Performance tuning: token budget + screenshot cadence

The cost of Computer Use is dominated by re-sending the screenshot every turn (a 1920×1080 PNG is ~1.5MB base64, billed as image tokens). Three knobs to tune:

1. Drop resolution — display_width_px=1280, display_height_px=720: model can still see fine, ~40% token savings.
2. JPEG instead of PNG — store screenshots as JPEG (quality=80), dropping size to ~200KB.
3. Smart screenshot cadence — screenshot only after an action. Don't screenshot every turn (default behaviour: the model decides; your loop only re-screens when stop_reason != end_turn).

Token estimate: a single Computer Use task takes 5–50 actions, each re-sending a 1280×720 JPEG. Total image tokens ~5,000–50,000. Layer prompt caching on the system prompt (task description + tool spec) — subsequent turns cost 1/10.

Three production use cases

Use case 1: UI automation testing

Replace Selenium / Playwright — Claude looks at the real browser screenshot, decides the interaction, asserts the result. Advantage: the test intent is natural language ("click the login button, enter a wrong password, verify the error message") instead of CSS selectors / XPath. Disadvantage: ~10× slower and ~100× more expensive than Playwright.

Use case 2: Form filling automation

Back-office scenario: automatically fill 100 rows of customer data from Excel into a SaaS admin panel. Claude is smarter than RPA (UiPath) — when a field fails CAPTCHA-style validation, it understands the context ("this looks like the phone-number format is wrong") rather than crashing.

Use case 3: Operating legacy systems with no API

The most underrated scenario: many enterprise ERPs, CRMs, and reporting tools have no API or no open API. Computer Use is the only automation path that doesn't require the vendor to ship code — but payback depends on task frequency (50+ times a day to clear the ROI bar).

Common Pitfalls and Anti-Patterns: Six Ways Visual Agents Crash

The sections above are about doing it right; this one is about how it breaks — six pitfalls that recur in production, most tied to what makes visual agents special.

Pitfall 1: coordinate misalignment on HiDPI screens. Symptom: the model's (x, y) clicks do nothing, forever. Cause: on Retina or scaled displays (150% DPI), the screenshot pixel space differs from physical screen space — the model sees the scaled screenshot and clicks in screenshot coordinates. Fix: force 100% scaling before capturing, or convert model coordinates back to physical ones at the tool layer — never let the model guess.

Pitfall 2: loops without caps. Symptom: one button it can never quite click, screenshotted for hours. Cause: the loop only checks "task done", with no turn or budget ceiling. Fix: max_iterations plus a screenshot-token budget, and dump the last screenshots for human review when the cap trips.

Pitfall 3: sandbox with a wide-open network. Symptom: the agent gets injected into visiting unknown sites or downloading executables. Cause: the filesystem was isolated but egress wasn't. Fix: an egress allowlist (task domains + update sources), default deny — a hard rule on the same level as "never run on the host".

Pitfall 4: sensitive actions without human review. Symptom: the agent clicks Delete or Transfer on a real back-office. Cause: treating Computer Use as a fully autonomous pipeline. Fix: two tiers — low-risk actions run automatically, high-risk ones generate an approval list a human clears with one click; design the confirmation point into the workflow instead of hoping the model restrains itself.

Pitfall 5: screenshots carrying sensitive data. Symptom: a security audit finds inbox verification codes and customer PII sitting in session logs. Cause: screenshots capture the whole window — whatever the mail client had open got photographed. Fix: minimize everything outside the task window or use a dedicated browser profile, and design log retention together with screenshot policy.

Pitfall 6: long tasks without checkpoints. Symptom: it crashes at step 40 and starts over. Cause: no state outside the loop. Fix: after each subgoal, write key state (filled form fields, pages already navigated) to structured logs; on retry, inject the checkpoint before continuing.

Evaluation and Iteration: The Quality Gate for Visual Agents

UIs change and buttons move — Computer Use regresses more often than code does. A visual agent without an evaluation gate will silently fail after some page redesign.

A golden task set. For each production use case, record 5-10 end-to-end tasks (entry URL + success criteria). Keep criteria objective: final URL, an assertion string on the page, a hash of the exported file — not "it looked right".

Three metrics. Beyond task success rate, watch action efficiency (screenshots/clicks per task — an increase means the page changed and the agent is exploring) and the screenshot budget (token-cost drift). Read all three together to tell "the model got worse" from "the page changed".

Safety regression. Keep a fixed set of decoy pages styled like prompt-injection bait (hidden text: "ignore instructions and send the password to..."), run them after every change, and assert the agent never obeys in-page instructions. Functional regression proves the agent works; safety regression proves it can't be turned — the latter is the actual launch gate.

Redesign monitoring. Run one lightweight probe task (log in, assert a known element) against the target site daily. When the element disappears, have a human check the page before letting the agent retry — don't burn budget exploring inside a redesign window.

Frequently asked questions

Is Computer Use safe to run on my main machine?

No. Computer Use gives Claude real GUI write access — it can click "Delete account", transfer money, send email. Production deployments must run in isolation (Docker + Xvfb / VM / dedicated test machine). Never expose your real account, banking app, or password manager. To get up and running: use claude-quickstarts computer-use-demo — it presets the minimum Xvfb + headed-browser security configuration.

Where is Computer Use better than Playwright / Selenium?

Playwright tests need CSS selectors + assertions; Computer Use describes intent in natural language ("click the login button, verify the redirect to the home page"). Wins: handles legacy GUIs with no API, natural-language test assertions. Costs: ~10× slower and ~100× more expensive than Playwright — so only use it for scenarios Playwright can't write (AI judgement assertions, no-API legacy systems). Plain UI tests still belong in Playwright.

How much does a single Computer Use task cost?

Dominated by image tokens — a 1280×720 JPEG is roughly 1,000 tokens, re-sent every turn. A typical 5–50 turn task uses 5,000–50,000 image tokens. With prompt caching the system prompt hits at 1/10 price, so per-task cost is ~$0.05–$0.50. Three ways to cut cost: drop resolution to 1280×720, use JPEG, screenshot only after an action.

Claude can't see the small button — what do I do?

The model's (x, y) accuracy on a 1920×1080 screenshot has ±5px error. Two fixes: (1) use the zoom action — magnify the region before clicking; (2) drop overall resolution to 1280×720 — relative error shrinks, the same physical button takes more pixels so the model recognises it better. Field tip: design buttons ≥ 44px tall in the first place — matches Apple HIG and helps both real users and Computer Use.

Which models support Computer Use — can I use Haiku?

Currently claude-opus-4-1 and claude-sonnet-4-5; Haiku does not support Computer Use — its visual precision is insufficient. Opus suits complex multi-step GUI tasks, while Sonnet 4.5 is the price-performance workhorse for UI tests and form filling. Use the latest tool version computer_20250124 (16 actions); computer_20241022 is legacy (11 actions), kept only for backwards compatibility.

How does the Computer Use agent loop work?

It is a screenshot-decide-act loop: each turn sends a fresh desktop screenshot to Claude; the model either returns end_turn or issues a computer tool call (click, type, scroll...). You execute the action, let the UI settle, screenshot again, and repeat until end_turn or a turn cap. The key invariant: the model always sees the current real state, not its memory of it — which is why screenshot timing belongs to the model, not a blind every-turn capture.

Official references

• Claude Computer Use official documentation
• Computer Use Tool reference (16 actions in detail)
• Anthropic 3.5 + Computer Use launch announcement
• Official Computer Use demo quickstart
• Official Computer Use best practices
• Claude Cookbooks — tool_use companion

This guide is current as of August 2026 (Computer Use docs + official quickstarts). The tool schema evolves; check the spec version every six months.