Computer and browser use

A coding agent gets a purpose-built Agent-Computer Interface: clean tools for read, search, edit, run. But most software has no API and no clean tool - a legacy desktop app, a canvas UI, a website that only exists as pixels. Computer-use and browser agents close that gap by driving the GUI the way you do: look at the screen, decide, move the mouse, so one model can operate any application with no per-app integration. The catch is that "look at a screen and click the right pixel" is far harder and far less reliable than it sounds, and here a mis-click has real, irreversible effects.

The loop: perceive, decide, act, observe

A huge amount of real work has no endpoints: an internal admin panel, a vendor portal behind a login, a form that only exists in a browser. For those, the only interface is the one built for humans - the graphical one. computer use closes that gap. Strip away the specifics and it is the same agent loop as everything else in this course - a model in a loop with a tool^[1] - but the tool is unusual. Anthropic's Computer Use exposes a single schema-less computer tool: a screenshot goes in, a low-level UI action comes out.^[2] The harness executes the action against the real display and returns a fresh screenshot, and the loop turns again.

The reference harness is almost boringly small - the same "while there is a tool call, run it" skeleton from the agent loop:^[2]

def sampling_loop(model, messages, max_iterations=10):
    for _ in range(max_iterations):
        response = client.beta.messages.create(
            model=model, max_tokens=4096, messages=messages,
            tools=TOOLS, betas=["computer-use-2025-11-24"],
        )
        messages.append({"role": "assistant", "content": response.content})
        tool_results = process_tool_calls(response)   # runs screenshot / click / type
        if not tool_results:
            return messages          # no tool call -> task complete
        messages.append({"role": "user", "content": tool_results})
    return messages                  # hit the safeguard limit

process_tool_calls dispatches on the action name - "screenshot" captures the display, "left_click" clicks input["coordinate"], "type" types input["text"] - and returns each result as a tool_result. OpenAI's API frames the same handshake with different nouns: the model returns a computer_call, your harness executes it and replies with a computer_call_output carrying the new screenshot (send it at full resolution, detail: "original", for click accuracy).^[3]

The action space

The action space is the finite vocabulary of moves the model may emit - the GUI equivalent of a coding agent's tool list. Anthropic's computer tool grew it version by version, which is a nice map of what "operate a computer" actually requires:^[2]

OpenAI's CUA returns a near-identical vocabulary: click, double_click, drag, scroll, type, keypress, move, wait, screenshot.^[4] The action spaces have converged because the target - a human GUI - is the same.

Grounding: the hard problem

visual grounding is turning "click the Save button" into the exact pixel (219, 168), and it is the reason these agents are fragile. The model has to localise a target in a raster image with enough precision to land inside a 36-pixel-tall button. Anthropic trains the model to reason about coordinates relative to reference points, but three grounding hazards recur:^[2]

Two ways to see: pixels versus the accessibility tree

The whole field splits on how the agent perceives the page. One family reads pixels; the other reads the structure - the DOM or accessibility tree. This is the split between browser use and general pixel-based computer use. Same task, two representations:

Browser use is more reliable and cheaper when the target is a web page, because the structure is already machine-readable. Pixel-based computer use is the universal fallback: it is the only option for native applications, but it inherits every difficulty of grounding intent in raw coordinates. Many production systems blend the two - lead with the DOM, fall back to pixels for a stubborn canvas or an embedded widget.

Reliability and safety: be honest about the limits

Benchmarks make the state of the art concrete, and cut through the demo hype. Read the last row carefully:

Curated web tasks look strong, but OSWorld about 38% is the honest number for general computer use: most real desktop tasks still fail if you walk away. External sites block automation, CAPTCHAs stall the agent, and a small UI change derails a memorised path - Project Mariner ships with exactly these warnings.^[8] Treat these agents as capable-but-supervised, not autonomous. This mirrors the benchmark-versus-reality gap from the evaluation lesson.

1. Grounding error. The model picks the right action but the wrong pixel, clicking a nearby link or an ad instead of the button it intended.
2. Latency and cost. Every step is a full model call over a large image, so a ten-step task is ten screenshots and ten inferences. Computer-use agents are slow and token-heavy compared with an API call.
3. Timing and dynamism. Pages load asynchronously. Act before content settles and the click hits stale layout, so the agent must wait, re-observe, and confirm state rather than fire actions blindly.
4. Blockers. Cookie banners, captchas, infinite scroll, and popups sit between the agent and its goal, and each is a place a long task can stall or derail.

Check yourself