Computer Use Explained: How AI Agents Control Your Screen
A plain-English guide to computer use, the AI capability that lets models read a screen and operate the mouse and keyboard, and why Anthropic, OpenAI and Google all treat it carefully.

Computer use is an AI capability that lets a model see a screen the way a person does, by reading screenshots, and then act on it, by moving a cursor, clicking, scrolling and typing, so it can operate ordinary software instead of calling a tailored API. Anthropic, OpenAI and Google each now offer a version of it, built on roughly the same loop: screenshot in, coordinate-based action out, repeat until the task is done or a safety check stops it.
What computer use actually means
Most AI tools work by calling an API: a weather app, a calendar, a code interpreter. Computer use skips that integration step entirely. Instead of asking a developer to build a connector, the model is given control of a real (or virtualized) desktop or browser and told to finish a task the way a human contractor would: by looking at what is on screen and operating the mouse and keyboard directly.
Anthropic introduced the idea publicly on October 22, 2024, describing it as letting Claude "use computers the way people do, by looking at a screen, moving a cursor, clicking buttons, and typing text," and calling Claude 3.5 Sonnet the first frontier model to offer the capability in public beta. OpenAI and Google have since shipped their own equivalents, and all three vendors frame it the same way: a general-purpose fallback for the huge number of tasks that live inside software with no API at all, from legacy enterprise tools to one-off web forms.
How the perception-action loop works
Strip away the branding and every vendor's implementation runs the same cycle:
- Screenshot. The client application captures the current screen (or browser viewport) and sends the image to the model along with the task and recent action history.
- Reasoning and action. The model looks at the pixels, decides what a person would do next, and returns a structured action, such as a click at a pixel coordinate, a string to type, a scroll, or a key press.
- Execution. The client application, not the model, actually performs that action against the real environment: a Docker container, a virtual machine, or a sandboxed browser tab.
- Repeat. A fresh screenshot goes back to the model, which checks whether the task is done, adjusts for anything that changed, and issues the next action.
This is more expensive than it sounds: Anthropic notes that every screenshot the model reviews adds roughly 1,000 to 1,800 input tokens, so a long multi-step task can burn through a meaningful amount of context just on "looking." It is also why coordinates matter so much in practice — actions are specified in screenshot pixel space, so if an application scales an image down before sending it, it has to scale the model's coordinates back up before clicking, or every click lands in the wrong place.
Because the model is reasoning from pixels rather than a clean API response, computer use is naturally imprecise compared with a purpose-built integration. Anthropic has been candid that early versions were "cumbersome and error-prone," and struggled specifically with scrolling, dragging and zooming — the fiddly, physical parts of using software that people do without thinking.
Who offers computer use today
All three major labs now ship some version of this, but they differ in packaging, scope and current model: Anthropic's is a general desktop-and-browser toolset built into the Claude API, OpenAI's runs through its Responses API using a frontier model rather than a dedicated preview, and Google's is tuned mainly for browsers and mobile UIs rather than a full desktop.
| Vendor | Product / tool name | Where it runs | Scope today |
|---|---|---|---|
| Anthropic | Computer use tool (computer_toolset_20260801) | Claude API, Amazon Bedrock, Google Cloud, Microsoft Foundry | Full desktop: screenshot, click, drag, scroll, type, key presses, zoom |
| OpenAI | Computer tool, Responses API | OpenAI API (Responses endpoint) | Browser and desktop actions; dedicated computer-use-preview model was retired, now runs on a current frontier model (gpt-5.6-terra) |
Gemini 2.5 Computer Use (computer_use tool) | Gemini API via Google AI Studio and Vertex AI | Public preview; optimized for web browsers, promising on mobile, "not yet optimized for desktop OS-level control" |
Anthropic: Claude's computer use tool
Claude's version has moved well past its original 2024 beta. The current toolset, documented as computer_toolset_20260801, bundles 17 member tools under a single entry in the API's tools array, covering screenshot capture, left and right clicks, click-and-drag, mouse movement, scrolling, typing, individual key presses, held keys, waiting, cursor position checks and zooming into a region of the screen. Claude can chain several of these in a single turn — for example click, then type, then screenshot — and the calling application executes each one in order before handing results back.
The feature is available across Claude's own API as well as Amazon Bedrock, Google Cloud's Vertex AI and Microsoft Foundry, and newer model versions no longer require a beta header to use it, a sign the capability has graduated out of pure experiment status. Anthropic's own reference implementation, published as an open-source quickstart on GitHub, runs the controlled desktop inside a Docker container with a virtual display, giving developers a ready-made sandbox to test against rather than pointing Claude at a real machine.
On Anthropic's original OSWorld benchmark, which scores how well an agent completes real desktop tasks from screenshots alone, Claude 3.5 Sonnet scored 14.9% in the screenshot-only setting — well ahead of the next-best system's 7.8% at the time, though both numbers underline how unsolved the underlying problem still is.
OpenAI: from Operator to a frontier-model computer tool
OpenAI's path here has been the most roundabout of the three. It first shipped the capability as a standalone product called Operator in January 2025, built on a dedicated model, then folded that web-control capability into a broader "ChatGPT agent" product and later retired Operator as a separate experience. On the developer side, the specialized computer-use-preview model that powered those products has itself since been deprecated: OpenAI's own deprecations list shows computer-use-preview shut down on July 23, 2026, with its workloads pointed to the current frontier model, gpt-5.6-terra, instead.
The mechanics for developers are close to Anthropic's: a computer tool call returns an ordered list of actions such as click, type or scroll, the application executes them and captures a fresh screenshot, and that screenshot comes back as a computer_call_output tied to the same call ID so the loop can continue. OpenAI's current guidance, aimed at developers building on its newest models, actually steers them toward a different pattern first — letting the model write and execute code in a persistent sandboxed environment — and treats the pixel-level computer tool as a fallback "alternative" for interfaces that have no programmatic hook at all.
That shift matters for anyone comparing this space to coding agents like Codex: code execution against a real environment is generally faster and more reliable than clicking through a UI pixel by pixel, so screen-level computer use tends to be the option of last resort rather than the default, even for the vendor that popularized consumer-facing "agent mode" browsing.
Google: Gemini 2.5 Computer Use and Project Mariner
Google's entry is the Gemini 2.5 Computer Use model, a specialized model built on Gemini 2.5 Pro that Google DeepMind announced in public preview on October 7, 2025, through the Gemini API in Google AI Studio and Vertex AI — distinct from the general-purpose Gemini 4 model lineup most people use day to day. It is the same underlying technology that has been powering Project Mariner, Google's browser-agent research project, as well as the Firebase Testing Agent and some agentic features inside AI Mode in Search.
Developers call it through a computer_use tool that runs in the same kind of loop as the others: the model receives the user's request, a screenshot and recent action history each turn, and typically returns a function call representing a single UI action. For anything sensitive, like a purchase, it can pause and ask the end user to confirm before continuing. Google has been explicit that, for now, the model is "mainly optimized for web browsers," shows promise on mobile interfaces, but is "not yet optimized for desktop OS-level control" — a narrower scope than Anthropic's full-desktop toolset. Google reports it leading rival systems on agentic web and mobile benchmarks including Online-Mind2Web, WebVoyager and AndroidWorld, with results drawn from a mix of its own testing and third-party evaluation harnesses.
The real risks: prompt injection, scams and why sandboxing matters
The risk that gets the most attention from all three vendors is indirect prompt injection: because the model is reading everything on the screen as potential input, text embedded in a webpage, a document, or even an image can look exactly like an instruction to it. A malicious pop-up, a hidden comment, or a cleverly worded support ticket can try to redirect the agent away from the user's actual request — toward entering payment details on a fake site, approving a permission it shouldn't, or exfiltrating data it had access to.
Every vendor responds with some combination of the same three defenses:
- Sandboxing. Running the agent in an isolated container, virtual machine or disposable browser profile, with no access to the user's real credentials, files or accounts, and a limited allowlist of reachable sites.
- Confirmation gates. Requiring an explicit human check before "consequential" actions — payments, agreeing to terms, accepting cookies, deleting data — rather than letting the model complete them autonomously.
- Automated screening. Anthropic runs classifiers over screenshots and tool outputs specifically to flag likely injection attempts and push Claude to confirm that an instruction really came from the user before acting on it; Google describes an equivalent per-step safety service that reviews each proposed action before it executes.
None of this makes the problem solved. Independent security researchers testing current computer-use agents have continued to find real susceptibility to indirect prompt injection on real webpages, particularly when a malicious instruction is disguised as something closely related to the task the agent was already doing — exactly the kind of social-engineering trick that is hard for a screenshot classifier to catch every time. That is a meaningfully different risk profile from ordinary chatbot prompt injection in an AI browser: a chatbot that gets tricked might say something wrong, while an agent with keyboard and mouse control that gets tricked can actually do something wrong.
- Anthropic introduced computer use publicly on October 22, 2024, alongside Claude 3.5 Sonnet and Claude 3.5 Haiku.
- OpenAI's dedicated
computer-use-previewmodel was shut down July 23, 2026; the computer tool now runs on its current frontier model instead. - Google's Gemini 2.5 Computer Use model entered public preview on October 7, 2025, and is tuned mainly for browsers and mobile, not full desktops.
- All three vendors recommend running computer-use agents in a sandboxed, low-privilege environment with no stored credentials.
What you can realistically do with it today
For most people, the honest summary is: this is still a developer and power-user capability, not a one-click consumer feature you should point at your everyday banking or email without supervision. The uses that make sense right now tend to be low-stakes and well-bounded: automated QA testing of your own web app's UI, filling out repetitive internal forms, scripted demo walkthroughs, or research tasks on a disposable browser profile where nothing sensitive is at risk if the agent gets something wrong.
If you want to try it yourself, the practical path depends on which ecosystem you are already in. Developers on Claude can start from Anthropic's open-source quickstart container and the computer use tool documentation; OpenAI's Responses API documentation covers its current computer tool and the code-execution alternative it now recommends trying first; Google's Gemini API documentation covers the computer_use tool for browser- and mobile-focused agents. In every case, the setup work — an isolated environment, a tight allowlist, no stored passwords — is not optional polish, it is the actual safety mechanism, since the model itself cannot reliably tell a legitimate instruction from an injected one every time.
What's next
Expect the gap between "screenshot-and-click" computer use and ordinary API-based automation to keep narrowing from both directions. Vendors are already pushing agents toward writing and running code against a sandboxed environment when an API or script is available, and saving pixel-level control for the genuine long tail of software that has no other hook — which is exactly OpenAI's current stance and likely where Anthropic and Google head too as their own code-execution tools mature. On the safety side, watch for sandboxing and confirmation requirements to get stricter rather than looser as adoption grows, since the one scenario every vendor is explicitly trying to avoid is an agent that can click "buy," "send," or "delete" on the strength of an instruction it found on a random webpage.
Frequently asked questions
What is computer use in AI?
Computer use is a capability that lets an AI model see a screenshot of a computer screen, decide what action a person would take next, and then click, type, scroll or drag to carry it out, repeating that loop until a task is finished. It lets the model operate ordinary software directly instead of requiring a custom API integration.
Which AI companies currently offer computer use?
Anthropic (Claude's computer use tool), OpenAI (a computer tool available through its Responses API) and Google (the Gemini 2.5 Computer Use model) each offer a version. Anthropic's covers a full desktop, while OpenAI's and Google's are currently more focused on browser and, in Google's case, mobile interfaces.
What happened to OpenAI's Operator?
OpenAI launched Operator as a standalone product in January 2025, then folded its web-control capability into a broader ChatGPT agent product and retired the separate Operator experience. The dedicated computer-use-preview model behind it was later deprecated too: OpenAI's deprecations page lists it as shut down on July 23, 2026, with workloads moved to its current frontier model.
Is Claude's computer use tool open source?
The tool itself is part of Anthropic's commercial API, but Anthropic publishes an open-source reference implementation, the computer-use-demo quickstart on GitHub, which runs a sandboxed desktop in a Docker container so developers can test the feature safely.
Is it safe to let an AI agent control my computer?
Only with precautions. Every vendor recommends running computer-use agents in an isolated, low-privilege environment with no access to real credentials, plus human confirmation before sensitive actions like payments. The main risk is prompt injection: text hidden on a webpage or in a document can try to redirect the agent, and independent researchers have found current safeguards do not catch every attempt.
Can computer use agents browse the open web unsupervised?
They can technically be pointed at the live web, but all three vendors advise restricting network access to an allowlist of trusted sites and keeping a human in the loop for anything consequential, because an agent reading an untrusted page can be manipulated by instructions embedded in that page.
Sources
- Anthropic - Computer use tool documentationplatform.claude.com
- Anthropic - Introducing computer use, Claude 3.5 Sonnet and Claude 3.5 Haikuanthropic.com
- Google DeepMind - Introducing the Gemini 2.5 Computer Use modelblog.google
- OpenAI developers - computer-use-preview model referencedevelopers.openai.com
- OpenAI developers - Computer use tool guidedevelopers.openai.com
- OpenAI developers - API deprecationsdevelopers.openai.com
Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.
