This post was written on Oct 22, 2025.
Models/pricing/policies may have changed. Check the latest anthropic posts.
Claude Computer Use Tool: AI Takes the Mouse
Claude now sees the screen, clicks the mouse, and types on the keyboard. We preview the revolution in automation brought by the 'Computer Use' tool.
Can legacy software without APIs be automated with AI? (Problem) Anthropic gave Claude eyes (screenshot analysis) and hands (mouse/keyboard control) through the Computer Use tool. (Solution) Now Claude can open a web browser, launch Excel, log in to an internal ERP system, and click buttons just like a human, without API integration. (Evidence)
Features of Computer Use
1. Visual Grounding
Claude analyzes screen screenshots in real-time to find the coordinates (x, y) of buttons. If you say "Click the login button," it accurately identifies the login button somewhere on the screen.
2. Precise Control
- Delicate manipulation is possible through commands like
mouse_move,left_click,double_click,type. - Supports complex interactions like drag-and-drop and scrolling.
3. Bash Tool Integration
It doesn't just use the GUI; it can also perform hybrid tasks by opening a terminal (Bash) in the background to install necessary packages or manipulate the file system.
Use Cases
- Replacement for RPA: You can automate repetitive data entry tasks with just Claude, without expensive RPA tools.
- Software Testing: Like a QA engineer, it performs E2E tests by running apps, pressing buttons, and checking for errors.
This technology means AI can handle all software in the digital world in the same way as humans. A truly 'General Purpose Agent' has been born.
Get updates
A weekly digest of what actually matters.
Found an issue? Report a correction so we can review and update the post.