News · SI
Vercel Labs releases agent-desktop, open-source computer use for coding agents
Vercel Labs has released agent-desktop 1.0, an Apache-2.0 Rust CLI and C library that lets AI agents read and operate macOS, Windows and Linux apps through the operating system's accessibility tree instead of guessing from screenshots. Actions run headless by default, so an agent can work in the background while you keep using the machine.
Vercel Labs has released agent-desktop, an open-source tool that lets AI coding agents see and operate native apps on macOS, Windows and Linux by reading the operating system's accessibility tree rather than guessing from screenshots[1][2]. Version 1.0.0 was tagged on 28 October[3], and Vercel's Chris Tate introduced it on 31 October as "agent-browser, for your whole computer"[5].
01 / 03
Refs instead of pixels
Most computer-use agents look at a screenshot and decide where to click. agent-desktop takes a different route. A snapshot command returns the app's UI as JSON, with a reference for every element an agent can act on, and the agent clicks, types or toggles that reference, then snapshots again to check the result[1].
Each reference belongs to the snapshot that produced it. When the interface changes, an action returns an error such as STALE_REF and the agent has to take a fresh snapshot, so a click can't quietly land on the wrong row[1]. Every command prints one JSON envelope, and errors say whether a retry is safe[1].
The tool is written in Rust and ships 60 command names covering observation, clicks and typing, keyboard and mouse, app and window management, notifications, the clipboard and session traces[1][4]. Each release also includes a C library, so a program can call it in process instead of starting the CLI for every command[1].
02 / 03
Works in the background
Actions are headless by default. They go through the accessibility API without moving the pointer or taking focus, and a --headed flag turns physical input on when it's needed[1]. Tate's launch post pitches that as agents running in the background while you keep working, with several agents using the computer in parallel[5].
Platform support isn't identical. macOS and Windows cover nearly every feature, while Linux works through AT-SPI and limits mouse input and window control to X11, with Wayland screenshots going through the desktop portal[1].
03 / 03
Getting it
It installs with npm install -g agent-desktop, which carries native binaries for x64 and ARM64, and the binary bundles its own agent instructions through agent-desktop skills get desktop[1]. It is licensed under Apache 2.0[1].
The project is blunt about privacy. Snapshots, screenshots, clipboard reads and traces contain whatever is on your desktop and should stay out of public logs. The native CLI doesn't need a remote model service, but an optional helper sends screen descriptions to whichever model provider you configure[4].
FAQ
Frequently asked questions.
01What is agent-desktop?
It is a command-line tool and C library from Vercel Labs that gives AI agents computer use on the desktop. A snapshot returns an app's accessibility tree as JSON with a reference for every element an agent can act on, and the agent clicks or types on those references rather than on pixel coordinates.
02Which operating systems does it support?
macOS, Windows and Linux, on x64 and ARM64. macOS needs Accessibility permission, Windows needs no grant for apps at the terminal's integrity level, and Linux needs the user's graphical session with at-spi2-core. Some features, such as mouse input and window control on Linux, are limited to X11.
03How do I install it?
Run npm install -g agent-desktop, which bundles the native binaries. Native archives are also on GitHub Releases. The project says official Homebrew and Scoop packages are being submitted but are not yet in the default catalogs.
04Will it take over my mouse and keyboard?
Not by default. Actions on references go through the accessibility API without moving the pointer or taking focus, and headless calls block foreground activation and synthesised input. A global --headed flag allows physical input when you want it.
05Does it send my screen to an AI model?
The native CLI does not need a remote model service. An optional helper in the repository sends goals and screen descriptions to a model provider you configure, and the project warns that snapshots, screenshots, clipboard reads and traces can contain private desktop content that should stay out of public logs.
That’s the story
Building with what’s new?
Spectrum Web Co ships products, places talent, builds social narratives and builds custom business tools as monthly services.
Related stories