claude-workman — computer use for Linux, as an MCP server

claude-workman is an open-source MCP server that gives an AI assistant human-mode control of a Linux desktop — it takes a screenshot, reads the accessibility tree, then moves the mouse, clicks, types, and switches windows. It works with Claude, Claude Code, and any other MCP client.

Named after the Walkman: a small, portable thing that just plays. Point it at a display and it works.

pip install claude-workman
claude mcp add --scope user workman -- python -m workman.server

---

Why accessibility-tree clicking beats pixel clicking

Most desktop automation guesses at pixel coordinates and breaks the moment a layout shifts, a font renders differently, or a window moves. claude-workman fuses two signals on every turn — a screenshot and the AT-SPI2 accessibility tree — so an action can be addressed semantically:

"click the push button named Continue"

…and land on that button's center, wherever it moved to. This is the same principle behind macOS AXUIElement-based computer use, ported to Linux via AT-SPI2 (gi.repository.Atspi).

Approach	Addressing	Survives layout change	Used here
Pixel-only automation (raw xdotool, image matching)	guessed x, y	❌ brittle	fallback only
Accessibility-tree automation	role + name → element	✅ robust	default

Tools

Tool	What it does
screenshot / screenshot_region	Capture the display (or a sub-rectangle) as PNG; max_dim downscales for faster round-trips
accessibility_tree	Actionable elements as {app, role, name, x, y, w, h} in screen coordinates
click_element	Find an element by role + name and click its center — no pixel guessing
enable_accessibility	Turn on AT-SPI tree export (and silence the screen reader — see safety)
list_windows	Windows with id, name, pid, geometry
focus_window	Raise a window by id or name; minimizes the frontmost blocker first
click / move / drag / scroll	Pointer actions at pixel coordinates
type_text / press_key	Keyboard input — press_key uses xdotool syntax (Return, ctrl+c, super+l, KP_0)
show_cursor	Visual click cursor overlay: a ring + click ripple showing exactly where the agent is acting

Watch what it clicks

show_cursor starts a transparent, click-through overlay that draws a cyan ring at the action point and an expanding ripple on every click, with a label. It exists for human oversight — you can watch an agent work and interrupt it, instead of guessing what it just did.

Requirements

Linux with X11, plus:

sudo apt install xdotool ffmpeg gir1.2-atspi-2.0 python3-gi
# at-spi2-core is normally already running on GNOME/KDE

• ffmpeg — screen capture (x11grab)
• xdotool — mouse and keyboard input
• python3-gi + gir1.2-atspi-2.0 — the accessibility tree (import is gi.repository.Atspi, not pyatspi)
• Pillow (optional) — only for max_dim screenshot downscaling
• GTK 3 (optional) — only for the show_cursor overlay

Install

pip install claude-workman          # from PyPI
# or from source:
git clone https://github.com/AdvisorAGI/claude-workman.git
cd claude-workman && pip install -e .

Usage

Run the stdio MCP server:

python -m workman.server

Register it with an MCP client — for Claude Code:

claude mcp add --scope user workman -- python -m workman.server

Target a specific X display with WORKMAN_DISPLAY (falls back to DISPLAY, then :0):

WORKMAN_DISPLAY=:1 python -m workman.server

The accuracy loop

1. screenshot — and enable_accessibility + accessibility_tree when precision matters
2. Locate — prefer an element by role + name; fall back to pixels
3. Act — click_element when possible, else click / type_text / press_key
4. screenshot again to verify before anything irreversible

Speech hazard (handled)

Exporting the accessibility tree from Chromium and Electron apps requires the desktop screen-reader flag — and that same flag makes Orca read the screen aloud through speech-dispatcher → espeak-ng. enable_accessibility therefore stops Orca immediately after enabling, using an exact process-name match (pkill -x orca, never pkill -f orca, which self-matches its own shell command and aborts).

GTK applications export their trees with toolkit-accessibility alone, so the web flag is opt-in.

FAQ

What is claude-workman?

An open-source MCP server that lets an AI assistant control a Linux desktop the way a person does — screenshot, locate, click, type, verify — with accessibility-tree targeting so clicks hit real UI elements instead of guessed coordinates.

How is this different from browser automation?

Browser tools (Playwright, Puppeteer, CDP) drive a web page. claude-workman drives the whole desktop: native apps, terminals, file managers, settings dialogs, and browsers alike.

Does it work with Claude Code?

Yes. Register it with claude mcp add --scope user workman -- python -m workman.server. It works with any MCP client, not only Claude.

Does it support Wayland or macOS?

Not yet — X11 today. A Wayland backend and a macOS AXUIElement backend are the natural next steps, and contributions are welcome.

Is it safe to let an agent control my desktop?

Treat it like handing over mouse and keyboard. Use show_cursor so you can see every action, prefer a dedicated display or VM for unattended runs, and keep the verify-by-screenshot step before anything irreversible.

Why is it called Workman?

After the Walkman — a small portable thing that just plays. Point it at a display and it works.

Does it solve CAPTCHAs?

No. It is a general desktop-control server, not an anti-bot bypass tool.

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md. Good first areas: a Wayland backend, a macOS AXUIElement backend, richer AT-SPI role coverage, and tests.

License

MIT © 2026 Atmosphere AI — see LICENSE.

---

_{Keywords: MCP server, Model Context Protocol, computer use, desktop automation, Linux automation, X11 automation, AT-SPI2 accessibility, GUI agent, AI agent desktop control, xdotool alternative, screenshot automation, Claude MCP connector, agentic computer use.}