desktop-mcp

Desktop vision and control for LLMs on Linux, including Wayland — the model takes a screenshot of the whole screen and clicks, types and scrolls on what it sees. Plain MCP over stdio, so it works in Claude Code, Claude Desktop, Cursor, or any other MCP client.

Read this first: you may not need this

Anthropic ships computer use natively, and where that works you should use it instead — it is better integrated and actually maintained:

• macOS and Windows: Claude Desktop has computer use built in, in Cowork and Claude Code. It drives your real desktop. (docs)
• Building your own agent: the Claude API has a first-party computer tool (GA as the computer_toolset_20260801 toolset). (docs)

The gap this fills is Linux. As of August 2026 computer use is not available in the Linux desktop app, and desktop automation on Wayland is genuinely awkward in a way X11 never was:

• COSMIC's compositor does not implement wlr-screencopy, so grim fails outright and every screenshot tool needs a per-compositor path.
• There is no protocol to query the cursor position, and none to enumerate another client's windows.
• Input has to go in below the compositor via /dev/uinput, because the commonly packaged ydotool exposes a relative pointer whose long jumps get mangled by pointer acceleration.

If you are on macOS or Windows, this repo is mostly redundant. If you are on Linux — particularly Wayland — it is the working option.

Desktop-automation MCP servers are not a new category; there are thousands of MCP servers. What is specific here is the Wayland-native absolute-uinput input path, measured pixel-exact on COSMIC.

Status

Written to be cross-platform, but only one configuration has actually been exercised. The rest is code paths, not evidence — treat it accordingly.

Platform	Screenshot	Input	Tested
Linux / Wayland	cosmic-screenshot, grim, spectacle, GNOME D-Bus	/dev/uinput	Yes — COSMIC only
Linux / X11	maim, ImageMagick import, mss	/dev/uinput, xdotool	No
macOS	screencapture	pyautogui	No
Windows	mss, PowerShell	pyautogui	No

What was tested, and how

Verified end-to-end on:

• Pop!_OS 24.04 LTS, kernel 7.0.11, COSMIC (cosmic-comp 1.0.0), Wayland, single 2560x1440 display, Python 3.13.

Results:

Check	Method	Result
Screenshot	cosmic-screenshot backend	2560x1440 captured, downscaled to 1400x788
Pointer accuracy	5 target points, read back through XWayland	5/5 exact, 0px error
Buttons and scroll	xev event log	ButtonPress/Release for buttons 1, 3, 4
Drag	xev motion count	26 MotionNotify events between press and release
Typing	Both wtype and raw uinput keycodes	Text landed in a focused editor
Chords	ctrl+a in an editor	Applied
Click via MCP	Full stdio round trip against a native Wayland app	New tab opened

Untested and worth knowing: multi-monitor layouts, HiDPI scaling, non-US keyboard layouts on the raw-keycode path (wtype handles those correctly; uinput keycodes assume US), and every non-COSMIC compositor.

Reports from other platforms are welcome — desktop_info dumps everything needed to diagnose a backend problem.

Install

git clone <this repo> && cd desktop-mcp
uv venv && uv pip install -e .

macOS, Windows, and Linux/X11 also need the input dependencies:

uv pip install -e ".[gui]"

Linux: one-time permission setup

Injecting input on Wayland means writing to /dev/uinput, which is root-owned by default:

bash scripts/setup-linux.sh   # needs sudo once
# then log out and back in

macOS: one-time permission setup

Grant Screen Recording and Accessibility to whichever app launches the server (Terminal, iTerm, Claude Desktop) in System Settings → Privacy & Security.

Wire it up

Claude Code:

claude mcp add desktop -- /absolute/path/to/desktop-mcp/.venv/bin/python -m desktop_mcp.server

Claude Desktop (claude_desktop_config.json) and Cursor (.cursor/mcp.json) use the same shape:

{
  "mcpServers": {
    "desktop": {
      "command": "/absolute/path/to/desktop-mcp/.venv/bin/python",
      "args": ["-m", "desktop_mcp.server"]
    }
  }
}

Tools

Tool	Purpose
screenshot	Capture the desktop
click	Left/right/middle click, single or double
move	Move the pointer without clicking (hover)
drag	Press, move, release — selections, sliders, window moves
scroll	Wheel scroll under a point
type_text	Type into the focused element
key	Chords like ctrl+s, alt+tab, cmd+space
cursor_position	Where the pointer is
list_windows	Open windows, where the OS permits it
wait	Pause, then look again
desktop_info	Environment, active backends, safety settings

Coordinates

Screenshots are downscaled (1400px long edge by default) before being sent, so the model works in image pixels; the server maps them back to screen pixels on every call. Pass exactly the coordinates you read off the screenshot.

This is the detail most home-grown computer-use tools get wrong. Miss it and every click lands at a consistent fraction of where it was aimed.

Why /dev/uinput instead of ydotool on Linux

Commonly packaged ydotool builds expose a relative pointer, so a jump to (1280, 720) is delivered as one large relative motion — which the compositor runs through pointer acceleration. The cursor lands near the target rather than on it, and the error grows with distance. An absolute uinput device sidesteps acceleration: measured 5/5 exact on a 2560x1440 screen. ydotool remains a fallback when /dev/uinput is not writable.

Click timing

Compositors dispatch pointer motion asynchronously, and toolkits ignore a button press that arrives before they have processed the motion that put the cursor over the widget. Measured on COSMIC: a 30ms gap between move and press reliably highlights a button but never activates it — the hover state is proof the coordinates are right, which makes this failure easy to misread as a targeting bug. 250ms activates it. A press also has to be held (~90ms) to register as a real click rather than a stray event.

Both delays are tunable; lower them only if your desktop tolerates it.

Safety

This gives a model the same reach over the machine as the person sitting at it. It can read anything on screen — open password managers, private messages, tokens — and click anything, including destructive buttons. Treat a session as if you had handed someone your keyboard.

Environment variables:

Variable	Default	Effect
DESKTOP_MCP_READ_ONLY	off	Screenshots allowed, all input refused
DESKTOP_MCP_RATE_LIMIT	240	Max input actions per minute
DESKTOP_MCP_BLOCK_KEYS	—	Extra comma-separated chords to refuse
DESKTOP_MCP_AUDIT_LOG	—	Append every action to this file
DESKTOP_MCP_MAX_EDGE	1400	Screenshot long edge in pixels
DESKTOP_MCP_MOVE_SETTLE	0.25	Seconds between moving the pointer and pressing
DESKTOP_MCP_PRESS_HOLD	0.09	Seconds a mouse button is held down

ctrl+alt+F1–F12 are always blocked; VT switching can drop the graphical session out from under you.

A read-only reviewer setup:

DESKTOP_MCP_READ_ONLY=1 python -m desktop_mcp.server

Verify

.venv/bin/python scripts/mcp_smoke.py   # real stdio MCP round trip