Ubuntu Desktop Control MCP Server

An MCP (Model Context Protocol) server that enables LLMs to control your Ubuntu desktop by taking screenshots and sending mouse clicks. This allows AI assistants to visually interact with your desktop applications.

⚡ NEW: Optimized Production Workflow

5x faster, 5x more accurate! Now using the same optimization techniques as Anthropic's Computer Use API:

• 📸 Smart Screenshots: Auto-downsampled to 1280x720 (5x smaller)
• 🎯 Numbered Elements: See what's clickable at a glance with overlaid IDs
• 🤖 AT-SPI Integration: Automatic UI element detection using accessibility API
• 📐 Percentage Coords: Resolution-agnostic positioning (no more pixel hunting!)
• ⚡ Workflow Batching: Execute multiple actions in one MCP call
• 🎪 Element Cache: Direct element interaction - "click element #5"

Example - Old way (8+ calls, ~15s):

take_screenshot() → analyze → grid overlay → zoom quadrant → find pixel → click → miss

Example - New way (1 call, ~3s):

take_screenshot() → "I see Pinta is element #5" → click_screen(element_id=5) → ✓

See README.md for full details.

Features

• 📸 Screenshot Capture: Annotated screenshots with automatic element detection
• 🔢 Element Detection: AT-SPI + CV fallback for robust UI element identification
• 🖱️ Smart Clicking: Click by element ID or percentage coordinates
• ⌨️ Keyboard Control: Type text and press keys/hotkeys
• 🎯 Mouse Movement: Smooth cursor positioning with animation
• 🚀 Workflow Batching: Execute multi-step tasks in single MCP call
• 📊 Diagnostics: Display scaling detection, warnings, and recommendations

Quick Start

1. Prerequisites

• Ubuntu Linux (X11 required, Wayland not fully supported)
• Python 3.9+

2. Installation

From PyPI (Recommended)

pip install ubuntu-desktop-control

From Source

# Clone repository
git clone https://github.com/charettep/ubuntu-desktop-control-mcp.git
cd ubuntu-desktop-control-mcp

# Install system dependencies (requires sudo)
chmod +x scripts/install.sh
./scripts/install.sh

# Install Python dependencies
pip install -e .

Configuration

Claude Code

<details> <summary>Installation Methods</summary>

Method 1: CLI (Recommended)

claude mcp add --transport stdio ubuntu-desktop-control -- \
  ubuntu-desktop-control

Method 2: Manual Config

Edit ~/.claude/claude_desktop_config.json:

{
  "mcpServers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": []
    }
  }
}

</details>

VS Code Insiders

<details> <summary>Installation Methods</summary>

Method 1: MCP Command

1. Open Command Palette (Ctrl+Shift+P)
2. Run MCP: Open Workspace Folder Configuration
3. Add the server configuration below.

Method 2: Manual Config

Create .vscode/mcp.json in your workspace:

{
  "servers": {
    "ubuntu-desktop-control": {
      "type": "stdio",
      "command": "ubuntu-desktop-control",
      "args": []
    }
  }
}

</details>

Codex CLI

<details> <summary>Installation Methods</summary>

Method 1: CLI

codex mcp add ubuntu-desktop-control -- \
  ubuntu-desktop-control

Method 2: Manual Config

Edit ~/.config/codex/config.toml:

[mcp_servers.ubuntu-desktop-control]
type = "stdio"
command = "ubuntu-desktop-control"
args = []

</details>

Tools

Core Capabilities

Tool	Description
take_screenshot	Capture the desktop (optionally per-monitor) with annotated elements.
click_screen	Click by element ID or percentage coordinates (supports per-monitor).
move_mouse	Move the cursor by element ID or percentage coordinates (supports per-monitor).
drag_mouse	Drag the cursor to coordinates while holding a mouse button.
type_text	Type text using the keyboard.
press_key	Press a specific key (e.g., 'enter', 'esc').
press_hotkey	Press a combination of keys simultaneously (e.g., Ctrl+Shift+C).
get_screen_info	Get screen dimensions and display server type (X11/Wayland).
get_display_diagnostics	Troubleshoot scaling and coordinate mismatches.
map_GUI_elements_location	Detect and map UI elements (hitboxes) using Computer Vision.
convert_screenshot_coordinates	Convert pixels from a screenshot to logical click coordinates.
list_prompt_templates	List available prompt templates (for clients without native prompt support).
execute_workflow	Execute a batch of actions (screenshot/click/move/type/wait).

Prompt Rendering Tools

These tools allow clients without native prompt support (like Codex CLI) to render prompt templates as text.

Tool	Description
render_prompt_baseline_display_check	Render the baseline display check prompt.
render_prompt_capture_full_desktop	Render the full desktop capture prompt.
render_prompt_capture_region_for_task	Render the region capture prompt.
render_prompt_convert_screenshot_coordinates	Render the coordinate conversion prompt.
render_prompt_safe_click	Render the safe click prompt.
render_prompt_hover_and_capture	Render the hover and capture prompt.
render_prompt_coordinate_mismatch_recovery	Render the mismatch recovery prompt.
render_prompt_end_to_end_capture_and_act	Render the end-to-end workflow prompt.

Prompts

Prompt	Description
baseline_display_check	Check display settings and scaling before starting tasks.
capture_full_desktop	Capture and summarize the full desktop state.
capture_region_for_task	Capture a specific region for detailed inspection.
safe_click	Perform a click with safety checks and scaling awareness.
hover_and_capture	Hover to reveal UI elements, then capture.
coordinate_mismatch_recovery	Diagnose and fix missed clicks.
end_to_end_capture_and_act	Plan and execute a full interaction loop.

Configuration & Customization

Environment Variables

The server relies on standard Linux/X11 environment variables to locate and interact with the desktop session.

Variable	Description	Default
DISPLAY	X11 display identifier. Required for the server to know which screen to control.	:0
XDG_SESSION_TYPE	Used to detect if running on X11 or Wayland.	unknown
XAUTHORITY	Path to X11 authority file. Required if running from a different user context (e.g., sudo, docker) or over SSH.	~/.Xauthority
UDC_FORCE_COORDS	Force coordinate clicks (disable AT-SPI action clicks).	unset

Passing Environment Variables

You can customize these variables in your MCP client configuration.

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": [],
      "env": {
        "DISPLAY": ":0",
        "XAUTHORITY": "/home/user/.Xauthority"
      }
    }
  }
}

VS Code (.vscode/mcp.json)

{
  "servers": {
    "ubuntu-desktop-control": {
      "command": "ubuntu-desktop-control",
      "args": [],
      "env": {
        "DISPLAY": ":0"
      }
    }
  }
}

Display Scaling & Coordinates

If clicks land in the wrong place, you likely have a HiDPI display scaling mismatch (e.g., logical 1920x1080 vs physical 3840x2160).

Solutions:

1. Auto-scale: Use click_screen(..., auto_scale=True) to let the server handle it.
2. Diagnostics: Run get_display_diagnostics() to see the scaling factor.
3. Element IDs: Use take_screenshot(detect_elements=True) and click via element_id or percentage coordinates.

Troubleshooting

<details> <summary><strong>Common Issues</strong></summary>

• "Screenshot failed": Ensure gnome-screenshot or scrot is installed (sudo apt install gnome-screenshot).
• "PyAutoGUI not installed": Ensure you are using the .venv python.
• Wayland Issues: This server requires X11. Check with echo $XDG_SESSION_TYPE. If "wayland", switch to "GNOME on Xorg" at login.
• Permission Denied: Run xhost +local: if you have X11 permission issues.

</details>

Security

⚠️ Warning: This server gives LLMs full control over your mouse and visibility of your screen.

• Only use with trusted clients.
• Be aware screenshots may capture sensitive data.
• Automated clicks can be destructive.

License

MIT License