macOS Computer Use MCP Server

146 MCP tools that give AI agents full control over macOS — screenshots, mouse, keyboard, window management, app automation, file system, OCR, and built-in app semantics (Calendar, Mail, Safari, Music, Messages, and more).

Built for Claude Code, LangGraph, Pi Agent, and any MCP-compatible agent framework.

---

中文文档

Table of Contents

• Why
• Quick Start
• macOS Permissions (TCC)
• Usage
  • Claude Code
  • Other MCP Clients
  • MCP Inspector
• AI Model Configuration
  • Option 1: Vision-capable model
  • Option 2: Text-only model + OCR
  • Framework Integration
• Tool Reference
  • L1 — OS Primitives (24 tools)
  • L2 — Deterministic Tools (54 tools)
  • L3 — App Semantics (68 tools)
• Architecture
• Examples
• Requirements
• Development
• Troubleshooting
• Contributing
• License

---

Why

Most computer-use agents depend on third-party binaries (Playwright, Puppeteer) or cloud services. macos-computer-use-mcp runs directly on the OS:

Benefit	Description
Auditable	Every tool call is plain Python + AppleScript — no black boxes
Extensible	Add your own tools by following the two-file pattern
Framework-agnostic	Works with any MCP client: Claude Code, VS Code, LangGraph, custom agents
macOS-native	Uses Quartz, Accessibility, IOKit, Vision — no external dependencies beyond pyobjc
High coverage	146 tools across 3 layers, from raw pixels to semantic app control

---

Quick Start

# One-command global install (recommended)
uv tool install macos-computer-use-mcp

# Or run without installing (like npx)
uvx macos-computer-use-mcp

That's it. The macos-computer-use-mcp command is now globally available.

Alternative: pipx install macos-computer-use-mcp also works.

Verify

# Check permissions
python -c "from computer_use_mcp.darwin.tcc import check_all; print(check_all().report())"

# List all 146 tools
python -c "
import asyncio
from computer_use_mcp.server import mcp
async def main():
    tools = await mcp.list_tools()
    print(f'{len(tools)} tools registered')
asyncio.run(main())
"

---

macOS Permissions (TCC)

Open System Settings → Privacy & Security and grant your terminal (or IDE):

Permission	Required by	Why
Screen Recording	screenshot, region_screenshot, cursor_screenshot, display_list, screen_size	Capture pixel data from display(s)
Accessibility	mouse_*, keyboard_*, window_*, ax_*, app_*	Control mouse, keyboard, and inspect UI elements
Automation	calendar_*, reminders_*, notes_*, mail_*, messages_*, contacts_*	AppleScript control of built-in apps
Full Disk Access	file_*, clipboard_*	Read/write files in protected directories

The server prints a clear status report on startup. Missing permissions do not prevent the server from running — affected tools simply return errors.

---

Usage

Claude Code

Add to your .mcp.json (project root) or Claude Code settings:

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "macos-computer-use-mcp"
    }
  }
}

Or run directly without installing (uvx auto-downloads from PyPI):

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "uvx",
      "args": ["macos-computer-use-mcp"]
    }
  }
}

Then ask Claude: "Take a screenshot, find the Safari window, and search for GitHub."

Other MCP Clients

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "macos-computer-use-mcp"
    }
  }
}

MCP Inspector

npx @anthropic-ai/mcp-inspector python -m computer_use_mcp

Opens a web UI at http://localhost:5173 where you can browse and call every tool interactively.

---

AI Model Configuration

This MCP server provides 146 macOS control tools. It does not include an AI model — you need an MCP client to drive the tools.

If you use Claude Code — you're done. Claude Code has built-in vision + tool-calling. No API keys to configure. Just add .mcp.json and start talking.

If you're building your own agent — you need to configure an AI model. Two approaches:

---

Option 1: Vision-capable model (recommended)

One model handles both seeing the screen and deciding actions:

# Pick one provider, set the key in ~/.zshrc:
export OPENAI_API_KEY="sk-..."        # gpt-4o — https://platform.openai.com/api-keys
export ANTHROPIC_API_KEY="sk-ant-..." # claude-sonnet-5 — https://console.anthropic.com
export ZHIPU_API_KEY="..."            # glm-4v — https://open.bigmodel.cn

# agent.py — minimal agent loop
import asyncio, base64, json, os
from openai import OpenAI
from mcp.client import ClientSession
from mcp.client.stdio import stdio_client, StdioServerParameters

async def main():
    goal = input("🎯 What should I do? ")
    server = StdioServerParameters(command="macos-computer-use-mcp")
    client = OpenAI()  # reads OPENAI_API_KEY from env

    async with stdio_client(server) as (read, write):
        async with ClientSession(read, write) as session:
            messages = [{"role": "system", "content": "You control a Mac desktop. Reply with JSON: {\"tool\": \"...\", \"args\": {...}} or {\"done\": true}."}]

            for step in range(15):
                # Screenshot → model decides → execute
                result = await session.call_tool("screenshot", {})
                img = result.content[0].data

                response = client.chat.completions.create(
                    model=os.getenv("MODEL", "gpt-4o"),
                    messages=messages + [{"role": "user", "content": [
                        {"type": "text", "text": f"Step {step+1}. Goal: {goal}"},
                        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img}"}}
                    ]}],
                )
                action = json.loads(response.choices[0].message.content.strip().removeprefix("```json").removesuffix("```"))
                if action.get("done"):
                    break
                await session.call_tool(action["tool"], action.get("args", {}))
                messages.append({"role": "assistant", "content": json.dumps(action)})

    print("✅ Done!")

asyncio.run(main())

Option 2: Text-only model + OCR

Models like DeepSeek can't see images. Use OCR to describe the screen first:

# screenshot → OCR → text model
ocr = await session.call_tool("ocr_screenshot", {})
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You control a Mac desktop. Reply with JSON tool calls."},
        {"role": "user", "content": f"Screen contents:\n{ocr}\n\nGoal: {goal}"}
    ]
)

# Required env vars:
export DEEPSEEK_API_KEY="sk-..."  # https://platform.deepseek.com/api_keys

Framework Integration

LangGraph:

from langchain_mcp import MCPToolkit
toolkit = MCPToolkit(command="macos-computer-use-mcp")
tools = await toolkit.get_tools()

Pi Agent: See Pi Agent MCP docs.

Claude Code (zero-config):

// .mcp.json at project root
{
  "mcpServers": {
    "macos-computer-use": {
      "command": "macos-computer-use-mcp"
    }
  }
}

---

Tool Reference

Layer Architecture

┌────────────────────────────────────────────────────────────┐
│ L3: App Semantics (68 tools)                                │
│ Calendar · Reminders · Notes · Mail · Messages · Contacts   │
│ Finder · Safari · Music · Shortcuts · Settings              │
├────────────────────────────────────────────────────────────┤
│ L2: Deterministic Tools (54 tools)                          │
│ Window Mgmt · App Lifecycle · AX Tree · OCR · Clipboard     │
│ File System · System Info · Battery · WiFi · Bluetooth      │
├────────────────────────────────────────────────────────────┤
│ L1: OS Primitives (24 tools)                                │
│ Screenshot · Mouse · Keyboard · Cursor · Input Source       │
│ Timing · Display                                            │
└────────────────────────────────────────────────────────────┘

---

L1 — OS Primitives

screenshot

Take a full-screen or region screenshot.

Tool	Description
screenshot	Capture entire primary display (PNG base64)
region_screenshot	Capture rectangle (x, y, w, h)
cursor_screenshot	Capture a small region around the cursor
screen_size	Get display dimensions in pixels
display_list	Enumerate all connected displays

mouse

Absolute-mouse positioning and buttons.

Tool	Description
mouse_move	Move to absolute (x, y)
mouse_click	Left-click at current position
double_click	Double-click at current position
right_click	Right-click at current position
mouse_drag	Drag from current to (x, y)
scroll	Vertical scroll (positive = up)
horizontal_scroll	Horizontal scroll

cursor

Cursor position and pixel inspection.

Tool	Description
cursor_get_position	Get current (x, y)
cursor_screenshot	Screenshot the ~100×100 px area around cursor
get_pixel_color	Get RGB color at (x, y)

keyboard

Text input and modifier keys.

Tool	Description
keyboard_type	Type a string (supports Unicode)
hotkey	Press a key combination ("cmd+c", "cmd+shift+4")
key_press	Press and hold a single key
key_release	Release a single key

input_source

Keyboard layout switching.

Tool	Description
input_source_get	Get current input source
input_source_list	List all available input sources
input_source_set	Switch to a specific input source
get_modifier_keys	Get current modifier key states

timing

Tool	Description
sleep	Pause execution for N seconds
timestamp	Get current Unix timestamp (seconds or ms)

---

L2 — Deterministic Tools

window

Window enumeration and manipulation via CGWindowList (Quartz).

Tool	Description
window_list	List all visible windows with position/size/owner
window_activate	Bring a window to the foreground
window_move	Move a window to absolute (x, y)
window_resize	Resize to (width, height)
window_close	Close a window
window_minimize	Minimize a window
get_frontmost_app	Get the frontmost application name/bundle
get_focused_element	Get the currently focused UI element

app

Application lifecycle via NSWorkspace + AppleScript.

Tool	Description
app_list_running	List all running GUI applications
app_launch	Launch an application by name or bundle ID
app_quit	Gracefully quit an application
app_force_quit	Force-quit an application
app_hide	Hide an application (Cmd+H equivalent)

ax_tree

Accessibility tree inspection and manipulation via AXUIElement (Quartz).

Tool	Description
ax_get_tree	Get the AX tree for a window or the whole screen
ax_get_element	Get detailed attributes of a specific element
ax_get_actions	List available actions on an element
ax_click_element	Click an element via accessibility
ax_set_value	Set the value of a text field or slider
ax_perform_action	Perform a named action (e.g. "press", "confirm")

ocr

Text recognition via macOS Vision framework.

Tool	Description
ocr_screenshot	Take a screenshot and OCR the entire display
ocr_region	OCR a specific rectangle
ocr_find_text	Search for text on screen, return bounding boxes
ocr_get_text_at	Get the text at a specific pixel position

clipboard

Clipboard read/write via pbcopy/pbpaste + NSImage (AppKit).

Tool	Description
clipboard_get	Get clipboard content (text, image, or file list)
clipboard_set_text	Set clipboard to plain text
clipboard_set_image	Set clipboard to image from file
clipboard_clear	Clear all clipboard contents

file

File-system operations via Python pathlib/shutil.

Tool	Description
file_list_dir	List directory contents
file_exists	Check if a path exists
file_read	Read file content (text or binary base64)
file_write	Write content to a file
file_delete	Delete a file or directory (recursive)
file_move	Move/rename a file or directory
file_copy	Copy a file or directory
file_mkdir	Create a directory (with parents=True)
file_get_info	Get file metadata (size, mtime, permissions)
file_search	Recursive file search with glob patterns
file_get_home_dir	Get the user home directory path
file_get_desktop_dir	Get the Desktop directory path
file_get_downloads_dir	Get the Downloads directory path

system

System information and control via IOKit, system_profiler, pmset, networksetup.

Tool	Description
system_info	Hostname, OS version, CPU, memory, disk
battery_info	Battery percent, charging, health, cycle count
get_volume	Get system output volume (0–100)
set_volume	Set system output volume
get_brightness	Get built-in display brightness (0.0–1.0)
set_brightness	Set built-in display brightness
get_dark_mode	Check if dark mode is active
wifi_info	SSID, BSSID, channel, RSSI, IP address
bluetooth_info	Power state and connected devices
sleep_display	Put all displays to sleep
lock_screen	Lock the screen (password required to unlock)
open_url	Open a URL in the default browser or specified app
reveal_in_finder	Reveal a file/folder in Finder
run_command	Execute a shell command (local trusted sessions)

---

L3 — App Semantics

All L3 tools use AppleScript targeting macOS built-in applications. Apps that are not running will be launched automatically by AppleScript.

calendar (Calendar.app)

Tool	Description
calendar_list	List upcoming events (default 7 days)
calendar_create	Create a new event with title, date, location, notes
calendar_delete	Delete an event by UID

reminders (Reminders.app)

Tool	Description
reminders_list	List reminders (by list, with filters)
reminders_create	Create a reminder with title, due date, priority
reminders_complete	Mark a reminder as completed
reminders_delete	Delete a reminder

notes (Notes.app)

Tool	Description
notes_list	List notes across all folders (with search)
notes_create	Create a note with title and body
notes_get	Get full note content by ID or name

mail (Mail.app)

Tool	Description
mail_list	List recent emails with optional filters
mail_send	Compose and send an email (to, cc, bcc)

messages (Messages.app)

Tool	Description
messages_list_conversations	List recent conversations with unread counts
messages_get	Get messages from a conversation (by chat_id or contact)
messages_send	Send an iMessage/SMS (text and/or attachment)
messages_search	Search all conversations by text
messages_mark_read	Mark a conversation as read
messages_delete_conversation	Delete an entire conversation
messages_get_attachment	Save attachments from a conversation to disk

contacts (Contacts.app)

Tool	Description
contacts_list	List contacts (by group, up to 500)
contacts_search	Search contacts by name, email, phone, org
contacts_get	Get full details of a specific contact
contacts_create	Create a new contact (name, org, email, phone)
contacts_update	Update an existing contact
contacts_delete	Delete a contact
contacts_export_vcard	Export contacts as .vcf file

finder (Finder.app)

Tool	Description
finder_get_selection	Get currently selected items
finder_select	Select files/folders by path
finder_get_windows	List all open Finder windows with target paths
finder_get_current_folder	Get the frontmost Finder window's folder
finder_navigate	Open a folder in Finder
finder_get_info	Get detailed Finder metadata for a file
finder_duplicate	Duplicate a file/folder (Cmd+D)
finder_make_alias	Create a Finder alias
finder_eject_volume	Eject a mounted disk by name
finder_empty_trash	Empty the Trash (irreversible)
finder_list_disks	List all mounted volumes with capacity/free space

safari (Safari.app)

Tool	Description
safari_list_tabs	List all open tabs across all windows
safari_get_current_tab	Get the active tab's URL and title
safari_open_url	Open a URL (new tab or window)
safari_close_tab	Close a specific tab
safari_search	Search the web using the default search engine
safari_go_back	Navigate back
safari_go_forward	Navigate forward
safari_get_bookmarks	List all bookmarks
safari_add_bookmark	Add a bookmark
safari_execute_javascript	Execute JavaScript in the current tab

music (Music.app)

Tool	Description
music_get_state	Get player state + current track info
music_play	Start playback
music_pause	Pause playback
music_playpause	Toggle play/pause
music_next	Skip to next track
music_previous	Go to previous track
music_search	Search library by name/artist/album
music_get_playlists	List all playlists with track counts
music_play_playlist	Play a specific playlist by name
music_set_volume	Set Music.app volume (0–100)

shortcuts (Shortcuts.app)

Tool	Description
shortcuts_list	List all shortcuts (with folders and colors)
shortcuts_run	Run a shortcut by name (optional text input)
shortcuts_run_with_input	Run a shortcut with file or text input
shortcuts_get_info	Get shortcut metadata (action count, subtitle, icon)
shortcuts_list_folders	List shortcut folders with item counts

settings (System Settings)

Tool	Description
settings_open_pane	Open a specific Settings pane (WiFi, Bluetooth, etc.)
settings_get_wallpaper	Get current desktop wallpaper path(s)
settings_set_wallpaper	Set desktop wallpaper from an image file
settings_get_display	Get display resolution, refresh rate, scaling
settings_get_sound	Get audio input/output device and volume
settings_get_general	Get appearance, accent color, sidebar size, Handoff

---

Architecture

src/computer_use_mcp/
├── server.py              # MCP entry point (stdio transport)
├── __init__.py            # Version, package metadata
│
├── darwin/                # macOS-specific implementations (no MCP dependency)
│   ├── cg_screen.py       #   CGDisplay / CGImage screenshot capture
│   ├── cg_input.py        #   CGEvent mouse + keyboard injection
│   ├── cg_keyboard.py     #   Text synthesis + key-code mapping
│   ├── ax_window.py       #   CGWindowList + AXUIElement window ops
│   ├── ax_tree.py         #   Accessibility tree walker (200+ lines)
│   ├── clipboard.py       #   pbcopy/pbpaste + NSImage clipboard
│   ├── ocr.py             #   VNRecognizeTextRequest (Vision framework)
│   ├── file_ops.py        #   Pure-Python pathlib/shutil file operations
│   ├── tcc.py             #   TCC permission checker (tccutil + osascript)
│   ├── system.py          #   IOKit brightness, pmset, networksetup, etc.
│   ├── calendar.py        #   Calendar.app AppleScript
│   ├── reminders.py       #   Reminders.app AppleScript
│   ├── notes.py           #   Notes.app AppleScript
│   ├── mail.py            #   Mail.app AppleScript
│   ├── messages.py        #   Messages.app AppleScript
│   ├── contacts.py        #   Contacts.app AppleScript
│   ├── finder.py          #   Finder.app AppleScript
│   ├── safari.py          #   Safari.app AppleScript
│   ├── music.py           #   Music.app AppleScript
│   ├── shortcuts.py       #   Shortcuts CLI + AppleScript
│   └── settings.py        #   System Settings + defaults CLI
│
├── tools/                 # MCP tool registration layer (thin wrappers)
│   ├── screen.py          #   @mcp.tool() async def screenshot()
│   ├── mouse.py           #   ... 21 more modules
│   ├── cursor.py          #   (each module has a register(mcp) entry point)
│   ├── keyboard.py
│   ├── input_source.py
│   ├── timing.py
│   ├── window.py
│   ├── app.py
│   ├── ax_tree.py
│   ├── ocr.py
│   ├── clipboard.py
│   ├── file.py
│   ├── system.py
│   ├── calendar.py
│   ├── reminders.py
│   ├── notes.py
│   ├── mail.py
│   ├── messages.py
│   ├── contacts.py
│   ├── finder.py
│   ├── safari.py
│   ├── music.py
│   ├── shortcuts.py
│   └── settings.py
│
└── tests/                 # One test file per domain (30+ files)
    ├── test_server.py     #   Verifies all 146 tools are registered
    ├── test_screen.py
    ├── test_mouse.py
    └── ... (28 more)

Design principles:

1. Two-layer separation: darwin/ modules contain pure macOS logic with zero MCP dependency. tools/ modules are thin MCP wrappers. This means you can reuse the darwin/ modules in a non-MCP agent.

2. Each tool returns a plain dict — MCP serializes them natively. No Pydantic models, no custom types.

3. AppleScript continuation: Long lines use ¬ (option-return) to stay under the 100-character line limit.

4. Applescript string escaping: Backslashes → \\\\, double-quotes → \\" before interpolation into AppleScript strings.

---

Examples

Screenshot + OCR → Click

# In your agent's tool-calling loop:
screenshot = await client.call_tool("screenshot")
# Feed to vision model...

text = await client.call_tool("ocr_find_text", {"text": "Submit"})
if text["found"]:
    x, y = text["bounds"]["x"] + text["bounds"]["w"] // 2
    text["bounds"]["y"] + text["bounds"]["h"] // 2
    await client.call_tool("mouse_move", {"x": x, "y": y})
    await client.call_tool("mouse_click", {})

Safari automation

await client.call_tool("safari_open_url", {"url": "https://github.com"})
await client.call_tool("safari_search", {"query": "macOS automation"})
tabs = await client.call_tool("safari_list_tabs")
# → {"tabs": [{"title": "...", "url": "...", ...}], "count": 5}

Full agent loop (pseudocode)

from mcp.client import ClientSession

async with ClientSession(stdio_transport) as session:
    while True:
        # 1. See the screen
        screen = await session.call_tool("screenshot")

        # 2. Vision model decides the next action
        action = vision_model.decide(screen, goal)

        # 3. Execute with MCP tools
        result = await session.call_tool(action.tool, action.params)

        # 4. Verify
        if action.done:
            break

---

Requirements

Requirement	Minimum	Recommended
macOS	13 Ventura	14 Sonoma+
Python	3.12	3.12+
RAM	2 GB	4 GB+
Disk	~50 MB	—

macOS Compatibility: Core L1/L2 tools work from macOS 10.13+ (High Sierra). L3 app-semantics tools require 13+ for full System Settings support. OCR requires 10.13+ (Vision framework). See the full compatibility table.

---

Development

# Clone for development
git clone https://github.com/yyyyyyyyiiiii/macos-computer-use-mcp.git
cd macos-computer-use-mcp
uv sync --all-extras

# Lint
uv run ruff check src tests

# Run all tests (requires macOS + permissions)
uv run pytest -q

# Run a subset
uv run pytest tests/test_server.py tests/test_safari.py -v

# Start the server locally (dev mode)
uv run python -m computer_use_mcp

Test conventions

• Tests are macOS-only: pytestmark = pytest.mark.skipif(sys.platform != "darwin", reason="...")
• L1 tests (screenshot, mouse, keyboard) require Screen Recording + Accessibility permissions
• L3 tests (Calendar, Reminders, etc.) require Automation permissions
• Input validation tests (empty strings, etc.) pass without any permissions

Adding a new tool

1. Implement the macOS logic in src/computer_use_mcp/darwin/<module>.py
2. Register the MCP wrapper in src/computer_use_mcp/tools/<module>.py
3. Add the import to server.py and the module to _MODULES
4. Write a test in tests/test_<module>.py
5. Run uv run ruff check src tests && uv run pytest -q

---

Troubleshooting

Safari: open_url AppleScript fails

Symptom: safari_open_url returns error -10024 ("can't create or move element into container").

Cause: Safari AppleScript permissions or window state (Stage Manager, minimized windows).

Workarounds:

1. Use open_url (L2 tool) instead — it uses the open CLI command, which is more reliable
2. Use keyboard_type + hotkey(["cmd", "l"]) to type URLs directly in Safari's address bar
3. Close and reopen Safari, then retry

Safari: execute_javascript fails

Symptom: safari_execute_javascript returns an error about "Allow JavaScript from Apple Events".

Fix: Open Safari → Develop menu → Settings → Advanced → check "Allow JavaScript from Apple Events".

If you don't see the Develop menu: Safari → Settings → Advanced → check "Show Develop menu in menu bar".

Window resize/move returns success: false

Cause: Some windows (especially Safari in Stage Manager) reject programmatic resize/move.

Workarounds:

1. Disable Stage Manager temporarily
2. Use hotkey(["cmd", "shift", "f"]) to toggle fullscreen
3. Use accessibility (ax_*) tools as an alternative click path

Screenshots too large for model context

Symptom: Screenshot data URLs exceed model token limits.

Solutions:

1. Use region_screenshot to capture only the relevant area
2. Use cursor_screenshot for a 60×60px region around the cursor
3. Use ocr_screenshot to get text-only screen descriptions (much smaller than images)
4. Resize screenshots before sending: PIL.Image.open(...).resize((1280, 800))

Model doesn't understand what's on screen

Solution: Use the built-in OCR tools before sending to the model:

# Get screen text as structured data
ocr = await session.call_tool("ocr_screenshot", {})
# Append OCR results to your model prompt for better grounding
prompt = f"Screen text visible:\n{ocr['text']}\n\nGoal: {goal}"

MCP server not found

Symptom: Claude Code shows "No MCP servers configured" or tools are unavailable.

Checklist:

1. .mcp.json must be at the project root (not in a subfolder or src/)
2. Run uv sync first to install dependencies
3. Verify the server starts: uv run python -m computer_use_mcp
4. Restart Claude Code after creating .mcp.json

---

Contributing

Contributions are welcome! See CONTRIBUTING.md for the full guide.

• Tool requests: Open an issue with the app name and desired operations
• Bug reports: Include macOS version + error output
• Pull requests: Follow the two-layer pattern, include tests

---

License

MIT — see LICENSE for full text.

---

<p align="center"> _{Built with ❤️ for the macOS agent ecosystem} </p>