mcp-scrcpy-vision

An MCP server that gives AI agents complete vision and control over Android devices.

Features:

• Real-time Vision: Continuous screen streaming via scrcpy H.264 + ffmpeg
• Fast Input Control: When streaming, input uses scrcpy control protocol (~5-10ms latency vs ~100-300ms with adb shell)
• UI Automation: Element detection via uiautomator with tap coordinates
• Full Input Control: Tap, swipe, long press, pinch, drag-drop, text, keycodes
• System Access: Shell commands, file transfer, clipboard, notifications
• Multi-device: Control multiple Android devices simultaneously
• WiFi ADB: Connect wirelessly for untethered automation

---

Quick Start

1. Prerequisites

Required:

• Node.js 18+
• ADB (Android Platform Tools) in PATH
• Android device with USB debugging enabled

For streaming (recommended for fast input):

• scrcpy - download release, extract scrcpy-server file
• ffmpeg - install and add to PATH

2. Install

git clone https://github.com/anthropics/mcp-scrcpy-vision.git
cd mcp-scrcpy-vision
npm install
npm run build

3. Configure

Create .env file:

# Required for streaming + fast input
SCRCPY_SERVER_PATH="C:\scrcpy-win64-v3.2\scrcpy-server"
SCRCPY_SERVER_VERSION="3.2"

# Optional (defaults shown)
ADB_PATH="adb"
FFMPEG_PATH="ffmpeg"
DEFAULT_MAX_SIZE="1024"
DEFAULT_MAX_FPS="30"
DEFAULT_FRAME_FPS="2"

4. Add to MCP Client

Claude Desktop (%APPDATA%\Claude\claude_desktop_config.json on Windows):

{
  "mcpServers": {
    "android": {
      "command": "node",
      "args": ["C:/path/to/mcp-scrcpy-vision/dist/index.js"],
      "env": {
        "SCRCPY_SERVER_PATH": "C:/scrcpy/scrcpy-server",
        "SCRCPY_SERVER_VERSION": "3.2"
      }
    }
  }
}

Cursor (Settings > MCP):

{
  "android": {
    "command": "node",
    "args": ["C:/path/to/mcp-scrcpy-vision/dist/index.js"],
    "env": {
      "SCRCPY_SERVER_PATH": "C:/scrcpy/scrcpy-server",
      "SCRCPY_SERVER_VERSION": "3.2"
    }
  }
}

5. Connect Device

1. Enable USB debugging on Android device (Settings > Developer Options > USB Debugging)
2. Connect via USB
3. Accept RSA fingerprint prompt on device
4. Verify: adb devices should show your device

---

How It Works

Two Modes of Operation

1. Snapshot Mode (No streaming required)

• Uses android.vision.snapshot for screenshots
• Input uses ADB shell commands (~100-300ms per action)
• Works without scrcpy/ffmpeg
• Best for simple automation or when streaming isn't available

2. Streaming Mode (Recommended)

• Start with android.vision.startStream
• Continuous JPEG frames available via resource URI
• Input uses scrcpy control protocol (~5-10ms per action)
• 10-20x faster than snapshot mode
• Best for real-time control and rapid interactions

Performance Comparison

Operation	Snapshot Mode	Streaming Mode
Tap	~100-300ms	~5-10ms
Swipe	~300-500ms	~50-100ms
Type text	~50ms/char	~5ms total
Screenshot	~500ms	~33ms (30fps)

---

Tools Reference (32 tools)

Device Management

Tool	Parameters	Description
android.devices.list	-	List connected devices
android.devices.info	serial	Get device info (model, SDK, etc.)
android.adb.enableTcpip	serial, port?	Enable WiFi debugging
android.adb.getDeviceIp	serial	Get device WiFi IP
android.adb.connectWifi	ipAddress, port?	Connect via WiFi
android.adb.disconnectWifi	ipAddress?	Disconnect WiFi

Vision

Tool	Parameters	Description
android.vision.startStream	serial, maxSize?, maxFps?, frameFps?	Start continuous stream (enables fast input)
android.vision.stopStream	serial	Stop stream
android.vision.snapshot	serial	Take PNG screenshot (works without streaming)
android.ui.dump	serial	Get UI hierarchy XML
android.ui.findElement	serial, text?, resourceId?, className?, contentDesc?	Find elements with tap coords

Input Control

Note: These automatically use fast scrcpy control when streaming, otherwise fall back to ADB.

Tool	Parameters	Description
android.input.tap	serial, x, y	Tap at coordinates
android.input.swipe	serial, x1, y1, x2, y2, durationMs?	Swipe gesture
android.input.longPress	serial, x, y, durationMs?	Long press
android.input.pinch	serial, centerX, centerY, startDistance, endDistance, durationMs?	Pinch zoom
android.input.dragDrop	serial, startX, startY, endX, endY, durationMs?	Drag and drop
android.input.text	serial, text	Type text
android.input.keyevent	serial, keycode	Send keycode

App Control

Tool	Parameters	Description
android.app.start	serial, packageName, activity?	Launch app
android.app.stop	serial, packageName	Force-stop app
android.apps.list	serial, system?	List installed apps
android.activity.current	serial	Get foreground activity

System

Tool	Parameters	Description
android.shell.exec	serial, command	Execute shell command
android.file.push	serial, localPath, remotePath	Push file to device
android.file.pull	serial, remotePath, localPath	Pull file from device
android.file.list	serial, path	List directory
android.clipboard.get	serial	Get clipboard
android.clipboard.set	serial, text	Set clipboard
android.notifications.get	serial	Get notifications

Screen Control

Tool	Parameters	Description
android.screen.wake	serial	Wake screen
android.screen.sleep	serial	Sleep screen
android.screen.isOn	serial	Check if screen is on
android.screen.unlock	serial	Unlock (unsecured only)

---

Resources

The server exposes these MCP resources:

• android://devices - JSON list of connected devices
• android://device/<serial>/frame/latest.jpg - Latest JPEG frame (when streaming)

---

Usage Examples

Basic Automation Loop (Streaming Mode)

1. Start stream: android.vision.startStream { serial: "ABC123" }
2. Read resource: android://device/ABC123/frame/latest.jpg
3. AI analyzes image, decides to tap "Login" button
4. Find element: android.ui.findElement { serial: "ABC123", text: "Login" }
5. Tap at returned coordinates: android.input.tap { serial: "ABC123", x: 540, y: 1200 }
6. Wait 500ms, read resource again, repeat
7. When done: android.vision.stopStream { serial: "ABC123" }

Simple Screenshot Mode

1. Take screenshot: android.vision.snapshot { serial: "ABC123" }
2. AI analyzes image
3. Find and tap: android.ui.findElement + android.input.tap
4. Take another screenshot to verify

WiFi Connection Workflow

1. Connect device via USB
2. android.adb.enableTcpip { serial: "ABC123" }
3. android.adb.getDeviceIp { serial: "ABC123" } → "192.168.1.50"
4. Disconnect USB cable
5. android.adb.connectWifi { ipAddress: "192.168.1.50" }
6. Now use "192.168.1.50:5555" as serial for all commands

App Testing Example

1. android.app.start { serial: "ABC123", packageName: "com.example.app" }
2. android.vision.startStream { serial: "ABC123" }
3. Wait for app to load, read frame
4. android.ui.findElement { serial: "ABC123", resourceId: "username_field" }
5. android.input.tap { serial: "ABC123", x: 540, y: 300 }
6. android.input.text { serial: "ABC123", text: "testuser@example.com" }
7. android.input.keyevent { serial: "ABC123", keycode: 66 }  // Enter
8. Read frame, verify login succeeded
9. android.vision.stopStream { serial: "ABC123" }

---

Common Keycodes

Key	Code	Key	Code
HOME	3	BACK	4
VOLUME_UP	24	VOLUME_DOWN	25
POWER	26	ENTER	66
DELETE	67	TAB	61
MENU	82	APP_SWITCH	187
WAKEUP	224	SLEEP	223

---

Troubleshooting

No devices found

adb kill-server
adb start-server
adb devices

Ensure USB debugging is enabled and RSA fingerprint accepted.

Scrcpy version mismatch

SCRCPY_SERVER_VERSION must exactly match your scrcpy-server file. Check the scrcpy release version you downloaded.

ffmpeg not found

• Windows: Download from https://ffmpeg.org/download.html, extract, add bin folder to PATH
• macOS: brew install ffmpeg
• Linux: apt install ffmpeg or yum install ffmpeg

Or set FFMPEG_PATH in .env to the full path.

uiautomator dump fails

Some devices need screen on. Try android.screen.wake first.

Clipboard not working (Android 10+)

Android 10+ restricts clipboard access. Use UI automation to paste instead.

Stream won't start

1. Check scrcpy-server path is correct
2. Verify version numbers match
3. Try running scrcpy standalone first to verify it works

---

Notes & Limitations

• Fast input when streaming: When a stream is active, tap/swipe/text/keyevent use the scrcpy control protocol (~5-10ms latency). Without streaming, falls back to adb shell input (~100-300ms).
• One stream per device at a time
• Snapshot works without scrcpy - useful fallback when streaming is not needed
• Clipboard has platform limitations on Android 10+
• Notifications may require permissions on newer Android
• Pinch gesture currently simulates single-finger; true multi-touch requires the streaming session

---

Security Warning

This MCP server provides full control over connected Android devices:

• Execute arbitrary shell commands
• Read/write files on device
• Control UI and input
• Access clipboard and notifications

Only connect devices you own and trust the AI agent.

---

Development

npm run dev     # Development with tsx
npm run build   # Compile TypeScript
npm start       # Run production build

See claude.md for developer documentation. See agents.md for AI agent integration guide.

---

License

MIT