AIsistent

MCP server for non-intrusive RDP automation. OCR, YOLO button detection, and click injection — zero footprint on the remote machine.

	GUI mode (default)	RDP headless mode
Capture	macOS RDP window / MSS fullscreen	Direct RDP framebuffer (simple-rdp)
OCR	Apple Vision / EasyOCR	Apple Vision / EasyOCR
Detection	YOLO (CUDA / MPS / CPU)	YOLO (CUDA / MPS / CPU)
Click	pyautogui	RDP protocol input channel

---

Quick Start

# macOS (Apple Silicon)
pip install aistent[apple]

# Windows / Linux (CPU)
pip install aistent[cpu]

# Windows (NVIDIA CUDA)
pip install aistent[cuda]

# RDP headless (any OS)
pip install aistent[rdp]

# Everything
pip install aistent[all]

aisistent                          # GUI mode (default)
aisistent --mode rdp --host HOST   # RDP headless mode

---

Tools

Tool	Description
connect_rdp	Open a headless RDP connection (switches transport to RDP mode)
disconnect_rdp	Close the RDP connection and switch back to GUI mode
capture_rdp_screen	Capture screen via active transport (GUI or RDP)
run_apple_ocr	OCR: Apple Vision (Mac) or EasyOCR (CPU/CUDA)
detect_rdp_buttons	YOLOv8 button detection on CUDA / MPS / CPU
inject_rdp_click	Click injection at percentage-based coordinates
benchmark	Run performance benchmark on OCR + YOLO (returns JSON)

Configuration

Env var	Default	Description
AISISTENT_YOLO_WEIGHTS	models/weights/best.pt	Path to YOLO weights file
AISISTENT_TEMP_DIR	temp_captures/	Screenshot temp directory
RDP_HOST	—	RDP server hostname/IP (for headless mode)
RDP_USER	—	RDP username
RDP_PASS	—	RDP password
RDP_DOMAIN	—	RDP domain (optional)

---

Cross-Platform Hardware Detection

Hardware is auto-detected at import time in aisistent/platform.py:

Backend	Detection	dtype	Use Case
CUDA (NVIDIA)	torch.cuda.is_available()	float16	Windows/Linux with NVIDIA GPU
MPS (Apple)	torch.backends.mps.is_available()	float16	macOS Apple Silicon (M1–M4)
CPU	fallback	float32	Any OS, no GPU

Install GPU backends

# CUDA (NVIDIA)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# MPS (Apple) — included in default torch on macOS
pip install torch torchvision

---

Benchmark

Run a quick performance test from the command line:

aisistent-bench                                 # captures a real screenshot & benchmarks
aisistent-bench --synthetic                     # use synthetic image (no screen capture)
aisistent-bench --skip-ocr                      # YOLO only
aisistent-bench --image screenshot.png          # use your own image
aisistent-bench --device cpu                    # force CPU backend

Or via MCP tool call:

benchmark(image_base64: "")              # empty = real screenshot, or pass base64

Real-world performance (Apple MacBook M5 — MPS GPU)

Benchmark on a real 1920×1080 desktop screenshot with text, buttons, and UI elements:

Platform : macOS (Apple Silicon M5)
Device   : MPS
──────────────────────────────────────
Capture  :  0.22s
OCR      :  0.43s  —  102 texts detected
YOLO     :  0.76s  —   59 buttons detected
──────────────────────────────────────
Total    : ~1.4s

Step	Time	Throughput
Screen capture	~0.22s	—
Apple Vision OCR (Neural Engine)	~0.43s	~237 texts/sec
YOLOv8 inference (MPS float16)	~0.76s	~78 detections/sec
End-to-end	~1.4s	—

These numbers are from the same machine running both the MCP server and the benchmark — no overhead from network or RDP. On NVIDIA CUDA, YOLO inference is typically 0.3–0.5s (RTX 3060+).

---

MCP Client Integration

AIsistent implements the standard MCP (Model Context Protocol), so it works with any MCP client. Below are detailed setup instructions for each platform.

---

Hermes MCP

Hermes is an AI agent that uses MCP tools to interact with your computer.

1. Install AIsistent

# macOS (Apple Silicon — Apple Vision OCR + MPS GPU)
pip install aistent[apple]

# Windows/Linux CPU
pip install aistent[cpu]

# Windows with NVIDIA GPU
pip install aistent[cuda]

2. Locate Hermes config file

OS	Path
macOS	~/.config/hermes/config.json
Windows	%APPDATA%\hermes\config.json
Linux	~/.config/hermes/config.json

3. Add AIsistent to Hermes config

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

If AIsistent is not on your PATH, use the full path:

{
  "mcpServers": {
    "aisistent": {
      "command": "/path/to/venv/bin/aisistent",
      "type": "stdio"
    }
  }
}

4. Start Hermes

hermes

Hermes will auto-discover AIsistent's tools on startup. You should see:

👁️ AIsistent — capture_rdp_screen, run_apple_ocr, detect_rdp_buttons, inject_rdp_click, benchmark

Example: Hermes asks AIsistent to read the screen

> What's on my screen right now?

Hermes will:

1. Call capture_rdp_screen → gets screenshot
2. Call run_apple_ocr(image) → extracts all text
3. Call detect_rdp_buttons(image) → finds buttons
4. Returns a structured summary of what's on screen

Example: Hermes clicks a button via AIsistent

> Open Chrome and go to youtube.com

Hermes will:

1. Call capture_rdp_screen → sees desktop
2. Call detect_rdp_buttons(image) → finds Chrome icon coordinates
3. Call inject_rdp_click(12.5, 8.3) → clicks Chrome
4. Repeats capture → detect → click until done

---

OpenCode

OpenCode is an agentic CLI that also supports MCP tools.

1. Install AIsistent

pip install aistent[all]

2. Add to OpenCode config

Create or edit ~/.config/opencode/opencode.jsonc:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Or per-project, add to .opencode.jsonc in your project root:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

3. Verify it works

opencode

Then ask:

capture the screen and tell me what applications are open

OpenCode will call capture_rdp_screen → run_apple_ocr and return the result.

---

Claude Desktop

Claude Desktop supports MCP tools via its config file.

1. Locate Claude Desktop config

OS	Path
macOS	~/Library/Application Support/Claude/claude_desktop_config.json
Windows	%APPDATA%\Claude\claude_desktop_config.json

2. Add AIsistent

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

3. Restart Claude Desktop

Claude will show a hammer icon with AIsistent's available tools.

---

Cursor

Cursor IDE supports MCP tools.

1. Open Cursor settings

Settings → Features → MCP Servers

2. Add server

Name: AIsistent
Type: stdio
Command: aisistent

3. Use in chat

In Cursor's AI chat, type:

@aisistent capture the screen and detect buttons

---

Any MCP Client (generic stdio)

If your MCP client uses stdio transport, the configuration is always the same pattern:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

For HTTP/SSE transport instead of stdio:

# Start AIsistent as an SSE server on port 8100
python -c "from aisistent.server import mcp; mcp.run(transport='sse', port=8100)"

Then configure:

{
  "mcpServers": {
    "aisistent": {
      "url": "http://localhost:8100/sse",
      "type": "sse"
    }
  }
}

---

Headless RDP Transport

AIsistent supports two transport modes that can be switched at runtime:

Feature	GUI mode (default)	RDP headless mode
Local window needed	Yes (Microsoft Remote Desktop)	No
Capture method	screencapture / MSS	Direct RDP framebuffer via simple-rdp
Click method	pyautogui (local screen)	RDP input channel
macOS support	Full	Full (no XQuartz needed)
Linux support	MSS fullscreen	Full
Windows support	MSS fullscreen	Full

CLI mode

# GUI mode (default)
aisistent

# RDP headless with inline credentials
aisistent --mode rdp --host 192.168.1.100 --user admin --password secret

# RDP headless with environment variables
export RDP_HOST=192.168.1.100
export RDP_USER=admin
export RDP_PASS=secret
aisistent --mode rdp

MCP tools (switch at runtime)

connect_rdp(host="192.168.1.100", username="admin", password="secret")
capture_rdp_screen()     # → remote framebuffer, no local window
inject_rdp_click(50, 50) # → click sent via RDP protocol
disconnect_rdp()         # → back to GUI mode

Credentials precedence

Arguments > Environment variables (RDP_HOST, RDP_USER, RDP_PASS) > Config file

Install

pip install aistent[rdp]    # headless RDP only
pip install aistent[all]    # everything including RDP

---

winremote-mcp Integration

AIsistent works alongside winremote-mcp for comprehensive Windows remote management. Run both MCP servers:

aisistent &                              # AIsistent (stdio)
winremote-mcp --transport sse --port 8100 # winremote-mcp (SSE)

AIsistent handles the visual layer (OCR, detection, clicks) while winremote-mcp handles system operations (registry, services, processes, files, etc.).

---

Docs

• Architecture
• MCP Server
• Arquitectura (ES)
• Servidor MCP (ES)

---

Project Structure

AIsistent/
├── aisistent/
│   ├── __init__.py      # Version
│   ├── __main__.py      # Entry point (argparse: --mode gui|rdp)
│   ├── server.py        # MCP server + tools
│   ├── platform.py      # OS + device detection
│   ├── config.py        # Settings management
│   ├── capture.py       # Screen capture (delegates to transport)
│   ├── ocr.py           # OCR (Apple Vision / EasyOCR)
│   ├── detection.py     # YOLOv8 button detection
│   ├── action.py        # Click injection (delegates to transport)
│   ├── benchmark.py     # Performance benchmark
│   └── transport/       # Pluggable transport layer
│       ├── __init__.py  # get/set transport singleton
│       ├── base.py      # Abstract Transport class
│       ├── gui.py       # GUI transport (screencapture + pyautogui)
│       └── rdp.py       # RDP headless transport (simple-rdp)
├── docs/                # Documentation
├── pyproject.toml
└── README.md

License

MIT

---

<div align="center">

🇪🇸 AIsistent

Servidor MCP para automatización RDP no intrusiva. OCR, detección de botones con YOLOv8 e inyección de clics — sin instalar nada en la máquina remota.

	Modo GUI (default)	Modo RDP headless
Captura	Ventana RDP macOS / MSS pantalla completa	Framebuffer RDP directo (simple-rdp)
OCR	Apple Vision / EasyOCR	Apple Vision / EasyOCR
Detección	YOLO (CUDA / MPS / CPU)	YOLO (CUDA / MPS / CPU)
Click	pyautogui	Canal de input RDP

Inicio Rápido

# macOS (Apple Silicon)
pip install aistent[apple]

# Windows / Linux (CPU)
pip install aistent[cpu]

# Windows (NVIDIA CUDA)
pip install aistent[cuda]

# RDP headless
pip install aistent[rdp]

aisistent                          # modo GUI
aisistent --mode rdp --host HOST   # modo RDP headless

Benchmark

aisistent-bench                          # pantallazo real
aisistent-bench --synthetic              # imagen sintética
aisistent-bench --image captura.png      # imagen propia

Resultados reales (MacBook M5 — MPS)

Paso	Tiempo	Elementos
Captura	~0.22s	—
OCR (Apple Vision)	~0.43s	102 textos
YOLO (MPS float16)	~0.76s	59 botones
Total	~1.4s	—

Transporte RDP headless

aisistent --mode rdp --host 192.168.1.100 --user admin --password pass
# O vía tool MCP:
# connect_rdp(host="...", username="...", password="...")
# disconnect_rdp()

Integración con MCP Clients

Hermes MCP

Añade AIsistent como servidor MCP en ~/.config/hermes/config.json:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Luego inicia Hermes: hermes

OpenCode

Añade en ~/.config/opencode/opencode.jsonc:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Claude Desktop

Añade en ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "aisistent": {
      "command": "aisistent",
      "type": "stdio"
    }
  }
}

Licencia

MIT

</div>