Clara Windows UI MCP

Local-only MCP server for Windows UI automation oriented toward difficult targets such as game windows, legacy desktop apps, and surfaces that do not expose UI Automation reliably.

Why this exists

The built-in computer-use flow is convenient, but it can struggle with windows that render through game engines or custom graphics pipelines. This MCP takes a lower-level path:

• enumerate and focus windows through HWND
• drive mouse and keyboard input through native Win32 APIs
• capture the desktop or a target window to PNG files
• expose the same behavior through MCP stdio and a small local HTTP API

Current toolset

• list_windows
• get_foreground_window
• focus_window
• capture_screen
• move_mouse
• click_mouse
• press_key
• type_text

Vision foundation

The src/domain/vision module defines the shared, consumer-facing vision contract:

• SceneObservation carries capture provenance, OCR, and detection results.
• SceneEvent leaves room for optional temporal change notifications.
• VisionProfileRegistry isolates examples, training-set references, and rules by applicationId.

Capture and detection implementations are intentionally not part of this foundation. Consumers should depend on the scene/event contract rather than detector internals.

Vision service and tray

The native projects use the same safe session boundary as the Clara Cursor:

ClaraVisionBroker Windows service -> local named pipe -> ClaraVisionTray user-session agent

• The broker persists the high-level observation state (paused or observing) and relays it locally.
• The tray shows that state and lets the user start or pause observation.
• Neither component captures the desktop or performs automation. A future session agent will connect the existing opt-in capture pipeline only after a target selection and capture policy are defined.

Build native/ClaraVisionBroker and native/ClaraVisionTray in Release. Then run scripts/install-vision-service.cmd from an elevated Command Prompt and scripts/install-vision-tray.cmd from the interactive user session.

Run

npm.cmd install
npm.cmd run build
npm.cmd run start:stdio

For local diagnostics:

npm.cmd run start:api

Architecture choices

• run mode: stdio, api, or both
• deployment profile: local-only
• authentication: none
• storage: local file output only for captures
• organization: CQRS-style commands and queries with transport adapters

Notes

• Screenshots are written under captures/.
• This first version avoids UIA on purpose so it remains useful for games and canvas-rendered apps.
• The next logical upgrade is target-aware matching by template image, pixel region, or OCR.