MCP Virtual User

A synthetic human in a Docker container. Has ears, a mouth, eyes, and hands.

What is this?

A fully self-contained environment that simulates a real user interacting with web apps via voice and browser. It has:

• Virtual microphone — MCP can inject audio (TTS or raw) that any app thinks is coming from a hardware mic
• Virtual speakers — MCP can capture and transcribe whatever audio the OS plays back
• Real browser — Chromium with Playwright control, persistent login sessions
• Real display — Xvfb + VNC for debugging (watch what's happening live)
• MCP interface — Everything exposed as tools via Streamable HTTP

Use Cases

• Test ChatGPT voice mode — inject "What's the weather?" into the mic, capture ChatGPT's spoken response, transcribe it, assert on it
• Test Gemini Live — same flow against Google's voice AI
• Test our Mobile Mesh UI — full end-to-end voice conversation testing against our own app
• Any voice-enabled web app — if it uses the browser's mic/speaker, we can test it

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Docker Container                                            │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  MCP Server (port 8360)                              │    │
│  │                                                     │    │
│  │  Audio Tools:                                       │    │
│  │    tts_to_mic(text)     → Piper TTS → virtual mic   │    │
│  │    transcribe_speakers() → parec → Whisper STT      │    │
│  │    inject_audio(b64)    → raw audio → virtual mic   │    │
│  │    capture_audio(secs)  → raw audio from speakers   │    │
│  │    wait_for_speech()    → detect + transcribe       │    │
│  │                                                     │    │
│  │  Browser Tools:                                     │    │
│  │    browser_navigate, click, type, screenshot, etc.  │    │
│  │    browser_grant_mic_permission(origin)             │    │
│  │                                                     │    │
│  │  Screen Tools:                                      │    │
│  │    screen_screenshot, screen_size, vnc_url          │    │
│  └──────────┬──────────────────────────┬───────────────┘    │
│             │                          │                     │
│  ┌──────────▼──────────┐  ┌───────────▼───────────────┐    │
│  │  PulseAudio         │  │  Playwright + Chromium    │    │
│  │                     │  │                           │    │
│  │  virtual_mic ◀──────│──│── browser reads as mic    │    │
│  │  (pipe-source)      │  │                           │    │
│  │                     │  │  browser plays audio ──▶  │    │
│  │  virtual_speaker ───│──│── captured via .monitor   │    │
│  │  (null-sink)        │  │                           │    │
│  └─────────────────────┘  └───────────────────────────┘    │
│                                                             │
│  ┌─────────────────────┐  ┌───────────────────────────┐    │
│  │  Xvfb :99           │  │  x11vnc + noVNC          │    │
│  │  1920x1080x24       │  │  port 5900 / 6080        │    │
│  └─────────────────────┘  └───────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Quick Start

# Build and start
docker compose up -d --build

# Watch the virtual display (open in your browser)
open http://localhost:6080/vnc.html?autoconnect=true

# Test the MCP server
curl http://localhost:8360/mcp -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"health","arguments":{}}}'

# Run smoke tests
bash test-smoke.sh

First-Time Setup (One-Time)

# 1. Build and start
docker compose up -d --build

# 2. Open VNC in your browser to see the desktop
open http://localhost:6080/vnc.html?autoconnect=true

# 3. In the VNC window, Chromium is running. Log into:
#    - https://chat.openai.com (ChatGPT)
#    - https://gemini.google.com (Gemini)

# 4. Export cookies to DragonsKeep so they persist
bash scripts/export-browser-cookies.sh

# 5. Done! Future container starts auto-inject the stored sessions.

Running Tests

# Infrastructure tests (no login required)
cd tests && pip install -r requirements.txt
pytest test_audio_roundtrip.py test_browser.py -v

# ChatGPT conversation tests (requires login)
pytest test_conversations.py -v -m chatgpt --timeout=120

# Gemini conversation tests
pytest test_conversations.py -v -m gemini --timeout=120

# Our Mesh UI conversation tests
pytest test_conversations.py -v -m meshui --timeout=120

# All conversation tests
pytest test_conversations.py -v --timeout=120

Example: Test ChatGPT Voice Mode

# From any MCP client (Kiro, our mesh agent, etc.)

# 1. Navigate to ChatGPT
browser_navigate("https://chat.openai.com")

# 2. Grant mic permission
browser_grant_mic_permission("https://chat.openai.com")

# 3. Click the voice mode button
browser_click("[data-testid='voice-mode-button']")

# 4. Speak a question (injected into the virtual mic)
tts_to_mic("What is the capital of France?")

# 5. Wait for ChatGPT to respond and transcribe what it says
response = wait_for_speech(timeout_seconds=15)
# response == "The capital of France is Paris."

# 6. Assert
assert "Paris" in response

Session Management

Login sessions persist in ./data/browser-profile/ (mounted volume).

For ChatGPT/Gemini auth tokens, use DragonsKeep:

1. Store cookies in DragonsKeep as chatgpt-session / gemini-session
2. On container start, inject them into the browser profile

Or: log in manually once via VNC (http://localhost:6080), session persists.

Ports

Port	Service
8360	MCP Server (Streamable HTTP)
6080	noVNC (browser-based VNC viewer)
5900	VNC direct

MCP Tools

Audio

Tool	Description
tts_to_mic	Synthesize text → inject as microphone input
transcribe_speakers	Capture speaker output → transcribe to text
inject_audio	Push raw audio bytes into the virtual mic
capture_audio	Record raw audio from the virtual speakers
wait_for_speech	Detect speech on speakers, wait for it to finish, transcribe

Browser

Tool	Description
browser_navigate	Go to a URL
browser_click	Click an element
browser_type	Type into an input
browser_press_key	Press a keyboard key
browser_screenshot	Take a page screenshot
browser_get_text	Get page text content
browser_evaluate	Execute JavaScript
browser_wait_for_text	Wait for text to appear
browser_url	Get current URL
browser_grant_mic_permission	Allow mic access for an origin

Screen

Tool	Description
screen_screenshot	Full desktop screenshot
screen_size	Get display resolution
vnc_url	Get the live VNC viewer URL