MCP CDP Scraper — Web Scraping via Chrome DevTools Protocol for AI Agents

An MCP (Model Context Protocol) server that gives AI agents the ability to scrape web pages, fill forms, take screenshots, and extract structured data — all through Chrome DevTools Protocol with zero external dependencies.

Why This Exists

Most web scraping MCP servers wrap Playwright or Puppeteer — heavy dependencies that break in Docker, require npm, and add 200MB+ to your image. This server uses raw CDP over WebSockets with Python stdlib only. It connects to any Chrome/Chromium instance running with --remote-debugging-port and gives your AI agent 12 scraping tools.

No Playwright. No Selenium. No Puppeteer. No npm. Just Python stdlib + a running Chrome.

Features

• 12 MCP Tools: list_tabs, scrape_page, extract_text, extract_links, extract_images, extract_table, fill_form, click_element, screenshot, get_html, wait_for, scroll_to
• Zero dependencies — pure Python stdlib (socket, json, urllib, base64, struct)
• React/Vue compatible form filling — uses native setters + synthetic events
• STDIO JSON-RPC mode — drop-in for any MCP-compatible client (Claude Desktop, Hermes, etc.)
• Structured extraction — title, meta, headings, paragraphs, lists, links, images in one call
• Screenshot capture — base64 PNG or save to file
• Form automation — fill fields, click submit, wait for dynamic content

Quick Start

1. Launch Chrome with remote debugging

# Linux
google-chrome --remote-debugging-port=9222 --headless=new &

# macOS
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 &

# Docker
docker run -d -p 9222:9222 chromedp/chrome --remote-debugging-port=9222 --headless

2. Run the MCP server

# STDIO mode (for MCP clients)
python -m src.server --stdio

# Or print the manifest
python -m src.server --manifest

3. Use as a library

from src.server import MCPCDPScraperServer

server = MCPCDPScraperServer()

# Scrape a page
result = server.handle_tool_call("scrape_page", {"url": "https://news.ycombinator.com"})
import json
data = json.loads(result)
print(data["structured"]["title"])
print(f"Found {data['link_count']} links")

# Fill a form
result = server.handle_tool_call("fill_form", {
    "fields": {
        "#email": "user@example.com",
        "#password": "secret123"
    },
    "submit": "#login-button"
})

MCP Tool Reference

Tool	Description	Required Params
list_tabs	List open browser tabs	—
scrape_page	Navigate + extract everything	url
extract_text	Get text from element	selector
extract_links	Get all links	—
extract_images	Get all images	—
extract_table	Get table as 2D array	selector
fill_form	Fill form fields (React-safe)	fields
click_element	Click by selector	selector
screenshot	Capture screenshot	—
get_html	Get element HTML	selector
wait_for	Wait for selector	selector
scroll_to	Scroll to element/coords	—

Integration with MCP Clients

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "cdp-scraper": {
      "command": "python",
      "args": ["-m", "src.server", "--stdio"],
      "cwd": "/path/to/mcp-cdp-scraper"
    }
  }
}

Hermes Agent (config.yaml)

mcp:
  servers:
    cdp-scraper:
      command: python
      args: ["-m", "src.server", "--stdio"]
      cwd: /path/to/mcp-cdp-scraper

Demo

# Requires Chrome running on port 9222
python demo.py

Tests

python -m pytest tests/ -v

Docker

docker build -t mcp-cdp-scraper .
docker run -p 9222:9222 mcp-cdp-scraper

Architecture

MCP Client (Claude/Hermes/any)
    │  JSON-RPC over STDIO
    ▼
MCPCDPScraperServer (src/server.py)
    │  Tool dispatch + schema
    ▼
CDPScraper (src/cdp_scrape.py)
    │  Raw WebSocket CDP commands
    ▼
Chrome/Chromium (--remote-debugging-port=9222)
    │
    ▼
Web Page → Structured Data

Use Cases

• AI agent web scraping — Let your agent browse and extract data from any page
• Form automation — Login flows, signups, search submissions
• Content monitoring — Scrape and diff pages on a schedule
• Data extraction pipelines — Tables, product listings, article content
• Screenshot generation — Visual capture for QA or documentation
• Anti-bot bypass — Real Chrome instance avoids headless detection

License

MIT

Author

aaameobius-crypto — github.com/aaameobius-crypto

Freelance portfolio: https://ameobius-space.github.io/kwork-portfolio/