silkworm-mcp

This is a full-featured MCP server for building scrapers with:

• silkworm-rs: async crawling, fetching, follow links, and spider execution
• scraper-rs: fast Rust-backed HTML parsing with CSS and XPath selectors

It is designed for LLM-assisted scraper development, so the server exposes both low-level page inspection tools and higher-level workflow helpers for validating selector plans and generating starter spider code.

An example: https://github.com/BitingSnakes/silkworm-example

Features

• Fetch pages through silkworm's regular HTTP client or CDP renderer.
• Query selectors directly against a CDP-rendered DOM snapshot.
• Analyze inline and linked CSS with tinycss2, then optionally map selectors back onto HTML.
• Extract structured records from live rendered pages before committing to a full crawl.
• Cache HTML in a local document store and reuse it via document_handle.
• Bound the document cache with max-document, max-bytes, and idle-TTL controls.
• Inspect pages with summaries, parsed DOM trees, prettified HTML, CSS/XPath queries, selector comparisons, and link extraction.
• Run ad hoc crawls from a structured CrawlBlueprint.
• Generate reusable silkworm spider templates from the same blueprint and statically validate them, including pattern-specific variants for list-only, list+detail, sitemap/XML, and CDP-heavy crawls.
• Expose MCP diagnostics plus HTTP /healthz and /readyz routes for production monitoring.
• Publish MCP resources and prompts so clients can discover workflows, Silkworm idioms, and blueprint schemas.

Tools

• store_html_document
• list_documents
• delete_document
• clear_documents
• server_status
• inspect_document
• parse_html_document
• parse_html_fragment
• prettify_document
• query_selector
• analyze_css_selectors
• find_selectors_by_text
• compare_selectors
• extract_links
• silkworm_fetch
• silkworm_fetch_cdp
• query_selector_cdp
• extract_structured_data_cdp
• run_crawl_blueprint
• generate_spider_template
• validate_spider_code

Run

Install dependencies:

uv sync

Run over stdio for a desktop MCP client:

uv run python mcp_server.py --transport stdio

Run over HTTP:

uv run python mcp_server.py --transport http --host 127.0.0.1 --port 8000

HTTP deployments also expose:

• GET /healthz: process liveness
• GET /readyz: readiness, optionally including a CDP browser probe

The project also exposes a console entrypoint:

uv run silkworm-mcp --transport stdio

Docker

Build the image:

docker build -t silkworm-mcp .

Run the container over HTTP on port 8000:

docker run --rm -it -p 8000:8000 silkworm-mcp

The container entrypoint starts two processes by default:

• the MCP server over HTTP on 0.0.0.0:8000
• a bundled Lightpanda browser on 127.0.0.1:9222 for CDP-backed tools such as silkworm_fetch_cdp, query_selector_cdp, and extract_structured_data_cdp

Useful container environment variables:

• MCP_TRANSPORT (default: http)
• MCP_HOST (default: 0.0.0.0)
• MCP_PORT (default: 8000)
• MCP_PATH
• LIGHTPANDA_ENABLED (default: 1)
• LIGHTPANDA_HOST (default: 127.0.0.1)
• LIGHTPANDA_PORT (default: 9222)
• LIGHTPANDA_ADVERTISE_HOST (default: unset, falls back to LIGHTPANDA_HOST)
• LIGHTPANDA_LOG_FORMAT (default: pretty)
• LIGHTPANDA_LOG_LEVEL (default: info)

When Lightpanda binds to 0.0.0.0 inside a container, set LIGHTPANDA_ADVERTISE_HOST to a reachable hostname such as the container DNS name. Otherwise /json/version can advertise ws://0.0.0.0:9222/, which remote CDP clients cannot use.

Example with custom document-cache limits:

docker run --rm -it \
  -p 8000:8000 \
  -e SILKWORM_MCP_DOCUMENT_MAX_COUNT=256 \
  -e SILKWORM_MCP_DOCUMENT_MAX_TOTAL_BYTES=64000000 \
  -e SILKWORM_MCP_DOCUMENT_TTL_SECONDS=7200 \
  silkworm-mcp

For local development, compose.yml provides the same setup with health checks and restart policy:

docker compose up --build

Then verify the container is ready:

curl http://127.0.0.1:8000/readyz

Key runtime environment variables:

• SILKWORM_MCP_DOCUMENT_MAX_COUNT
• SILKWORM_MCP_DOCUMENT_MAX_TOTAL_BYTES
• SILKWORM_MCP_DOCUMENT_TTL_SECONDS
• SILKWORM_MCP_DOCUMENT_STORE_PATH
• SILKWORM_MCP_LOG_LEVEL
• SILKWORM_MCP_READINESS_REQUIRE_CDP
• SILKWORM_MCP_READINESS_CDP_WS_ENDPOINT

Example Workflow

1. Call silkworm_fetch for the target page.
2. Use the returned document_handle with inspect_document.
3. Use parse_html_document or parse_html_fragment when you need exact parser structure, node types, or parser errors.
4. Use find_selectors_by_text to derive candidates from visible text, then iterate on query_selector, compare_selectors, and analyze_css_selectors when stylesheet structure or hidden elements matter.
5. For JS-heavy pages, use query_selector_cdp or extract_structured_data_cdp against the rendered DOM.
6. Use extract_links to verify pagination or detail pages.
7. Feed the stable plan into run_crawl_blueprint.
8. Convert the same blueprint into code with generate_spider_template, then check it with validate_spider_code.

Useful built-in MCP references:

• silkworm://reference/overview
• silkworm://reference/silkworm-cheatsheet
• silkworm://reference/silkworm-playbook
• silkworm://reference/template-variants
• silkworm://reference/scraper-rs-cheatsheet
• silkworm://reference/crawl-blueprint-schema

Use transport: "cdp" when pages require JavaScript rendering. run_crawl_blueprint will connect to the configured CDP endpoint, and generate_spider_template will emit a starter spider that runs through CDPClient instead of the default HTTP client.

Both run_crawl_blueprint and generate_spider_template accept a variant override. When omitted, they infer a crawl style from the blueprint:

• list_only: listing pages emit items directly, with optional pagination
• list_detail: listing pages schedule detail requests and a separate parse_detail
• sitemap_xml: sitemap/XML entrypoints are fetched with meta={"allow_non_html": True} and parsed before scheduling page requests
• cdp_heavy: rendered-page crawls keep the CDP execution path and a general-purpose parse/follow flow

run_crawl_blueprint returns the resolved execution_variant, and generate_spider_template returns the resolved template_variant, so clients can see which crawl shape was actually used.

Testing

Run the automated test suite with:

just test

Acknowledgement

This project builds on the excellent work behind FastMCP, silkworm-rs, and scraper-rs. Together they provide the MCP server framework, crawling runtime, and HTML parsing foundations that make this project possible.