cloudflare-crawl-mcp

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

Built with FastMCP and uv.

Tools

Tool	Description
scrape_url	Fetch a single page as Markdown. The primary tool — use this when you know the URL.
map_url	Discover URLs on a site without fetching content. Use to find the right page first.
crawl_url	Crawl multiple pages and return all content as Markdown.

Typical workflow: map_url to find pages → scrape_url to read the right one.

Prerequisites

1. A Cloudflare account with Browser Rendering enabled
2. An API token with Account > Browser Rendering > Edit permission (create one here)
3. uv installed

Configuration

Variable	Required	Description
CF_API_TOKEN	Yes	Cloudflare API token
CF_ACCOUNT_ID	Yes	Cloudflare Account ID
CF_RATE_LIMIT	No	API requests per minute (default: 6 for Free, set to 600 for Paid)

Setup

Claude Code

claude mcp add cloudflare-crawl \
  -e CF_API_TOKEN=your_api_token \
  -e CF_ACCOUNT_ID=your_account_id \
  -- uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Codex

codex mcp add cloudflare-crawl \
  -- env CF_API_TOKEN="your_api_token" CF_ACCOUNT_ID="your_account_id" \
  uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project or ~/.cursor/mcp.json globally:

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Tool Reference

scrape_url

Fetch a single page and return its content as Markdown.

url (string, required)    — The URL to fetch
render (boolean, optional) — Render JavaScript with headless browser (default: true, set false for static pages)

map_url

Discover URLs on a website without fetching full content.

url (string, required)                  — Starting URL
limit (number, optional)                — Max URLs to discover (default: 50)
depth (number, optional)                — Link depth to follow (default: 2)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs (e.g. "https://example.com/docs/**")
exclude_patterns (string[], optional)   — Skip matching URLs

crawl_url

Crawl multiple pages and return all content as Markdown.

url (string, required)                  — Starting URL
limit (number, optional)                — Max pages to crawl (default: 10)
depth (number, optional)                — Link depth (default: 1)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs
exclude_patterns (string[], optional)   — Skip matching URLs
render (boolean, optional)              — Render JavaScript (default: true)

Cloudflare Plan Limits

	Free	Paid
Browser time	10 min/day	10 hrs/month
API rate limit	6 req/min	600 req/min
Concurrent browsers	3	10
Max pages per job	100,000	100,000
Max job duration	7 days	7 days
Results available	14 days	14 days

render: false crawls run on Workers instead of a headless browser and do not consume browser time.

License

MIT