crawler-mcp

A web-crawler MCP server, built in the same style as weather-mcp (TypeScript + @modelcontextprotocol/sdk + Express + Zod).

It fetches pages with a fast axios + cheerio path and automatically falls back to a Playwright headless browser when a page needs JavaScript to render. Designed to be used as a Claude connector (local stdio or remote HTTP).

Tools

Tool	What it does
fetch_page	Fetch one URL and return its content as Markdown, text, or raw HTML.
extract_links	List all hyperlinks on a page (absolute URLs, optional same-domain filter).
crawl_site	Recursively crawl from a start URL up to a max depth/page count.
extract_by_selector	Pull specific data via a CSS selector (text or an attribute).

Each tool accepts a render option: auto (default — static first, browser if needed), static (never use a browser), or browser (always render with Playwright).

Setup

cd crawler-mcp
npm install

# Optional — only needed for JavaScript-heavy / SPA sites:
npx playwright install chromium

cp .env.example .env   # then edit if you want

Run

# Local (stdio) — for Claude Desktop:
npm start

# Remote (HTTP) — for a custom/remote connector:
npm run start:http     # serves http://localhost:3001/mcp

Use it as a Claude connector

Claude Desktop (local, stdio)

Add to your claude_desktop_config.json (Settings → Developer → Edit Config):

{
  "mcpServers": {
    "crawler": {
      "command": "npx",
      "args": ["tsx", "D:\\MCP\\crawler-mcp\\server.ts"]
    }
  }
}

Restart Claude Desktop. The four crawler tools appear under the connectors menu.

Remote HTTP connector (claude.ai / Desktop "Add custom connector")

1. Start the server in HTTP mode and expose it over HTTPS (e.g. behind a reverse proxy or a tunnel such as cloudflared / ngrok):

   npm run start:http
   

2. In Claude → Settings → Connectors → Add custom connector, enter the URL:

   https://your-host/mcp
   

Remote connectors on claude.ai generally require a public HTTPS URL. http://localhost:3001/mcp works for local testing tools but not for the hosted claude.ai web app.

Configuration (.env)

Variable	Default	Purpose
MCP_TRANSPORT	stdio	stdio or http.
MCP_HTTP_PORT	3001	Port for HTTP mode.
CRAWLER_USER_AGENT	crawler-mcp/1.0 …	User-Agent for all requests.
CRAWLER_TIMEOUT_MS	15000	Per-request timeout.
CRAWLER_MAX_CHARS	20000	Default cap on returned page content.

Notes

• Fetched pages are cached in memory for 5 minutes to avoid refetching during a crawl.
• crawl_site is bounded (max_depth ≤ 3, max_pages ≤ 50) and adds a politeness delay between requests.
• Playwright is an optional dependency — if it isn't installed, render: auto still works for static sites and render: browser returns a clear error.
• This server does not currently parse robots.txt; crawl responsibibly and only sites you are authorized to crawl.