Web Content Scraper MCP Worker

A high-performance Cloudflare Worker project that presents an agent-ready Model Context Protocol (MCP) Tool and REST API endpoint for dynamic web content extraction. Built with TypeScript, Cloudflare Workflows, Cloudflare Browser Rendering (@cloudflare/puppeteer), and the MCP SDK (@modelcontextprotocol/sdk), monetized with the x402 payment protocol ($0.005 USDC per call on Base).

---

🌟 Key Features

1. Dynamic Page Scraping & Headless Chrome:
   • Uses Cloudflare Browser Rendering (@cloudflare/puppeteer) to execute dynamic JavaScript, handle client-side rendering (SPAs/React/Vue/Next.js), and bypass anti-bot layout hurdles.
2. Boilerplate & Ad Stripping:
   • In-browser DOM parsing automatically strips scripts, stylesheets, navigation bars, headers, footers, sidebars, cookie banners, popups, and advertisement blocks.
3. Structured Content Output:
   • Markdown Format (default): Converts page hierarchy into clean structured Markdown (# Headings, paragraphs, lists, blockquotes, code blocks, links, and images).
   • JSON Format: Extracts metadata (title, author, siteName, description), text content, headings array, link objects, and image objects.
4. Cloudflare Workflows Integration:
   • Includes ScraperWorkflow for step-by-step asynchronous processing, step retries, and error isolation.
5. x402 Protocol Monetization:
   • Protected endpoints require $0.005 USDC payment per execution on the Base network via the x402 microtransaction standard.

---

🚀 MCP Tool Specification

Tool Name: scrape_web_content

Description:

"Scrapes a dynamic web page, bypasses client-side rendering, removes clutter/ads, and extracts core content as clean Markdown or structured JSON. Use this tool when you need to read web articles, documentation, or extract main body text from any URL."

Parameters:

• url (string, required): The HTTP/HTTPS web page URL to scrape.
• format (string, optional): "markdown" or "json" (default: "markdown").

Response Schema:

{
  "title": "Article Title",
  "sourceUrl": "https://example.com/article",
  "extractedContent": "# Article Title\n\nCore content formatted in clean markdown...",
  "wordCount": 482
}

---

📡 REST API Endpoints

1. Health Diagnostics

GET /health

Response:

{
  "status": "ok",
  "service": "web-scraper-mcp-worker",
  "timestamp": "2026-08-05T16:00:00.000Z",
  "browserConfigured": true,
  "workflowConfigured": true,
  "monetization": {
    "protocol": "x402",
    "price_usdc": "0.005",
    "network": "base",
    "payee_address": "0x742d35Cc6634C0532925a3b844Bc454e4438f44e"
  }
}

---

2. Scraping Endpoint (x402 Protected)

POST /api/scrape
Content-Type: application/json

{
  "url": "https://example.com/blog/dynamic-article",
  "format": "markdown"
}

---

3. Workflow Trigger Endpoint

POST /api/workflow/scrape
Content-Type: application/json

{
  "url": "https://example.com/docs/api",
  "format": "json"
}

---

4. MCP JSON-RPC Protocol Endpoint (x402 Protected)

POST /mcp
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "scrape_web_content",
    "arguments": {
      "url": "https://news.ycombinator.com",
      "format": "markdown"
    }
  }
}

---

💳 x402 Monetization Flow

Unauthenticated requests to protected endpoints return an HTTP 402 Payment Required status code with x402 headers:

HTTP/1.1 402 Payment Required
WWW-Authenticate: X-402 payee="0x742d35Cc6634C0532925a3b844Bc454e4438f44e" asset="0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913" amount="5000" network="eip155:8453"
Content-Type: application/json

{
  "x402Version": 1,
  "error": "Payment Required",
  "payment": {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "5000",
    "human_amount": "$0.005 USDC",
    "payee": "0x742d35Cc6634C0532925a3b844Bc454e4438f44e",
    "resource": "/api/scrape",
    "description": "Dynamic Web Scraper & Content Extractor ($0.005 USDC per execution)"
  }
}

Include payment authorization proof in the X-PAYMENT header to execute requests.

---

🛠 Local Development & Deployment

Prerequisites

• Node.js 18+
• Cloudflare Workers Paid account (for Browser Rendering in production)

Commands

# Install dependencies
npm install

# Type Check
npm run type-check

# Local Development Server
npm run dev

# Deploy to Cloudflare Workers
npm run deploy

---

📄 License

MIT License.