Agent-Scrape

Pay-per-call web scraping for AI agents — no API keys, no signup, no account.

Agents connect over MCP (Model Context Protocol), call scrape_url(), get a 402 Payment Required challenge, their wallet pays micro-USDC on Base via the x402 protocol, and the scraped markdown comes back. Payment is the auth.

Live endpoint

https://scrape.diazites.online/mcp — production MCP endpoint (Base mainnet, x402/USDC).

Agent ──tools/call(scrape_url)──▶ Agent-Scrape
        ◀── 402 + PAYMENT-REQUIRED ──
        ── signs USDC payment (wallet) ──▶  Base chain
        ◀────────── 200 + markdown ──────  (verified by facilitator)

Endpoints

Endpoint	Purpose
POST /mcp	MCP streamable-HTTP endpoint (initialize / tools/list / tools/call)
GET /pricing	Agent-readable pricing discovery (payTo, price, network)
GET /healthz	Health check
GET /	Landing info

Tool pricing

Tool	Price	Description
scrape_url(url, max_chars?)	$0.01 USDC	Main content of any page as clean markdown

Network: Base mainnet (eip155:8453). Assets: USDC (0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913).

Run it

python3 -m venv venv
venv/bin/pip install -r requirements.txt   # or: mcp x402 fastapi uvicorn trafilatura httpx
cp .env.example .env                        # then fill in keys
venv/bin/python app.py                      # http://0.0.0.0:8093

Configuration (.env)

Variable	Required	Description
AGENT_SCRAPE_PAY_TO	✅	Base wallet that receives USDC
AGENT_SCRAPE_CDP_API_KEY_ID	for payments	Coinbase CDP API key (portal.cdp.coinbase.com)
AGENT_SCRAPE_CDP_API_KEY_SECRET	for payments	Coinbase CDP API key secret (PEM or Ed25519)
AGENT_SCRAPE_DEV_MODE		1 = bypass payments (demo), 0 = enforce x402
AGENT_SCRAPE_NETWORK		eip155:8453 mainnet, eip155:84532 Base Sepolia
AGENT_SCRAPE_PRICE		Price in USD per call (default 0.01)
AGENT_SCRAPE_PORT		Default 8093

Seller setup (5 minutes, free)

1. Generate a Base wallet (or use any EVM wallet you control):

   python -c "from eth_account import Account; a=Account.create(); print(a.address, a.key.hex())"
   

   Put the address in AGENT_SCRAPE_PAY_TO. Fund it with USDC on Base to withdraw (optional — payments can also stay on-chain).
2. Create CDP API keys at https://portal.cdp.coinbase.com → API keys. Paste into .env.
3. Set AGENT_SCRAPE_DEV_MODE=0, restart. GET /pricing now shows production mode.

How the payment flow works

1. Agent calls scrape_url → server responds 402 Payment Required with a PAYMENT-REQUIRED header (base64 JSON: scheme exact, network eip155:8453, USDC amount, payTo wallet).
2. Agent's x402-capable wallet (Coinbase Smart Wallet, any EIP-3009 signer) signs a USDC payment authorization.
3. Server verifies via the Coinbase CDP x402 facilitator (verify → settle on Base).
4. On success the tool executes and the markdown is returned with a PAYMENT-RESPONSE settlement header.

Dev mode (AGENT_SCRAPE_DEV_MODE=1) skips steps 1-3 for testing.

Try it

# demo client (dev mode or production)
venv/bin/python e2e_test.py http://localhost:8093/mcp https://example.com

# see the 402 challenge a real agent gets
curl -s -i -X POST http://localhost:8093/mcp \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}'

License

MIT. Payments protocol: x402 (Apache-2.0).