MCP 服务器

mcp-common-crawl

MCP server for Common Crawl CDX that enables backlink discovery, expired domain finding, and competitor gap analysis without requiring API keys.

README

mcp-common-crawl

Built by Artur Ferreira @ The GEO Lab · 𝕏 @TheGEO_Lab · LinkedIn · Reddit

MCP server for Common Crawl CDX — backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).

Tools

Tool	Description
`discover_backlinks`	Find backlinks to any domain across 3 CC indexes
`find_expired`	Search for expired/parked domains in a niche via CC CDX
`check_domain`	Deep single domain check — live/expired/parked + CC page count
`competitor_gap`	Find domains linking to competitors but not to you

Features

✅ Production-tested — patterns used in production at TheGEOLab

Install

# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl

# Or in .mcp.json
{
  "mcpServers": {
    "common-crawl": {
      "command": "npx",
      "args": ["mcp-common-crawl"]
    }
  }
}

No API Keys Required

Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.

Usage

> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitors

Important Notes

Uses native fetch() for CC CDX (axios returns 404 on CC CDX — known issue)
Queries the 3 most recent CC indexes for best coverage
Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains

Attributions & Licence

Built and maintained by Artur Ferreira @ TheGEOLab.

Email: artur@thegeolab.net

Best Practice Attribution

This MCP server was built following the open source Best Practice Approach — reading community work for inspiration, then writing original content, and crediting every source.

Based on:

Model Context Protocol specification by Anthropic
MCP SDK (MIT)

Data source:

Common Crawl — free, open web archive (non-profit)
Common Crawl CDX API — index search endpoint

Backlink analysis concepts inspired by:

Ahrefs — backlink discovery and competitor gap methodology
Semrush — backlink analytics and domain comparison
Majestic — historic backlink index concepts

Technical decisions:

Native fetch() used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express — persistent debugging issue documented in geolab-backlinks)

All server code is original writing. No files were copied or adapted from any source. MIT licence.

Found this useful? ⭐ Star the repo and connect: 🌐 thegeolab.net · 𝕏 @TheGEO_Lab · LinkedIn · Reddit