Scrapy MCP — Universal Web Crawling & Scraping

A general-purpose MCP (Model Context Protocol) server for crawling and extracting structured data from any website. Inspired by Scrapy's architecture.

Architecture

User (Claude) → MCP Tools → Engine → Scheduler → Downloader → Spider → Pipeline → Result
                                ↓            ↓           ↓
                             Settings    Middleware    Extractors

• Engine — Central coordinator (BFS crawl loop)
• Scheduler — FIFO queue with URL deduplication
• Downloader — Async HTTP via httpx with middleware chain (retry, UA rotation, rate limiting)
• Spider — Parses responses using configurable CSS/XPath/regex extractors
• Pipeline — Post-processing chain (clean, validate, transform, dedup)
• Middleware — Pluggable hooks for request/response processing

Installation

pip install -r requirements.txt

Register with Claude Code

Edit ~/.claude/mcp.json:

{
  "mcpServers": {
    "scrapy-mcp": {
      "command": "python",
      "args": ["d:/project/scrapy-mcp/server.py"]
    }
  }
}

MCP Tools

1. crawl — Structured crawling

Crawl a website and extract structured data using CSS selectors or XPath.

Parameters:

Param	Type	Required	Description
urls	string[]	✅	Starting URLs
spider	string	❌	Pre-configured spider name (from spiders/ directory)
item_selector	string	❌	CSS/XPath selector for item containers
fields	object	❌	Field extraction rules
pagination	object	❌	Pagination config
max_pages	integer	❌	Max pages to crawl (default: 10)
max_depth	integer	❌	Max link depth (default: 2)
output_format	string	❌	json / markdown / csv (default: json)
allowed_domains	string[]	❌	Restrict to domains

Example:

crawl({
  "urls": ["https://books.toscrape.com"],
  "item_selector": "article.product_pod",
  "fields": {
    "title": {"selector": "h3 a", "attr": "title"},
    "price": {"selector": ".price_color", "regex": "([\\d.]+)"}
  },
  "pagination": {"type": "next_link", "selector": ".next a", "max_pages": 3}
})

2. scrape_page — Single page extraction

Fetch and extract content from a single page. Supports multiple modes.

Parameters:

Param	Type	Required	Description
url	string	✅	Target page URL
mode	string	❌	auto / markdown / structured / raw (default: auto)
extract_schema	boolean	❌	Extract Schema.org / Open Graph (default: true)
extract_links	boolean	❌	Extract all links (default: false)
extract_images	boolean	❌	Extract all images (default: false)
selectors	object	❌	Custom CSS selectors for specific elements

Modes:

• auto — Smart: returns structured JSON if schema data found, otherwise markdown
• markdown — Clean readable text with headings, images, links
• structured — Full JSON with all extracted metadata + schema data
• raw — Original HTML source

3. search_and_crawl — Search then crawl

Search any website and crawl each result page for details.

Parameters:

Param	Type	Required	Description
search_url	string	✅	URL template with {query} placeholder
query	string	✅	Search keywords
spider	string	❌	Spider name for result pages
result_item_selector	string	❌	Selector for result items on search page
result_link_selector	string	❌	Selector for links within results
fields	object	❌	Fields to extract from each result page
max_results	integer	❌	Max results (default: 10)
crawl_each_result	boolean	❌	Crawl each result page (default: true)

4. extract_schema — Structured data extraction

Extract Schema.org JSON-LD, Open Graph, Twitter Cards, and meta tags.

Parameters:

Param	Type	Required	Description
url	string	❌	Page URL
html	string	❌	Or provide raw HTML
schema_types	string[]	❌	Filter by type (e.g. ["Product", "Article"])
include_raw	boolean	❌	Include raw JSON-LD objects

5. list_spiders — Show available spiders

Lists all pre-configured spider definitions from the spiders/ directory.

Spider Configuration (YAML)

Create YAML files in spiders/ to define reusable spiders without writing code:

name: my-spider
description: "What this spider does"
start_urls:
  - "https://example.com/items"
item_selector: ".item"
fields:
  title:
    selector: ".title"
    type: text
  link:
    selector: "a"
    type: attr
    attr: href
    transform: absolute_url
pagination:
  type: next_link
  selector: ".next a"
  max_pages: 5

See spiders/example.yml for an annotated template and spiders/books_toscrape.yml for a working example.

Field config reference

Key	Description
selector	CSS or XPath expression
type	text (default), attr, html, list
attr	HTML attribute to extract (e.g. href, src)
regex	Regex to apply after extraction (first group returned)
transform	strip, int, float, absolute_url, lower, upper
default	Fallback value

Pagination types

Type	Description
next_link	Follow a "next" link using a CSS selector
url_pattern	Generate URLs from a {page} template
scroll	Scroll-based (requires JS renderer — not yet supported)

Settings

Default settings (overridable via spider config):

Setting	Default	Description
DOWNLOAD_DELAY	0.5s	Delay between requests to same domain
CONCURRENT_REQUESTS	8	Max simultaneous requests
TIMEOUT	30s	Request timeout
MAX_DEPTH	3	Max link depth
MAX_PAGES	50	Max pages to crawl
MAX_ITEMS	500	Max items to collect
RETRY_TIMES	3	Retry attempts on failure
RETRY_HTTP_CODES	[500, 502, 503, 504, 408, 429]	Codes to retry
ROBOTSTXT_OBEY	false	Respect robots.txt

Supported Extraction

• CSS selectors — .class, #id, div > p, a[href]
• XPath — /html/body/div, //article//h2
• Regex — Post-process extracted text
• Schema.org — JSON-LD, microdata
• Open Graph — og:title, og:description, og:image, etc.
• Meta tags — description, keywords, etc.
• Auto-extraction — All links, images, tables from a page

Quick Test

cd d:/project/scrapy-mcp

# Test extraction
python -c "
from mcp_scrapy.extractors import FieldExtractor
import httpx, asyncio

async def test():
    async with httpx.AsyncClient() as c:
        r = await c.get('https://httpbin.org/html')
    ext = FieldExtractor(r.text)
    print('Title:', ext.extract_field({'selector': 'h1'}))
    print('Links:', len(ext.extract_all_links()))

asyncio.run(test())
"

# Test spider loading
python -c "
from mcp_scrapy.spider import SpiderLoader
loader = SpiderLoader(['spiders'])
for s in loader.list_all():
    print(f'{s[\"name\"]}: {s[\"description\"]}')
"

Project Structure

scrapy-mcp/
├── server.py                  # MCP entry point
├── requirements.txt
├── mcp_scrapy/
│   ├── __init__.py
│   ├── engine.py              # CrawlEngine coordinator
│   ├── scheduler.py           # Request queue + dedup
│   ├── downloader.py          # Async HTTP + middleware
│   ├── spider.py              # Spider + YAML loader
│   ├── request.py             # Request/Response/Item types
│   ├── pipelines.py           # Item post-processing
│   ├── middleware.py          # Downloader middleware
│   ├── extractors.py          # CSS/XPath/regex field extraction
│   ├── exporters.py           # JSON/Markdown/CSV output
│   ├── schema_extractor.py    # Schema.org / Open Graph
│   ├── settings.py            # Configuration
│   └── tools/
│       ├── __init__.py
│       ├── crawl.py           # crawl tool
│       ├── scrape_page.py     # scrape_page tool
│       ├── search.py          # search_and_crawl tool
│       ├── extract_schema.py  # extract_schema tool
│       └── list_spiders.py    # list_spiders tool
├── spiders/                   # Spider YAML configs
│   ├── example.yml
│   ├── hackernews.yml
│   └── books_toscrape.yml
└── tests/
    ├── test_extractors.py
    ├── test_pipelines.py
    └── test_scheduler.py