Site Crawler MCP

A powerful Model Context Protocol (MCP) server for crawling websites and extracting assets including images and SEO metadata. Built for e-commerce sites and general web crawling needs.

Features

• Comprehensive website analysis: 12 different extraction modes for complete website insights
• Multi-mode crawling: Extract multiple data types in a single pass
• Smart extraction: Advanced pattern matching for accurate data extraction
• Performance optimized: Concurrent crawling with rate limiting
• Security analysis: HTTPS, security headers, SSL/TLS information
• SEO analysis: Complete SEO audit including meta tags, structured data, and more
• Legal compliance: KVKK, GDPR, privacy policy detection
• Business intelligence: Brand info, references, contact details extraction

Installation

From PyPI (when published)

pip install site-crawler-mcp

From Source (Development)

Using uv (Recommended)

# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment with Python 3.12
uv venv --python 3.12
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies and package
uv sync

Using pip

# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment (recommended)
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install package in development mode
pip install -e .

Usage

As an MCP Server

Add to your MCP configuration file:

• Windows: %APPDATA%\Claude\claude_desktop_config.json
• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

Using uvx (Recommended)

{
  "mcpServers": {
    "site-crawler": {
      "command": "uvx",
      "args": ["--from", "/path/to/site-crawler-mcp", "site-crawler-mcp"]
    }
  }
}

Using uv run

{
  "mcpServers": {
    "site-crawler": {
      "command": "uv",
      "args": ["run", "site_crawler"],
      "cwd": "/path/to/site-crawler-mcp"
    }
  }
}

Using python directly

{
  "mcpServers": {
    "site-crawler": {
      "command": "python",
      "args": ["-m", "site_crawler.server"],
      "cwd": "/path/to/site-crawler-mcp/src",
      "env": {
        "PYTHONPATH": "/path/to/site-crawler-mcp/src"
      }
    }
  }
}

Note: Replace /path/to/site-crawler-mcp with your actual project path. On Windows, use backslashes and drive letters (e.g., C:\\Users\\YourName\\site-crawler-mcp).

Available Tools

site_crawlAssets

Crawl a website and extract various assets based on specified modes.

Parameters:

• url (string, required): The URL to start crawling from
• modes (array, required): Array of extraction modes (see below)
• depth (number, optional): Crawling depth (default: 1)
• max_pages (number, optional): Maximum pages to crawl (default: 50)

Available Modes:

• images: Extract all images with metadata (alt text, dimensions, format)
• meta: Basic SEO metadata (title, description, H1 tags)
• brand: Company branding information (logo, name, about pages)
• seo: Comprehensive SEO analysis (meta tags, structured data, open graph)
• performance: Page load metrics and performance indicators
• security: Security headers and HTTPS configuration
• compliance: Accessibility and regulatory compliance checks
• infrastructure: Server technology and CDN detection
• legal: Privacy policies, terms, KVKK compliance
• careers: Job opportunities and career pages
• references: Client testimonials and case studies
• contact: Contact information (email, phone, social media, address)

Example Requests:

1. Basic image extraction:

{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["images"],
    "depth": 1
  }
}

2. Full SEO and security audit:

{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["seo", "security", "performance"],
    "depth": 2
  }
}

3. Business intelligence gathering:

{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["brand", "contact", "references", "careers"],
    "depth": 3
  }
}

4. Legal compliance check:

{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["legal", "compliance"],
    "depth": 2
  }
}

Development

Requirements

• Python 3.10+
• BeautifulSoup4
• aiohttp
• MCP SDK
• uv (recommended for development)

Setup Development Environment

Using uv (Recommended)

# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment with Python 3.12
uv venv --python 3.12
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies and package
uv sync

Using pip

# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .

Running the Server

Using uv

# Run the MCP server
uv run site_crawler
# or
uv run site-crawler-mcp
# or
uv run python -m site_crawler.server

Using python directly

python -m site_crawler.server

Running Tests

# Using uv
uv run pytest tests/

# Using pip
pytest tests/

Project Structure

site-crawler-mcp/
├── README.md
├── requirements.txt
├── pyproject.toml
├── src/
│   └── site_crawler/
│       ├── __init__.py
│       ├── server.py
│       ├── crawler.py
│       └── utils.py
└── tests/
    ├── __init__.py
    └── test_crawler.py

Configuration

Environment Variables

• CRAWLER_MAX_CONCURRENT: Maximum concurrent requests (default: 5)
• CRAWLER_TIMEOUT: Request timeout in seconds (default: 30)
• CRAWLER_USER_AGENT: Custom user agent string

Rate Limiting

The crawler respects robots.txt and implements polite crawling:

• 1-2 second delay between requests to the same domain
• Maximum 5 concurrent requests
• Automatic retry with exponential backoff

Use Cases

E-commerce Analysis

Extract product images, pricing, and brand information:

"Analyze the e-commerce site example.com for product images, brand info, and contact details"

SEO and Performance Audit

Comprehensive SEO and performance analysis:

"Perform a full SEO audit of example.com including performance metrics and structured data"

Security Assessment

Check security headers and HTTPS configuration:

"Analyze the security posture of example.com including headers and SSL configuration"

Legal Compliance Check

Verify KVKK/GDPR compliance and privacy policies:

"Check example.com for KVKK compliance, privacy policies, and data protection measures"

Business Intelligence

Gather company information and references:

"Extract business information from example.com including company details, references, and career opportunities"

Contact Information Extraction

Find all contact details:

"Find all contact information on example.com including emails, phones, social media, and addresses"

Performance Considerations

• Images smaller than 50KB are filtered out by default
• Concurrent crawling limited to 5 pages simultaneously
• Memory-efficient streaming for large sites
• Automatic deduplication of URLs

Error Handling

The crawler handles various error scenarios gracefully:

• Network timeouts
• Invalid URLs
• Rate limiting (429 responses)
• JavaScript-heavy sites (graceful degradation)
• Memory limits

Contributing

Contributions are welcome! Please read our Contributing Guide for details on our code of conduct and the process for submitting pull requests.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built with MCP SDK
• Inspired by the need for better e-commerce crawling tools
• Thanks to the open-source community

Support

For issues and feature requests, please use the GitHub issue tracker.