Spider MCP Server

An MCP server for crawling and spidering documentation websites, extracting clean text content, and using LLM-powered analysis to provide intelligent summaries and context through the Model Context Protocol.

Features

• Crawl entire documentation websites with configurable depth limits
• Extract clean text content from HTML pages using multiple extraction methods
• Intelligent content parsing that removes navigation, ads, and other noise
• LLM-powered content analysis using Anthropic Claude (Haiku/Sonnet)
• Automatic content summarization and key point extraction
• Intelligent code example extraction and categorization
• Code example detection across multiple programming languages
• Intelligent link discovery and relevance ranking
• Content type classification (tutorial, reference, API docs, etc.)
• Respect robots.txt and implement proper crawling etiquette
• File-based caching with TTL and compression
• Search through cached documentation content with LLM enhancement
• Concurrent crawling with rate limiting
• Support for various documentation layouts and formats

Installation

git clone <repository-url>
cd spider-mcp
bun install
cp .env.example .env

Usage

As MCP Server

Run the server to expose MCP tools:

bun run dev

MCP Client Integration

This server implements the Model Context Protocol (MCP) specification and communicates via stdio transport. MCP enables AI assistants to securely access external tools and data sources. To use this server, you need an MCP-compatible client like Claude Desktop, or you can integrate it programmatically using the MCP SDK.

Claude Desktop Integration

Add to your Claude Desktop MCP configuration:

{
  "mcpServers": {
    "spider-mcp": {
      "command": "bun",
      "args": ["run", "/path/to/spider-mcp/src/index.ts"],
      "env": {
        "ANTHROPIC_API_KEY": "your_api_key_here"
      }
    }
  }
}

Programmatic Usage Examples

// Example using @modelcontextprotocol/sdk
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

const client = new Client({
  name: "spider-client",
  version: "1.0.0"
}, {
  capabilities: {}
});

// Connect to the spider MCP server
const transport = new StdioClientTransport({
  command: "bun",
  args: ["run", "/path/to/spider-mcp/src/index.ts"]
});

await client.connect(transport);

// Extract code examples from documentation
const result = await client.request({
  method: "tools/call",
  params: {
    name: "spider_docs",
    arguments: {
      url: "https://docs.anthropic.com/en/docs/quickstart",
      enable_llm_analysis: true,
      llm_analysis_type: "code_examples",
      max_depth: 2
    }
  }
});

Available Tools

spider_docs

Crawl a documentation website and cache the content with optional LLM analysis.

Parameters:

• url (required): Base URL of the documentation site
• max_depth (optional): Maximum crawl depth (default: 3)
• include_patterns (optional): URL patterns to include
• exclude_patterns (optional): URL patterns to exclude
• enable_llm_analysis (optional): Enable LLM-powered content analysis
• llm_analysis_type (optional): Type of analysis:
  • full - Complete analysis with summary, key points, links, and code examples
  • summary - Content summarization only
  • links - Link analysis and relevance ranking
  • classification - Content type classification
  • code_examples - Extract and categorize code examples only

Example with code extraction:

{
  "url": "https://docs.example.com/api",
  "max_depth": 2,
  "enable_llm_analysis": true,
  "llm_analysis_type": "code_examples"
}

get_page

Retrieve a specific page from the cache.

Parameters:

• url (required): URL of the page to retrieve

search_docs

Search through cached documentation content.

Parameters:

• query (required): Search query
• limit (optional): Maximum results to return (default: 10)

list_pages

List all cached pages with optional filtering and sorting.

Parameters:

• filter (optional): Filter pages by URL pattern
• sort (optional): Sort field (url, title, timestamp)
• order (optional): Sort order (asc, desc)

clear_cache

Clear cached pages matching a pattern.

Parameters:

• url_pattern (optional): Pattern to match for clearing

analyze_content

Perform LLM-powered analysis on a specific cached page.

Parameters:

• url (required): URL of the page to analyze
• analysis_type (optional): Same options as spider_docs above

Example for code extraction:

{
  "url": "https://docs.example.com/api/users",
  "analysis_type": "code_examples"
}

get_summary

Get an intelligent summary of a cached page.

Parameters:

• url (required): URL of the page to summarize
• summary_length (optional): Length of summary (short, medium, long)
• focus_areas (optional): Specific areas to focus on in the summary

Configuration

Configuration can be provided through:

1. Environment variables (see .env.example)
2. JSON configuration files in config/ directory
3. Runtime parameters passed to tools

LLM Integration

To enable LLM-powered content analysis, set your Anthropic API key:

export ANTHROPIC_API_KEY=your_api_key_here

The server will automatically detect the API key and enable LLM features. Without an API key, the server operates in basic mode with programmatic content extraction only.

Example Configuration

{
  "maxDepth": 3,
  "maxPages": 100,
  "concurrency": 5,
  "userAgent": "SpiderMCP/1.0 Documentation Crawler",
  "timeout": 10000,
  "retryAttempts": 3,
  "cacheTTL": 86400000,
  "respectRobotsTxt": true,
  "includePatterns": ["/docs/*", "/api/*"],
  "excludePatterns": ["/blog/*", "*.pdf"]
}

Development

bun test
bun run typecheck
bun run build

Testing the Spider

Use the included test script to try out the functionality locally:

# Test with code example extraction on a documentation site
bun run test-spider.ts https://docs.anthropic.com/en/docs/quickstart

# Test with a specific site focusing on code examples
ANTHROPIC_API_KEY=your_key bun run test-spider.ts https://docs.example.com

# Test code-only extraction
bun run test-spider.ts https://docs.python.org/3/tutorial/

The test script demonstrates the spider functionality outside of MCP and will show you:

• Crawled pages with metadata
• Extracted code examples with languages and categories
• LLM analysis results including summaries and classifications
• Cache statistics and performance metrics

Note: The test script uses the spider functionality directly, not through MCP protocol.

Architecture

• src/spider/ - Core crawling and parsing logic
• src/mcp/ - MCP server implementation and tool handlers
• src/extractors/ - Content extraction strategies
• src/llm/ - LLM integration and content analysis
• src/utils/ - Utility functions and configuration
• cache/ - File system cache storage
• config/ - Configuration files

Extraction Methods

The server supports multiple content extraction methods:

1. Readability - Mozilla Readability algorithm for article extraction
2. Cheerio - Custom CSS selector-based extraction
3. Markdown - HTML to Markdown conversion with Turndown
4. LLM Analysis - Anthropic Claude for intelligent content understanding and code extraction

Code Example Extraction

The LLM analysis can intelligently extract and categorize code examples from documentation:

• Language Detection: Automatically detects programming languages (JavaScript, Python, bash, JSON, etc.)
• Code Categories:
  • api_call - API requests, SDK calls, HTTP requests
  • configuration - Config files, settings, environment setup
  • implementation - Complete functions, classes, modules
  • usage_example - How to use a feature or library
  • snippet - Small code fragments or utilities
  • complete_example - Full working examples or applications
• Format Preservation: Maintains exact formatting, indentation, and syntax
• Context Awareness: Provides meaningful descriptions for each code example

Example Output Format

When using code example extraction, the output includes structured code examples:

{
  "codeExamples": [
    {
      "language": "javascript",
      "code": "const response = await fetch('/api/users', {\n  method: 'GET',\n  headers: {\n    'Authorization': 'Bearer ' + apiKey\n  }\n});",
      "description": "Fetch users from API with authentication",
      "category": "api_call"
    },
    {
      "language": "python",
      "code": "import requests\n\nresponse = requests.get('/api/users', headers={\n    'Authorization': f'Bearer {api_key}'\n})",
      "description": "Python example for API authentication",
      "category": "api_call"
    },
    {
      "language": "json",
      "code": "{\n  \"database\": {\n    \"host\": \"localhost\",\n    \"port\": 5432\n  }\n}",
      "description": "Database configuration file",
      "category": "configuration"
    }
  ]
}

Caching

• Two-tier caching: in-memory + file system
• Configurable TTL and cache size limits
• Automatic cleanup of expired entries
• Domain-based cache organization

Robots.txt Compliance

• Fetches and parses robots.txt automatically
• Respects crawl-delay directives
• Honors disallow/allow rules
• Discovers sitemaps

Rate Limiting

• Configurable concurrent request limits
• Exponential backoff for failed requests
• Per-domain crawl delays
• Timeout handling

License

MIT