SpiderFoot MCP Server

A Model Context Protocol (MCP) server that provides SpiderFoot scanning capabilities through a standardized interface.

Overview

This MCP server allows you to interact with SpiderFoot's OSINT scanning capabilities through Claude and other MCP-compatible tools. It provides comprehensive scan management, result retrieval, and export functionality.

Features

• Scan Management: Start, stop, delete, and monitor SpiderFoot scans
• Real-time Status: Get scan status, progress, and completion notifications
• Result Access: Retrieve scan results, summaries, and logs
• Export Capabilities: Export scan data in JSON, CSV, and Excel formats
• Search Functionality: Search across scan results
• Server Health: Ping server and check connectivity/version
• Module Management: Access available SpiderFoot modules

Prerequisites

• Python 3.8+
• Access to a SpiderFoot server (local or remote)
• Required Python packages (see Installation)

Installation

1. Install required dependencies:

pip install requests python-dotenv mcp

2. Set up environment variables in .env:

SPIDERFOOT_URL=https://your-spiderfoot-server.com
SPIDERFOOT_USERNAME=your-username
SPIDERFOOT_PASSWORD=your-password

Configuration

The server expects these environment variables:

• SPIDERFOOT_URL: Base URL of your SpiderFoot instance (default: http://localhost:5001)
• SPIDERFOOT_USERNAME: Username for HTTP digest authentication (default: admin)
• SPIDERFOOT_PASSWORD: Password for authentication (required)

Available MCP Tools

Core Scan Operations

• start_scan(target, scan_name, modules?, use_case?) - Start a new scan
• get_scan_status(scan_id) - Get current scan status
• list_scans() - List all scans on the server
• stop_scan(scan_id) - Stop a running scan
• delete_scan(scan_id) - Delete a scan and its data

Results and Analysis

• get_scan_results(scan_id, event_type?) - Get scan results
• get_scan_summary(scan_id, by?) - Get scan summary by module/type
• get_scan_log(scan_id, limit?, from_rowid?) - Get scan log entries
• export_scan_results(scan_id, export_format?) - Export results (JSON/CSV/Excel)
• search_scan_results(query, scan_id?) - Search across results

Utility Functions

• ping() - Test server connectivity and get version
• get_available_modules() - List available SpiderFoot modules
• wait_for_scan_completion(scan_id, poll_interval?, timeout?) - Wait for scan completion
• get_active_scans_summary() - Get summary of tracked scans

Usage Examples

Starting a Scan

# Passive scan with default modules
start_scan("example.com", "example-scan", use_case="passive")

# Custom scan with specific modules
start_scan("example.com", "custom-scan", modules=["sfp_dnsresolve", "sfp_dnscommonsrv"])

Monitoring Progress

# Check status
status = get_scan_status("scan-id")

# Wait for completion
wait_for_scan_completion("scan-id", poll_interval=5, timeout=300)

Retrieving Results

# Get all results
results = get_scan_results("scan-id")

# Get summary by module
summary = get_scan_summary("scan-id", by="module")

# Export to JSON
export_data = export_scan_results("scan-id", "json")

Running the Server

Development/Testing

python test_client.py  # Test client functionality
python server.py       # Run MCP server

Production

The server automatically validates environment variables and tests connectivity on startup.

API Implementation Details

Authentication

Uses HTTP Digest Authentication as required by SpiderFoot API v4.0.

Response Handling

The implementation properly handles SpiderFoot's unique response formats:

• List responses: ['SUCCESS', 'data'] or ['ERROR', 'message']
• Scan data: Arrays with positional fields [id, name, target, created, started, completed, status, ...]
• JSON responses: Standard dictionaries for modules and complex data

Key Fixes Applied

1. JSON Accept Header: Required for JSON responses instead of HTML
2. Module Specification: Even with use cases, specific modules must be provided
3. Parameter Names: Correct parameter names for each endpoint (ids vs id, by parameter for summaries)
4. Response Format Handling: Proper parsing of list-based responses

Supported SpiderFoot Versions

• Primary: SpiderFoot v4.0.0
• Compatibility: Should work with SpiderFoot v4.x series

Use Cases

Passive Reconnaissance

start_scan("target-domain.com", "recon-scan", use_case="passive")

Investigation

start_scan("suspicious-domain.com", "investigation", use_case="investigate") 

Footprinting

start_scan("company-domain.com", "footprint", use_case="footprint")

Error Handling

The server provides comprehensive error handling with detailed messages:

• Connection failures
• Authentication errors
• Invalid scan parameters
• API endpoint errors
• Response parsing issues

Security Considerations

• Credentials are loaded from environment variables
• HTTP Digest Authentication for API security
• No secrets logged or exposed in responses
• Secure handling of scan data

Troubleshooting

Common Issues

1. 404 Errors: Usually indicate incorrect endpoint or missing parameters
2. Authentication Failures: Check username/password and server accessibility
3. Module Errors: Ensure modules are specified even with use cases
4. Connection Issues: Verify server URL and network connectivity

Testing Connectivity

ping_result = ping()
# Check ping_result['success'] and ping_result['server_version']

Debug Logging

Enable debug logging in the client for detailed API call information.

Development Notes

• The implementation follows SpiderFoot's official sfcli patterns
• All endpoints tested against live SpiderFoot v4.0.0 instance
• MCP tools provide structured responses with success/error handling
• Maintains compatibility with both single and batch operations

Contributing

When extending functionality:

1. Follow existing error handling patterns
2. Handle both list and dict response formats
3. Test against actual SpiderFoot instance
4. Update documentation for new tools

License

This project follows the same licensing as SpiderFoot for compatibility.