Better Fetch - Advanced Web Content MCP Server

A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.

🚀 Key Features

🕸️ Smart Web Crawling

• Nested URL Fetching: Automatically discovers and crawls linked pages up to configurable depth
• Single Page Mode: Option for simple single-page content extraction
• Domain Filtering: Stay within the same domain or allow cross-domain crawling
• Pattern Matching: Include/exclude URLs based on regex patterns

🧠 Intelligent Content Processing

• Content Cleaning: Removes ads, navigation, scripts, and other noise automatically
• Smart Section Detection: Identifies main content areas (<main>, <article>, .content)
• Automatic Titles: Generates meaningful section headers based on page titles and URL structure
• Table of Contents: Creates organized TOC with proper nesting

📝 Advanced Markdown Generation

• Clean Formatting: Converts HTML to well-structured markdown
• Code Block Preservation: Maintains formatting for code snippets and technical content
• Link Preservation: Keeps all important links with proper markdown syntax
• Metadata Integration: Includes source URLs, generation timestamps, and site information

⚙️ Highly Configurable

• Crawl Depth Control: Set maximum levels to crawl (default: 2)
• Page Limits: Control maximum pages to process (default: 50)
• Timeout Settings: Configurable request timeouts
• Respectful Crawling: Built-in delays between requests
• Error Handling: Graceful handling of failed requests and invalid URLs

📋 Available Tools

1. fetch_website_nested

Comprehensive web crawling with nested URL processing.

Parameters:

• url (required): Starting URL to crawl
• maxDepth (optional, default: 2): Maximum crawl depth
• maxPages (optional, default: 50): Maximum pages to process
• sameDomainOnly (optional, default: true): Restrict to same domain
• excludePatterns (optional): Array of regex patterns to exclude
• includePatterns (optional): Array of regex patterns to include
• timeout (optional, default: 10000): Request timeout in milliseconds

2. fetch_website_single

Simple single-page content extraction.

Parameters:

• url (required): URL to fetch
• timeout (optional, default: 10000): Request timeout in milliseconds

💡 Use Cases

📚 Documentation Processing

• API Documentation: Convert REST API docs, SDK guides, and technical references
• Framework Docs: Process React, Vue, Angular, or any framework documentation
• Library Guides: Extract comprehensive guides from library documentation sites
• Tutorial Series: Gather multi-part tutorials into single organized documents

🔍 Content Analysis & Research

• Competitive Analysis: Gather competitor documentation and feature descriptions
• Market Research: Extract product information from multiple related pages
• Academic Research: Collect and organize web-based research materials
• Knowledge Base Creation: Transform scattered web content into structured knowledge bases

🤖 AI Training & Context

• LLM Context Preparation: Create clean, structured content for AI model training
• RAG System Input: Generate high-quality documents for Retrieval-Augmented Generation
• Chatbot Knowledge: Build comprehensive knowledge bases for customer service bots
• Content Summarization: Prepare web content for automated summarization tasks

🛠️ Installation & Setup

Installing via Smithery

To install Better Fetch for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @flutterninja9/better-fetch --client claude

Prerequisites

• Node.js 18+
• npm or yarn
• MCP-compatible client (Claude Desktop, VS Code with MCP extension, etc.)

Step 1: Clone and Install

git clone https://github.com/yourusername/better-fetch.git
cd better-fetch
npm install

Step 2: Build the Project

npm run build

Step 3: Test the Server (Optional)

# Quick test
npm run dev

# Or run comprehensive tests
node test-mcp.js

Step 4: Configure Your MCP Client

For Claude Desktop:

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "better-fetch": {
      "command": "node",
      "args": ["/absolute/path/to/better-fetch/dist/server.js"],
      "env": {
        "NODE_ENV": "production"
      }
    }
  }
}

For VS Code MCP Extension:

{
  "better-fetch": {
    "command": "node",
    "args": ["/Users/yourusername/better-fetch/dist/server.js"]
  }
}

For Custom MCP Client:

{
  "name": "better-fetch",
  "command": "node",
  "args": ["/path/to/better-fetch/dist/server.js"],
  "stdio": true
}

📖 Usage Examples

Basic Documentation Crawling

Fetch all the web contents from this Flutter Shadcn UI documentation site:
https://flutter-shadcn-ui.mariuti.com/

Use nested fetching with a maximum depth of 3 levels and process up to 100 pages.

Advanced Configuration

Fetch content from the React documentation but exclude any URLs containing 'api' or 'reference' and only process pages containing 'tutorial' or 'guide':

URL: https://react.dev
Max Depth: 2
Exclude Patterns: ["/api/", "/reference/"]
Include Patterns: ["/tutorial/", "/guide/"]
Max Pages: 30

Single Page Extraction

Extract the content from this specific page only:
https://nextjs.org/docs/getting-started/installation

Use single page mode to avoid crawling related links.

📄 Sample Output

The server generates comprehensive markdown files with the following structure:

# Site Name Documentation

*Scraped from: https://example.com*
*Generated on: 2024-01-15T10:30:00.000Z*

## Table of Contents

- [Getting Started](#getting-started)
  - [Installation](#installation)
  - [Quick Start](#quick-start)
- [API Reference](#api-reference)
  - [Core Functions](#core-functions)

---

## Getting Started

*Source: [https://example.com/getting-started](https://example.com/getting-started)*

[Clean markdown content here...]

---

## Installation

*Source: [https://example.com/installation](https://example.com/installation)*

[Installation instructions in markdown...]

For a complete example, refer to output.md which demonstrates the server's output when processing a real documentation site.

🔧 Development

Project Structure

better-fetch/
├── src/
│   └── server.ts          # Main server implementation
├── dist/                 # Compiled JavaScript
├── test-mcp.js          # Testing utilities
├── output.md            # Sample output file
├── package.json
├── tsconfig.json
└── README.md

Available Scripts

npm run dev          # Run in development mode with hot reload
npm run build        # Compile TypeScript to JavaScript
npm run start        # Run the compiled server
npm run clean        # Clean dist directory
npm test             # Run test suite

Testing Your Changes

# Interactive testing
node interactive-test.js

# Automated test suite
node test-mcp.js

# Manual JSON-RPC testing
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node dist/index.js

🚦 Performance & Limits

Default Limits

• Max Depth: 2 levels (configurable)
• Max Pages: 50 pages (configurable)
• Request Timeout: 10 seconds (configurable)
• Crawl Delay: 500ms between requests (respectful crawling)

Performance Tips

• Set appropriate maxPages limits for large sites
• Use includePatterns to focus on relevant content
• Enable sameDomainOnly to avoid external link crawling
• Adjust timeout based on target site response times

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests
4. Commit your changes: git commit -m 'Add amazing feature'
5. Push to the branch: git push origin feature/amazing-feature
6. Open a Pull Request

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support & Issues

• Bug Reports: GitHub Issues
• Feature Requests: GitHub Discussions
• Documentation: Check the Wiki

🙏 Acknowledgments

• Built with the Model Context Protocol SDK
• Powered by Cheerio for HTML parsing
• Markdown conversion by Turndown

---

Made with ❤️ for the AI and developer community