MCP Docs Server

A Model Context Protocol (MCP) server that fetches and parses documentation from llms.txt files, enabling AI assistants to access and query documentation dynamically.

Overview

This MCP server provides tools for AI assistants to:

1. Discover available documentation sources via llms.txt files
2. Fetch and parse documentation content from URLs or local files
3. Convert HTML documentation to markdown for easier consumption

The server follows the llms.txt convention, which provides a standardized way for websites to expose their documentation in an LLM-friendly format.

How It Works

Architecture

The server implements the Model Context Protocol (MCP) to expose two main tools:

1. list_doc_sources: Lists all configured documentation sources with their llms.txt URLs or file paths
2. fetch_docs: Fetches and parses documentation content from a given URL or local file path

Workflow

1. Discovery: AI assistant calls list_doc_sources to see available documentation
2. Parse llms.txt: AI assistant fetches the llms.txt file using fetch_docs to see the documentation structure
3. Query Documentation: AI assistant analyzes the URLs in llms.txt and fetches specific relevant pages
4. Content Processing: Server converts HTML to markdown and returns cleaned content

llms.txt Format

The llms.txt file is a simple text file that lists URLs to documentation pages. The server supports both absolute and relative URLs.

Absolute URLs:

# Stripe API Documentation
https://docs.stripe.com/api/authentication
https://docs.stripe.com/api/charges
https://docs.stripe.com/api/customers

Relative URLs (when only one documentation source is configured):

# Stripe API Documentation
/api/authentication
/api/charges
/api/customers

Note: Relative URLs are automatically resolved against the base domain of the llms.txt file. This only works when a single documentation source is configured. With multiple sources, you must use absolute URLs to avoid ambiguity.

Features

• Domain Whitelisting: Only allows fetching from pre-configured trusted domains
• Local File Support: Can serve documentation from local markdown/text files
• Relative URL Support: Automatically resolves relative URLs in llms.txt files against the base domain
• HTML to Markdown Conversion: Automatically converts HTML documentation to markdown
• Security: Built-in protections against unauthorized URL access
• Configurable Timeouts: Customizable request timeouts
• Redirect Handling: Optional support for following HTTP redirects

Installation

npm install

Configuration

Edit the DOC_SOURCES array in src/index.ts to configure your documentation sources:

const DOC_SOURCES: DocSource[] = [
  {
    name: "Stripe API Documentation",
    llms_txt: "https://docs.stripe.com/llms.txt",
    description: "Stripe API documentation",
  },
  {
    name: "Local Docs",
    llms_txt: "file:///path/to/local/llms.txt",
    description: "Local documentation files",
  },
];

Configuration Options

• name (optional): Display name for the documentation source
• llms_txt: URL or file path to the llms.txt file
• description (optional): Description of the documentation

Advanced Configuration

Modify the CONFIG object for additional settings:

const CONFIG = {
  followRedirects: false,  // Follow HTTP redirects
  timeout: 10000,          // Request timeout in milliseconds
  allowedDomains: [],      // Additional allowed domains (use ["*"] to allow all)
};

Usage

Build

npm run build

Development

npm run dev

Run

npm start

The server runs on stdio and communicates via the Model Context Protocol.

MCP Integration

To use this server with an MCP-compatible client (like Claude Desktop), add it to your MCP configuration:

{
  "mcpServers": {
    "mcp-docs": {
      "command": "node",
      "args": ["/path/to/mcp-docs/build/index.js"]
    }
  }
}

Project Structure

mcp-docs/
├── src/
│   ├── index.ts      # Main server implementation
│   ├── utils.ts      # Utility functions (fetching, parsing, conversion)
│   └── types.ts      # TypeScript type definitions
├── build/            # Compiled JavaScript (gitignored)
├── package.json      # Project dependencies
├── tsconfig.json     # TypeScript configuration
└── README.md         # This file

Dependencies

• @modelcontextprotocol/sdk: MCP server SDK
• node-html-parser: HTML parsing
• turndown: HTML to Markdown conversion
• zod: Schema validation

Security

The server implements several security measures:

• Domain Whitelisting: Only domains with configured llms.txt files can be accessed
• Path Validation: Local files must be explicitly allowed in configuration
• No Arbitrary Fetching: Cannot fetch random URLs without proper configuration

Example Use Case

An AI assistant helping a developer with Stripe API integration can:

1. Call list_doc_sources to discover Stripe docs are available
2. Fetch https://docs.stripe.com/llms.txt to see all available documentation pages
3. Fetch specific pages like https://docs.stripe.com/api/charges to answer user questions
4. Provide accurate, up-to-date answers based on official documentation

License

MIT