api-docs-mcp

An MCP (Model Context Protocol) server that gives AI coding assistants access to up-to-date API documentation via Retrieval-Augmented Generation (RAG). Crawl any documentation site, index it into a vector store, and query it semantically — so your LLM always has accurate, current docs at its fingertips.

Problem

LLMs have stale or incomplete knowledge of specific APIs and libraries. This project solves that by letting you:

1. Crawl any documentation website
2. Index the content into a local vector database
3. Query it semantically through an MCP server

AI assistants (like GitHub Copilot, Claude Desktop, Cursor, etc.) can then retrieve precise, current API documentation on demand.

How It Works

┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Crawling   │ ──▶│   Chunking   │ ──▶│  Embedding   │ ──▶│   Storage    │
│  (crawl4ai)  │     │ (headings +  │     │ (sentence-   │     │  (ChromaDB)  │
│              │     │  paragraphs) │     │ transformers)│     │              │
└──────────────┘     └──────────────┘     └──────────────┘     └──────────────┘
                                                                   ▲
┌──────────────┐     ┌──────────────┐                              │
│   MCP Client │ ──▶│  Retrieval   │ ──▶  Similarity Search ─────┘
│  (Copilot,   │     │   Engine     │
│   Claude,    │     │              │
│   Cursor…)   │     └──────────────┘
└──────────────┘

Pipeline Stages

1. Crawling — Uses crawl4ai (headless browser) to recursively crawl a documentation site from a root URL, following links within the same domain/path prefix. Extracts clean markdown from each page.

2. Chunking — Splits markdown on h1/h2/h3 headings to preserve logical structure. Builds breadcrumb heading paths (e.g., std > HashMap > insert). Sub-splits oversized sections at paragraph boundaries with overlap. Tags chunks with has_code=true if they contain fenced code blocks.

3. Embedding & Storage — Embeds chunks using all-MiniLM-L6-v2 (384-dim vectors via sentence-transformers), then upserts them into ChromaDB collections (one collection per programming language).

4. Retrieval — Embeds the user's query, performs filtered similarity search in ChromaDB, and returns formatted results with source URLs and context.

Incremental Updates

Re-running an ingestion for the same library is efficient:

• Pages are hashed (MD5) and compared against stored hashes
• Unchanged pages are skipped entirely
• Pages removed from the site have their chunks automatically deleted

Installation

Prerequisites

• Python 3.10+
• uv package manager

Setup

git clone https://github.com/akshaysadanand/api-docs-mcp.git
cd api-docs-mcp
uv sync

Usage

CLI

The project ships with a api-docs-mcp command-line tool for managing documentation indexes.

Index a Documentation Site

# Index Rust standard library docs
uv run api-docs-mcp add "https://doc.rust-lang.org/std/" --language rust --library std

# Index Python docs with custom limits
uv run api-docs-mcp add "https://docs.python.org/3/library/" \
  --language python --library stdlib --max-pages 100 --max-depth 2

# Index with a specific version
uv run api-docs-mcp add "https://numpy.org/doc/stable/reference/" \
  --language python --library numpy --version 1.26

Search Indexed Documentation

# Search across all indexed docs for a language
uv run api-docs-mcp search "how to parse JSON" --language python

# Search within a specific library
uv run api-docs-mcp search "HashMap insert or update" --language rust --library std

# Get more results (default: 5)
uv run api-docs-mcp search "async await coroutine" --language kotlin -k 10

List Indexed Sources

# List all indexed sources
uv run api-docs-mcp list

# Filter by language
uv run api-docs-mcp list --language rust

Remove Indexed Documentation

uv run api-docs-mcp remove --language rust --library tokio

CLI Reference

Command	Description	Key Arguments
add	Crawl and index a documentation URL	URL, -l/--language, -L/--library, -v/--version, --max-pages, --max-depth
search	Search indexed docs semantically	QUERY, -l/--language, -L/--library, -k/--top-k
list	List all indexed sources	-l/--language (optional filter)
remove	Remove indexed docs for a library	-l/--language, -L/--library

MCP Server

Configure the MCP server in your AI assistant's settings to enable documentation search from within your editor.

VS Code (GitHub Copilot)

Add to your .vscode/mcp.json or user MCP settings:

{
  "inputs": [],
  "servers": {
    "api-docs-mcp": {
      "command": "bash",
      "args": ["<path-to-project>/start.sh"]
    }
  }
}

Or start the server directly:

uv run api-docs-mcp serve

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "api-docs-mcp": {
      "command": "uv",
      "args": ["run", "api-docs-mcp", "serve"],
      "cwd": "<path-to-project>"
    }
  }
}

Available MCP Tools

Tool	Description	Parameters
search_documentation	Semantic search across indexed docs	query, language, library (optional), top_k (default: 5)
add_documentation	Crawl and index a new doc site	url, language, library, version, max_pages, max_depth
list_sources	List all indexed sources with statistics	language (optional filter)
get_code_examples	Search specifically for code snippets	query, language, library (optional), top_k
lookup_api_symbol	Exact-match symbol lookup (e.g., HashMap::insert)	symbol, language, library (optional)

Storage

• Database: ChromaDB (persistent, disk-based)
• Default location: ~/.local/share/api-docs-mcp/ (follows XDG_DATA_HOME)
• Structure: One ChromaDB collection per programming language
• Chunk metadata: library, version, source_url, page_hash, heading_path, has_code, chunk_index
• Embeddings: 384-dimensional normalized vectors from all-MiniLM-L6-v2 (~80MB model)

Configuration

Option	Default	Description
--max-pages	200	Maximum pages to crawl per run
--max-depth	3	Maximum link depth from start URL
--version	latest	Documentation version string
--top-k	5	Number of search results (max: 20)
Embedding batch size	32	Chunks processed per embedding batch
Upsert batch size	5000	Chunks upserted to ChromaDB per batch

Project Structure

api-docs-mcp/
├── pyproject.toml              # Project metadata & dependencies
├── start.sh                    # MCP server startup script
├── api_docs_mcp/
│   ├── cli.py                  # Click-based CLI commands
│   ├── server.py               # MCP server (stdio transport)
│   ├── embeddings/
│   │   └── model.py            # Sentence-transformer embedding model
│   ├── ingestion/
│   │   ├── crawler.py          # Headless browser web crawler
│   │   ├── chunker.py          # Markdown-aware document chunking
│   │   └── processor.py        # Orchestration & incremental updates
│   ├── retrieval/
│   │   └── engine.py           # Semantic search engine
│   └── storage/
│       └── vector_store.py     # ChromaDB vector store wrapper
├── tests/
│   ├── test_vector_store.py
│   └── test_get_metadata.py
└── docs/
    └── implementation_plan.md

Dependencies

Package	Purpose
mcp	MCP server framework (stdio transport)
click	CLI framework
chromadb	Vector database for persistent storage
sentence-transformers	Text embedding model (all-MiniLM-L6-v2)
crawl4ai	Headless browser web crawler with markdown extraction

Examples

Indexing Multiple Libraries

# Rust ecosystem
uv run api-docs-mcp add "https://doc.rust-lang.org/std/" -l rust -L std
uv run api-docs-mcp add "https://docs.rs/tokio/latest/tokio/" -l rust -L tokio

# Python ecosystem
uv run api-docs-mcp add "https://docs.python.org/3/library/" -l python -L stdlib --max-pages 150
uv run api-docs-mcp add "https://numpy.org/doc/stable/reference/" -l python -L numpy

# Kotlin
uv run api-docs-mcp add "https://kotlinlang.org/docs/home.html" -l kotlin -L standard --max-pages 200

Typical Workflow

1. Index the documentation you work with most frequently
2. Configure the MCP server in your editor
3. Ask your AI assistant questions like:
   • "How do I use HashMap::entry in Rust?"
   • "Show me examples of pandas DataFrame filtering"
   • "What's the Kotlin coroutine flow API?"

License

This project is open-sourced under the MIT License