Linked Documentation System

A production-ready RAG system with dual interfaces: REST API + MCP Protocol
Enables web applications and AI assistants to intelligently search and reference documentation using hybrid semantic + keyword search.

🎯 What Is This?

This is a complete RAG (Retrieval Augmented Generation) system that provides two ways to access powerful documentation search:

1. REST API Server (main.py) - FastAPI-based HTTP server for web applications and integrations
2. MCP Server (mcp_server.py) - Model Context Protocol server for AI assistants (Cursor, Claude Desktop)

Both servers share the same hybrid search engine, enabling accurate documentation retrieval whether you're building a web app or empowering an AI assistant.

Key Features

• Intelligent Hybrid Search: Combines semantic understanding (FAISS embeddings) with keyword matching (BM25)
• Smart Ranking: Title/metadata boosting, multi-chunk document expansion, relevance scoring
• Multi-Format Support: PDF, Markdown, and web documentation (via built-in scraper)
• MCP Native: Works seamlessly in Cursor, Claude Desktop, and other MCP-compatible tools
• Enterprise Ready: Access control, audit logging, local-first architecture
• Fast: <200ms search latency, optimized chunking and indexing
• Zero Cost: Runs 100% locally, no API keys or cloud dependencies

🏗️ How It Works

┌─────────────────────────────────────────────────────────┐
│              AI Assistant (Cursor/Claude)               │
└────────────────────┬────────────────────────────────────┘
                     │ MCP Protocol (JSON-RPC over stdio)
                     ▼
┌─────────────────────────────────────────────────────────┐
│                   mcp_server.py                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Tools: search_documentation(), list_sources()  │   │
│  └─────────────────────────────────────────────────┘   │
└────────────────────┬────────────────────────────────────┘
                     │
    ┌────────────────┼────────────────┐
    ▼                ▼                ▼
┌─────────┐   ┌──────────┐   ┌──────────────┐
│ Hybrid  │   │ Access   │   │ Audit Logger │
│ Search  │   │ Control  │   │              │
└────┬────┘   └──────────┘   └──────────────┘
     │
     ├─ Semantic Search (FAISS + embeddings)
     │   • Title/metadata boosting
     │   • Multi-chunk document expansion
     │
     └─ Keyword Search (BM25)
         • Exact term matching
         • Traditional ranking

Project Structure

LinkedDocsMCP/
├── mcp_server.py          # Main MCP server (stdio interface)
├── main.py                # FastAPI server (for testing/debugging)
├── download_docs.py       # Web documentation scraper CLI
├── connectors/            # Document format handlers
│   ├── pdf.py            # PDF extraction
│   └── markdown.py       # Markdown parsing
├── indexing/             # Search engine core
│   ├── chunker.py        # Semantic text chunking
│   ├── embedder.py       # Sentence transformers
│   ├── vector_store.py   # FAISS vector database
│   ├── keyword_search.py # BM25 implementation
│   └── hybrid_search.py  # Combined search with boosting
├── schemas/              # Data models
│   ├── config.py         # Settings & configuration
│   └── document.py       # Document schemas
├── core/                 # Cross-cutting concerns
│   ├── access_control.py # Permission system
│   └── audit.py          # Query logging
└── data/                 # Local storage
    ├── sources/          # Your documents (PDF, MD)
    └── vector_store/     # Indexed vectors & metadata

🚀 Quick Start (5 Minutes)

Prerequisites

• Python 3.10+
• Cursor or Claude Desktop (for MCP integration)

1. Install Dependencies

pip install -r requirements.txt

First run: Downloads ~80MB embedding model (one-time)

2. Add Documentation

Option A: Download from web

# Download Factorio wiki (example)
python download_docs.py https://wiki.factorio.com/Tutorials --crawl --max 20

Option B: Add your own files

# Copy PDFs or Markdown files
copy your-docs.pdf data/sources/
copy your-guide.md data/sources/

3. Set Up MCP in Cursor (or other LLM service)

Add to your Cursor MCP config (~/.cursor/mcp.json or C:\Users\<USER>\.cursor\mcp.json):

{
  "mcpServers": {
    "linked-docs": {
      "command": "python",
      "args": ["C:/full/path/to/LinkedDocsMCP/mcp_server.py"]
    }
  }
}

4. Restart Cursor & Use!

In Cursor's chat:

What are the different enemy types in Factorio?

The AI will automatically search your documentation and provide accurate, cited answers! ✨

Key Features Explained

Hybrid Search

Combines two complementary approaches:

Semantic Search (70% weight)

• Uses sentence-transformers (all-MiniLM-L6-v2 model)
• Understands meaning: "authentication setup" matches "configuring auth"
• Converts text to 384-dimensional vectors
• Fast similarity search with FAISS

Keyword Search (30% weight)

• Uses BM25 algorithm (same as Elasticsearch)
• Exact term matching: great for technical terms, code, etc.
• Traditional ranking with document length normalization

Smart Ranking Enhancements:

• Title Boosting: Documents whose titles match the query get 3x boost
• Multi-Chunk Expansion: Returns up to 3 sequential chunks from highly relevant documents
• Document Grouping: Results grouped by source document for better context

Semantic Chunking

Unlike naive character-splitting, this uses smart boundaries:

1. Markdown headers (##, ###) - keeps sections together
2. Paragraph breaks (\n\n) - maintains topical coherence
3. Sentences - fallback for unstructured text

Settings:

• Chunk size: 2048 characters (whole sections, not fragments)
• Overlap: 200 characters (prevents context loss at boundaries)

Web Documentation Scraper

Built-in tool to download and convert web docs:

# Download single page
python download_docs.py https://wiki.example.com/Guide

# Crawl multiple pages (with smart duplicate detection)
python download_docs.py https://wiki.example.com/Main --crawl --max 50

# Force re-download (skip existing detection)
python download_docs.py https://wiki.example.com/Main --crawl --force

# Filter by language
python download_docs.py https://wiki.example.com/Main --crawl --languages en,de

Features:

• Auto-detects and skips existing pages (saves time & bandwidth)
• Respects same-domain and link patterns
• Polite crawling with configurable delays
• Converts HTML to clean Markdown with metadata
• Preserves document structure (headers, lists, tables)

🔒 Security & Access Control

Built-in features:

• 4-tier access hierarchy: PUBLIC → INTERNAL → RESTRICTED → CONFIDENTIAL
• Query-time filtering based on user permissions
• Full audit logging (every query tracked)
• Local-only processing (no data leaves your machine)

Audit logs (data/audit.log):

{
  "timestamp": "2025-10-21T14:30:00Z",
  "event_type": "search",
  "user_id": "mcp_client",
  "query": "enemy types",
  "results_count": 5,
  "search_time_ms": 143
}

⚙️ Configuration

Edit schemas/config.py or set environment variables:

# Search weights
SEMANTIC_WEIGHT = 0.7  # Meaning-based search
KEYWORD_WEIGHT = 0.3   # Exact term matching

# Chunking
CHUNK_SIZE = 1280      # Larger chunks for complete sections
CHUNK_OVERLAP = 128    # Overlap for context continuity

# Model
EMBEDDING_MODEL = "all-MiniLM-L6-v2"  # Fast, accurate, small

🔧 Advanced Usage

REST API (for testing/debugging)

# Start FastAPI server
python main.py

# Search via REST
curl -X POST http://localhost:8000/api/v1/search_docs \
  -H "Content-Type: application/json" \
  -d '{"query": "getting started", "top_k": 5}'

# Interactive API docs
open http://localhost:8000/docs

Programmatic Usage

from indexing.embedder import Embedder
from indexing.vector_store import VectorStore
from indexing.hybrid_search import HybridSearchEngine

# Initialize
embedder = Embedder(model_name="all-MiniLM-L6-v2")
vector_store = VectorStore(embedding_dim=384)
search_engine = HybridSearchEngine(vector_store, keyword_searcher, embedder)

# Search
results = search_engine.search("how to configure auth", top_k=5)
for chunk, score in results:
    print(f"{score:.3f}: {chunk.text[:100]}...")

Technical Highlights

1. Hybrid search outperforms pure semantic or keyword alone
2. Smart chunking preserves document structure
3. Title boosting dramatically improves ranking quality
4. Multi-chunk expansion provides complete context
5. Zero cloud dependencies - privacy-first architecture
6. MCP native - works with any compatible AI assistant

🙏 Acknowledgments

Built with:

• FastAPI - Modern Python web framework
• Sentence Transformers - Semantic embeddings
• FAISS - Vector similarity search
• BM25 (rank-bm25) - Keyword ranking
• MCP - Model Context Protocol by Anthropic

---

Status: ✅ Demo Ready | Version: 1.0.0 | Updated: October 2025