Crawl4AI RAG MCP Server

A Retrieval-Augmented Generation (RAG) MCP server built with Python that enables AI assistants to crawl, index, and retrieve information from technical documentation using semantic search.

The server integrates Crawl4AI for web crawling, Supabase (pgvector) for vector storage, OpenAI embeddings for semantic retrieval, and Neo4j for repository knowledge graph validation.

---

Features

• Model Context Protocol (MCP) server
• Intelligent web crawling with Crawl4AI
• Recursive website indexing
• Automatic document chunking
• OpenAI embedding generation
• Supabase pgvector vector database
• Semantic document retrieval
• Optional Hybrid Search
• Optional Contextual Embeddings
• Optional Agentic RAG
• Optional Cross-Encoder Reranking
• Neo4j Knowledge Graph integration
• AI code hallucination detection
• Repository structure validation
• Docker support
• SSE & STDIO transport support

---

Tech Stack

Category	Technologies
Language	Python
AI	OpenAI API
Protocol	Model Context Protocol (MCP)
Web Crawling	Crawl4AI
Vector Database	Supabase + pgvector
Knowledge Graph	Neo4j
Containerization	Docker
Retrieval	RAG

---

Architecture

                    AI Assistant
                         │
                         ▼
                  MCP Server (Python)
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
   Crawl4AI         Knowledge Graph     RAG Pipeline
        │              (Neo4j)             │
        ▼                                  ▼
 Crawl Websites                    Document Chunking
                                          │
                                          ▼
                                 OpenAI Embeddings
                                          │
                                          ▼
                              Supabase (pgvector)
                                          │
                                          ▼
                                  Semantic Search
                                          │
                                          ▼
                                  Generated Response

---

MCP Tools

Crawling

• Crawl a single page
• Crawl complete documentation websites
• Recursive crawling
• Sitemap crawling

Retrieval

• Semantic RAG search
• Source filtering
• Hybrid retrieval
• Context-aware retrieval

Knowledge Graph

• Parse GitHub repositories
• Validate AI-generated Python code
• Detect hallucinated imports
• Detect invalid methods/classes
• Query repository graph

---

Project Structure

src/
├── crawl4ai_mcp.py
├── tools/
├── knowledge_graphs/
├── rag/
├── utils/

public/

docker/

.env
README.md

---

Installation

Clone the repository

git clone <repository-url>
cd crawl4ai-rag-mcp-server

Install dependencies

pip install -r requirements.txt

---

Environment Variables

Create a .env file.

OPENAI_API_KEY=

SUPABASE_URL=
SUPABASE_SERVICE_KEY=

NEO4J_URI=
NEO4J_USER=
NEO4J_PASSWORD=

---

Running

Using Python

python src/crawl4ai_mcp.py

Using Docker

docker build -t crawl4ai-rag .
docker run --env-file .env -p 8051:8051 crawl4ai-rag

---

Retrieval Pipeline

1. Crawl technical documentation
2. Clean extracted content
3. Split documents into chunks
4. Generate embeddings
5. Store vectors in pgvector
6. Perform semantic similarity search
7. Inject retrieved context into prompts
8. Generate grounded responses

---

Knowledge Graph Pipeline

• Parse GitHub repositories
• Extract classes
• Extract methods
• Extract imports
• Build Neo4j graph
• Validate AI-generated code
• Detect hallucinated APIs

---

Future Improvements

• Multiple embedding model support
• Local embedding models
• Incremental indexing
• Authentication
• Citation support
• Multi-user support
• Document upload
• PDF ingestion
• Monitoring & Observability

---