mcp-rag-service

A production-minded Retrieval-Augmented Generation (RAG) service you can plug into any LLM through the Model Context Protocol (MCP) — with PII redaction guardrails and a retrieval-first evaluation harness, packaged for Docker and Kubernetes.

I built this because I wanted a single, safe "answer questions over my documents" tool that any MCP-aware assistant (Claude, an IDE agent, my own scripts) could call — without leaking personal data and without me having to trust the answers blindly. So it answers only from retrieved passages, cites its sources, redacts PII before anything touches the model, and ships with an eval script that measures retrieval quality rather than vibes.

Why it exists (the real problem)

An LLM on its own hallucinates and can't see your private documents. Wrapping retrieval around it fixes both — but three things are usually missing from demo RAG:

1. It isn't callable from other tools. Here it's exposed as an MCP server, so any MCP client can use it as a first-class tool.
2. It leaks PII. Emails, phone numbers, IBANs and card-like numbers are redacted before retrieval and before the prompt (asymmetric-cost thinking: a redaction is cheap, an accidental leak into a model/log is expensive).
3. Nobody measures it. A hit@k + grounded-answer eval harness makes quality a number, not a feeling. Most "RAG hallucination" is really a retrieval miss, so I measure retrieval first.

Architecture

MCP client (Claude / agent)
        │  tool: rag_query(question)
        ▼
   MCP server  (src/mcp_rag/server.py)
        │
   RAG pipeline (rag.py)
        │  1. redact PII        (guardrails.py)
        │  2. hybrid retrieve   (retriever.py: dense + BM25, score-fused)
        │  3. generate grounded, cited answer  (LLM, or extractive fallback)
        ▼
   {answer, citations[], redactions[]}

• Hybrid retrieval: dense embeddings (meaning) fused with BM25 (exact terms like IDs and names). Runs fully offline with a hashing embedder + pure-Python BM25 fallback, so there is no API key required for development — the paid LLM is used only for the final wording.
• Guardrails: guardrails.py redacts PII with well-tested patterns (and optionally Microsoft Presidio if installed).
• Grounding: answers are built only from retrieved chunks and carry [n] citations; if nothing relevant is retrieved it says so instead of guessing.

Run it

pip install -r requirements.txt
python -m mcp_rag.ingest data/            # index your documents
python -m mcp_rag.server                  # start the MCP server (stdio)

Query it directly (no MCP client needed):

python -m mcp_rag.rag "What are the payment terms in the Nordwind contract?"

Evaluate retrieval quality:

python eval/run_eval.py                   # prints hit@k and grounded-answer rate

Deploy (Docker + Kubernetes)

docker build -t mcp-rag-service -f deploy/Dockerfile .
kubectl apply -f deploy/k8s.yaml          # Deployment + Service, readiness/liveness probes

Tech

Python · MCP (FastMCP) · FAISS / hashing-embedder fallback · BM25 · FastAPI (HTTP mode) · Presidio-optional PII redaction · pytest · Docker · Kubernetes.

Status

Working core (retrieval, guardrails, MCP tool, eval) + container/K8s manifests. Corpus in data/ is a small sample; point ingest at your own documents to use it for real.