hybrid-rag-mcp

A production-minded Model Context Protocol (MCP) server that gives any LLM agent a high-quality search_docs tool backed by hybrid retrieval (BM25 + dense vectors, fused with Reciprocal Rank Fusion) and cross-encoder reranking — with a real evaluation harness, prompt-injection guardrails, and OpenTelemetry tracing built in.

One tool, exposed over MCP, that is measurably good and demonstrably safe.

---

Documentation

Full documentation lives in docs/. Quick links:

Overview · Installation · Usage	Get started
Architecture · API Reference · Configuration	Reference
Retrieval · Evaluation · Security · Observability	Deep dives
Development · FAQ	Extend & troubleshoot

---

Why this project

Most retrieval demos stop at "embed the query, take cosine top-k." Real systems don't. This repo shows the parts that actually matter in production:

Capability	What it demonstrates
MCP server (FastMCP, stdio transport)	Tool-calling integration any MCP client (Claude Desktop, IDEs, custom agents) can use
Hybrid retrieval (BM25 + dense, RRF fusion)	You understand lexical vs. semantic search and how to combine them
Cross-encoder reranking	You can improve precision@k, not just recall — and measure it
Eval harness (recall@k, MRR)	You prove quality with numbers, before/after each stage
Prompt-injection guardrail	You treat tool inputs as untrusted (OWASP LLM01)
OpenTelemetry tracing	You can debug and observe an agent tool in production

Architecture

flowchart LR
    A[MCP Client] -- search_docs query --> B[FastMCP Server]
    B --> G{Injection guardrail}
    G -- flagged --> R[Reject + reason]
    G -- clean --> P[Retrieval pipeline]
    subgraph P [Retrieval pipeline]
        C[BM25 lexical] --> F[RRF fusion]
        D[Dense vector] --> F
        F --> E[Cross-encoder rerank]
    end
    P --> H[Top-k passages]
    H --> A
    B -. spans .-> T[(OpenTelemetry)]

Quickstart

Requires Python 3.10+ (tested on 3.13).

# 1. Create an isolated environment
python3.13 -m venv .venv && source .venv/bin/activate

# 2. Install the core (BM25 works immediately — no model downloads)
pip install -e .

# 3. Run the tests and the retrieval eval
make test
make eval

# 4. Start the MCP server (stdio)
make run

The server ships with a small sample corpus (src/hybrid_rag_mcp/corpus/sample_docs.jsonl) so everything runs end-to-end on first clone. Swap in your own corpus to make it yours.

Optional: enable dense vectors + reranking

The advanced retrieval stages activate automatically when their (heavier) dependencies are installed; otherwise the pipeline gracefully degrades to BM25-only.

pip install -e ".[full]"   # sentence-transformers, flashrank, opentelemetry

Connect it to an MCP client

Add this to your client's MCP config (example for Claude Desktop claude_desktop_config.json):

{
  "mcpServers": {
    "hybrid-rag": {
      "command": "/absolute/path/to/.venv/bin/python",
      "args": ["-m", "hybrid_rag_mcp.server"]
    }
  }
}

Evaluation

make eval scores retrieval quality on evals/qa_dataset.jsonl and prints a table so you can see the contribution of each stage. Reranking should lift precision — prove it:

Config	Recall@5	MRR
BM25 only	run make eval	.
+ dense (RRF fusion)	.	.
+ cross-encoder rerank	.	.

Fill this table with your real numbers and screenshot it in your write-up. Numbers win interviews.

Security / red-teaming

make redteam runs a battery of prompt-injection payloads (redteam/injection_payloads.jsonl) through the input guardrail and reports how many were caught. Extend the payload set and the detector rules — closing the gap between them is the interesting part.

Observability

Every tool call is wrapped in an OpenTelemetry span (query, stage latencies, result count). By default spans print to the console; point OTEL_EXPORTER_OTLP_ENDPOINT at a collector (Jaeger, Grafana Tempo, Langfuse) to visualize traces.

Repository layout

hybrid-rag-mcp/
├── src/hybrid_rag_mcp/
│   ├── server.py            # FastMCP entrypoint, exposes search_docs
│   ├── pipeline.py          # composes guardrail -> retrieve -> fuse -> rerank
│   ├── retrieval/           # bm25 / vector / hybrid (RRF) / rerank
│   ├── security/            # prompt-injection guardrail
│   ├── observability/       # OpenTelemetry tracing helpers
│   └── corpus/              # sample corpus (jsonl)
├── evals/                   # recall@k + MRR harness, QA dataset, promptfoo config
├── redteam/                 # injection payloads + runner
└── tests/                   # pytest

Roadmap — make it yours

These are deliberately left for you to implement and defend in interviews:

• [ ] Replace the sample corpus with a real one (your notes, a docs site, arXiv abstracts).
• [ ] Add a second MCP tool (e.g., fetch_document(id) or summarize(query)).
• [ ] Swap the embedding model and benchmark quality vs. latency.
• [ ] Add caching for embeddings and rerank scores; measure cost/latency savings.
• [ ] Wire traces into Jaeger or Langfuse and add a screenshot to the README.
• [ ] Expand the red-team set and report your catch rate over time.
• [ ] Migrate to the MCP SDK 2.0 MCPServer API once it stabilizes (currently pinned to the stable 1.x FastMCP line for maximum tutorial/Claude Desktop compatibility).

License

MIT — see LICENSE.