Hybrid Agentic RAG — LangChain + LangGraph + Evaluation

This project implements hybrid agentic RAG over Confluence using Python, LangChain, LangGraph, FAISS, BM25Retriever, Cohere reranker, OpenAI embeddings, RAGAS, BERTScore, LangSmith, and FastMCP. An MCP server and Streamlit chatbot are implemented as core functionalities, with a full evaluation suite and LangSmith monitoring.

---

Table of Contents

• How it Differs from Hybrid RAG
• Setup
• Pipeline
• Testing Without a Confluence Instance
• Monitoring: LangSmith
• Evaluation: RAGAS + Custom Metrics
• MCP Server

---

How it differs from hybrid RAG

The retrieval quality and pipeline structure are identical. What changes is the underlying library stack and what you get on top of it.

Component	Hybrid RAG	This project (LangChain)
BM25	bm25s	BM25Retriever (langchain-community)
Dense index	numpy .npy	FAISS vectorstore (langchain-community)
RRF fusion	custom _rrf()	EnsembleRetriever (same c=60 constant)
Reranking	cohere.Client()	CohereRerank + ContextualCompressionRetriever
Agent	pydantic-ai	create_react_agent (LangGraph)
Tracing	none	LangSmith (automatic, zero code)
Evaluation	none	RAGAS + custom MRR/NDCG/BERTScore

---

Setup

uv pip install -r requirements.txt
cp .env.example .env
# Fill in CONFLUENCE_*, OPENAI_API_KEY, COHERE_API_KEY, ANTHROPIC_API_KEY
# Optionally add LANGCHAIN_API_KEY for LangSmith monitoring

---

Pipeline

1-fetch-confluence.py   Fetch Confluence pages → chunks/*.json
1-fetch-k8s.py          Fetch public K8s docs  → chunks/*.json  (no account needed)
        ↓
2-build-index.py        Build indexes/bm25.pkl + indexes/faiss/ + indexes/meta.json
        ↓
3-hybrid-search.py      Interactive CLI to test the four-stage retrieval chain
        ↓
4-agent.py              LangGraph ReAct agent with structured citations
        ↓
5-evaluate.py           Full evaluation suite (retrieval + generator + end-to-end)
        ↓
6-mcp-server.py         FastMCP server for Claude Desktop / Cursor / Claude Code
7-chatbot.py            Streamlit chatbot with LangSmith trace links

Run each step

uv run 1-fetch-confluence.py
uv run 2-build-index.py
uv run 3-hybrid-search.py
uv run 4-agent.py "What is our deployment process?"
uv run 5-evaluate.py
uv run 6-mcp-server.py
uv run streamlit run 7-chatbot.py

---

Testing without a Confluence instance

1-fetch-k8s.py scrapes the public Kubernetes documentation (kubernetes.io/docs) and saves chunks in the exact same JSON format that 2-build-index.py expects, so the full pipeline — indexes, agent, evaluation, MCP server, chatbot — runs unchanged. No Confluence account or API token needed.

Prerequisites

# beautifulsoup4 is the only extra dependency
uv pip install beautifulsoup4

Running the scraper

# Scrape ~190 Kubernetes docs pages (≈ 2 min at 0.5 s/request)
uv run 1-fetch-k8s.py

# Then continue with the normal pipeline
uv run 2-build-index.py
uv run 3-hybrid-search.py
uv run 6-mcp-server.py              # terminal 1
uv run streamlit run 7-chatbot.py   # terminal 2

Expected output:

Fetching sitemap: https://kubernetes.io/en/sitemap.xml
Found 192 pages to index

[  1/192] Concepts                                           3 chunk(s)
[  2/192] Kubernetes Components                              4 chunk(s)
...
Done: 189 pages → 847 chunks  (2 empty, 1 errors)
Chunks saved to: .../chunks/

Configuration

Edit the constants at the top of 1-fetch-k8s.py:

Constant	Default	What it controls
INCLUDE_SECTIONS	concepts/, tasks/, tutorials/, setup/, reference/glossary/, reference/kubectl/	Sections of kubernetes.io/docs to crawl
SKIP_PATTERNS	reference/kubernetes-api/, contribute/	Sub-paths excluded even within included sections
REQUEST_DELAY	0.5 s	Pause between HTTP requests
MAX_CHUNK_CHARS	1500	Maximum characters per chunk
OVERLAP_CHARS	150	Overlap between consecutive chunks

Example queries once running

• "What is the difference between a Deployment and a StatefulSet?"
• "How do I configure resource limits for a Pod?"
• "What happens when a node fails?"
• "How does the Kubernetes scheduler decide where to place a Pod?"

For evaluation, add K8s questions to eval_set.json and run uv run 5-evaluate.py to get MRR, NDCG@10, RAGAS faithfulness, and BERTScore against this corpus.

---

Monitoring: LangSmith

Add three lines to your .env:

LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls_...
LANGCHAIN_PROJECT=hybrid-agentic-rag

That's it. Every LangChain and LangGraph call — embedding lookups, retriever invocations, LLM completions, tool calls — is automatically traced. Open smith.langchain.com to see latency breakdowns, token usage, and full input/output for every step.

Free tier: 5,000 traces/month.

---

Evaluation: RAGAS + custom metrics

Edit eval_set.json to add your own questions:

[
  {
    "question": "What is our deployment process?",
    "ground_truth": "...",
    "relevant_chunk_ids": ["page123_c0", "page456_c2"]
  }
]

• ground_truth: needed for ROUGE, BERTScore, context precision/recall
• relevant_chunk_ids: needed for MRR, NDCG@10, Precision@10, Recall@10 (chunk IDs are visible in 3-hybrid-search.py output)

If fields are omitted, those metric groups are skipped gracefully.

Metrics covered

Category	Metric	Implementation
Retrieval	MRR	utils/evaluation.py
Retrieval	NDCG@10	utils/evaluation.py
Retrieval	Precision@10	utils/evaluation.py
Retrieval	Recall@10	utils/evaluation.py
Generator	ROUGE-1/2/L	rouge-score library
Generator	BERTScore F1	bert-score library
End-to-End	Faithfulness	RAGAS (LLM-as-judge)
End-to-End	Answer Relevancy	RAGAS (LLM-as-judge)
End-to-End	Context Precision	RAGAS (needs ground_truth)
End-to-End	Context Recall	RAGAS (needs ground_truth)
Monitoring	Latency, tokens, errors	LangSmith (automatic)

RAGAS uses OpenAI gpt-4o-mini as judge by default. Override with RAGAS_JUDGE_MODEL=gpt-4o in .env for higher accuracy.

---

MCP server

uv run 6-mcp-server.py   # starts on http://localhost:8051/sse

Add to Claude Desktop config (%APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "confluence": { "url": "http://localhost:8051/sse" }
  }
}

Or to .mcp.json in any repo root for Claude Code:

{
  "mcpServers": {
    "confluence": { "type": "sse", "url": "http://localhost:8051/sse" }
  }
}