MCP RAG Agent

Production-ready RAG system combining LangGraph agent with Model Context Protocol (MCP) integration. Features hybrid search using Reciprocal Rank Fusion (RRF) via MongoDB vector and full-text searches, grounded responses using COSTAR prompting, and automated RAGAS-based evaluation for building reliable, context-aware AI agents.

Overview

The MCP RAG Agent is a sophisticated question-answering system that:

• Uses hybrid search to find relevant documents from a policy corpus
• Employs a LangGraph agent to reason about and retrieve information
• Integrates via the Model Context Protocol (MCP) for modular, reusable components
• Ensures grounded responses using the COSTAR prompting framework
• Stores and retrieves documents using MongoDB Atlas Vector Search
• Provides comprehensive evaluation tools using RAGAS metrics

Key Features

• MCP Integration: Standardized protocol for tool exposure and agent communication
• Hybrid Search: Combines vector similarity and keyword search using Reciprocal Rank Fusion (RRF)
• Semantic Search: Vector-based document retrieval using OpenAI embeddings
• Text Search: Full-text keyword search with stemming and relevance scoring
• MongoDB Atlas: Scalable vector storage with efficient similarity search
• Grounded Responses: Strict context-based answering with no hallucinations
• COSTAR Prompting: Structured prompt design for consistent, high-quality outputs
• LangGraph Agent: Reasoning and acting cycles for intelligent tool usage
• Automated Evaluation: RAGAS-based metrics for answer quality assessment

Architecture

The system architecture diagram illustrates two main workflows:

1. Document Indexing Flow (Setup Phase): Documents are processed, embedded using OpenAI, and stored in MongoDB Atlas Vector Search with appropriate indexing for efficient retrieval.

2. Question-Answering Flow (Runtime): User queries trigger the LangGraph ReAct agent, which uses MCP tools to search relevant documents via semantic search, then formulates grounded responses based on retrieved context.

Additionally, the system includes a third workflow not shown in the diagram:

3. Evaluation Flow (Quality Assurance): The system generates answers for predefined test questions and evaluates them using RAGAS metrics (relevancy, similarity, correctness) to ensure response quality and accuracy.

Project Structure

mcp-rag-agent/
├── data/
│   ├── ingested_documents/         # Source documents (policies)
│   │   └── policies/
│   │       ├── 1 - Remote Working.txt
│   │       ├── 2 - Expenses.txt
│   │       ├── 3 - Annual Leave.txt
│   │       ├── 4 - IT Security.txt
│   │       └── 5 - Sustainability.txt
│   └── evaluation_documents/       # Test cases for evaluation
│       └── expected_behaviour.xlsx
├── evaluation/                     # Automated testing and metrics
│   ├── main.py                     # Main evaluation orchestration script
│   ├── answer_generator.py         # Generates answers using the agent
│   ├── metrics_evaluator.py        # Evaluates answers using RAGAS metrics
│   ├── metrics.py                  # RAGAS metrics wrapper and definitions
│   ├── results/                    # Evaluation output (CSV files)
│   └── README.md                   # Evaluation module documentation
├── src/mcp_rag_agent/
│   ├── agent/                      # LangChain agent implementation
│   │   ├── create_agent.py         # Agent creation and configuration
│   │   ├── prompts/                # COSTAR-based system prompts
│   │   │   ├── __init__.py         # Prompts module exports
│   │   │   └── system_prompt.py    # System prompt definitions
│   │   ├── utils/                  # Agent utility functions
│   │   │   ├── mcp_rag_agent_creator.py  # MCP-enabled agent factory
│   │   │   └── rag_agent_creator.py      # Base RAG agent factory
│   │   └── README.md               # Agent module documentation
│   ├── embeddings/                 # Document processing and indexing
│   │   ├── embedding_generator.py  # OpenAI embeddings generation
│   │   ├── index_documents.py      # Document indexing pipeline
│   │   ├── semantic_search.py      # Vector similarity search
│   │   ├── hybrid_search.py        # Hybrid search combining vector + text
│   │   └── README.md               # Embeddings module documentation
│   ├── mcp_server/                 # MCP server implementation
│   │   ├── server.py               # FastMCP server with tools
│   │   ├── tools.py                # MCP tool implementations
│   │   └── README.md               # MCP server documentation
│   ├── mongodb/                    # Database client
│   │   ├── client.py               # MongoDB wrapper with vector search
│   │   └── README.md               # MongoDB module documentation
│   └── core/                       # Configuration and utilities
│       ├── config.py               # Environment-based configuration
│       └── log_setup.py            # Logging configuration
├── tests/                          # Tests
│   └── unit_tests                  # Unit tests
├── .env.example                    # Example environment configuration
├── .gitignore                      # Git ignore patterns
├── requirements.txt                # Production dependencies
├── requirements_dev.txt            # Development dependencies
├── setup.py                        # Package installation configuration
├── start.cmd                       # Windows startup script
└── README.md                       # This file

Quick Start

Prerequisites

• Python 3.8+
• MongoDB Atlas account (for vector search)
• OpenAI API key

Installation

1. Clone the repository:

git clone <repository-url>
cd mcp-rag-agent

2. Run the start file:

# Windows:
start.cmd

# Linux/macOS:
chmod +x start.sh
./start.sh

This script will automatically:

• Install and upgrade pip
• Create and activate a virtual environment
• Install all development dependencies
• Install the package in editable mode

3. Configure environment variables:

cp .env.example .env
# Edit .env with your settings

Setup Workflow

1. Index documents:

python -m mcp_rag_agent.embeddings.index_documents

This will:

• Read documents from data/ingested_documents/
• Generate embeddings using OpenAI
• Store vectors in MongoDB Atlas
• Create vector search index

2. Test the MCP server (optional - requires Node.js):

mcp dev src/mcp_rag_agent/mcp_server/server.py

This opens a UI to test the search_documents tool and other resources.

3. Run the agent:

python -m mcp_rag_agent.agent.create_agent

This runs a demo query showing the agent in action.

4. Evaluate performance (optional):

python evaluation/main.py

Runs automated evaluation using RAGAS metrics.

Usage Examples

Basic Agent Query

import asyncio
from mcp_rag_agent.agent.create_agent import create_mcp_rag_agent
from mcp_rag_agent.agent.prompts import system_prompt
from mcp_rag_agent.core.config import config

async def main():
    # Create agent
    agent = await create_mcp_rag_agent(
        system_prompt=system_prompt,
        config=config
    )
    
    # Query the agent
    result = await agent.ainvoke({
        "messages": [{
            "role": "user",
            "content": "What is the remote working policy?"
        }]
    })
    
    # Get the answer
    answer = result["messages"][-1].content
    print(answer)

asyncio.run(main())

Direct Semantic Search

import asyncio
from mcp_rag_agent.mongodb.client import MongoDBClient
from mcp_rag_agent.embeddings.embedding_generator import EmbeddingGenerator
from mcp_rag_agent.embeddings.semantic_search import SemanticSearch
from mcp_rag_agent.core.config import config

async def main():
    # Setup
    mongo_client = MongoDBClient(config.db_url, config.db_name)
    mongo_client.connect()
    
    embedder = EmbeddingGenerator(
        api_key=config.model_api_key,
        model=config.embedding_model
    )
    
    search = SemanticSearch(mongo_client, embedder)
    
    # Search
    results = await search.search(
        query="annual leave entitlement",
        limit=3
    )
    
    for doc in results:
        print(f"File: {doc['file_name']}")
        print(f"Score: {doc['score']:.3f}")
        print(f"Content: {doc['content'][:200]}...\n")
    
    mongo_client.disconnect()

asyncio.run(main())

Hybrid Search (Recommended)

import asyncio
from mcp_rag_agent.mongodb.client import MongoDBClient
from mcp_rag_agent.embeddings.embedding_generator import EmbeddingGenerator
from mcp_rag_agent.embeddings.hybrid_search import HybridSearch
from mcp_rag_agent.core.config import config

async def main():
    # Setup
    mongo_client = MongoDBClient(config.db_url, config.db_name)
    mongo_client.connect()
    
    embedder = EmbeddingGenerator(
        api_key=config.model_api_key,
        model=config.embedding_model
    )
    
    hybrid = HybridSearch(
        mongo_client=mongo_client,
        embedding_generator=embedder,
        default_collection=config.db_vector_collection
    )
    
    # Perform hybrid search (combines semantic + keyword matching)
    results = await hybrid.search(
        query="What are the sustainability initiatives?",
        limit=5,
        semantic_weight=0.7  # 70% semantic, 30% keyword (default)
    )
    
    for doc in results:
        print(f"RRF Score: {doc['rrf_score']:.4f}")
        print(f"Vector Rank: {doc['vector_rank']}, Text Rank: {doc['text_rank']}")
        print(f"Content: {doc['content'][:200]}...\n")
    
    mongo_client.disconnect()

asyncio.run(main())

Indexing New Documents

import asyncio
from mcp_rag_agent.embeddings.index_documents import index_documents
from mcp_rag_agent.core.config import config

async def main():
    await index_documents(
        directory_path="data/ingested_documents",
        config=config
    )

asyncio.run(main())

Module Documentation

Each module has detailed documentation:

• Agent: LangGraph ReAct agent with MCP integration
• MCP Server: FastMCP server providing RAG tools
• MongoDB: Database client with vector, text, and hybrid search capabilities
  • See SEARCH_GUIDE.md for detailed comparison of search methods
• Embeddings: Document indexing, semantic search, and hybrid search
• Evaluation: Automated testing with RAGAS metrics

Configuration

Configuration is managed through two layers:

1. Environment Variables (.env): Most settings are configured via environment variables, although only the external dependencies are included in the .env.sample file.
2. Code Configuration (src/mcp_rag_agent/core/config.py): Some advanced settings are configured directly in the Config class, such as text generation parameters (temperature,...)

Note: To modify these settings, edit src/mcp_rag_agent/core/config.py directly. The Config class loads environment variables and provides default values for all configuration parameters.

Key Technologies

• LangChain: Agent framework and orchestration
• Model Context Protocol (MCP): Standardized tool integration
• FastMCP: MCP server implementation
• MongoDB Atlas: Vector storage and search
• OpenAI: LLM and embedding models
• RAGAS: RAG evaluation framework

Development

Running Tests

pytest tests/

Code Structure

• Follow Python best practices and PEP 8
• Use type hints for all functions
• Add docstrings to public APIs
• Keep modules focused and cohesive

Adding New Features

1. New MCP Tool:

   • Add @mcp.tool() decorated function in server.py
   • Document in MCP server README
   • Test with mcp dev

2. New Document Type:

   • Update index_documents.py to handle new format
   • Ensure metadata is preserved
   • Re-index documents

3. New Metric:

   • Add to evaluation/metrics.py
   • Update evaluator to compute and save metric
   • Document in evaluation README

Evaluation

The project includes comprehensive evaluation tools using RAGAS:

python evaluation/evaluator.py

Metrics computed:

• Answer Relevancy
• Answer Similarity
• Answer Correctness

Results are saved to evaluation/results/ with timestamps.

Troubleshooting

Common Issues

MongoDB connection fails:

• Verify MongoDB Atlas cluster is running
• Check IP whitelist in Atlas
• Validate connection URI in .env

MCP server won't start:

• Ensure MongoDB is connected
• Check OpenAI API key is valid
• Verify all dependencies are installed

No search results:

• Run index_documents.py to populate database
• Check vector index exists in MongoDB Atlas
• Verify embedding dimensions match

Agent doesn't call tools:

• Check MCP server is accessible
• Review system prompt encourages tool usage
• Increase model temperature if needed

Evaluation errors:

• Ensure expected_behaviour.xlsx exists
• Check OpenAI API quota
• Verify evaluation model is accessible

Performance Considerations

• Indexing: ~1-2 seconds per document (depends on document size)
• Query: ~2-5 seconds per query (embedding + search + generation)
• Vector Search: Sub-second for collections up to 100K documents
• Batch Operations: Use insert_documents() for bulk indexing

Best Practices

1. Prompt Engineering: Use COSTAR framework for all prompts
2. Error Handling: Always handle connection failures gracefully
3. Logging: Use structured logging for debugging
4. Testing: Run evaluation after significant changes
5. Vector Index: Create during setup, not runtime
6. Connection Pooling: Reuse MongoDB client instances
7. API Rate Limits: Implement exponential backoff for OpenAI calls

Security

• Never commit .env file to version control
• Rotate API keys regularly
• Use MongoDB Atlas IP whitelisting
• Implement rate limiting for production deployments
• Sanitize user inputs before processing

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Update documentation
6. Submit a pull request

License

MIT