RAG Factory — AutoML for RAG

RAG Factory turns RAG configuration from guesswork into measurement. From a short conversation plus a sample of your own documents, it recommends candidate RAG configurations, builds an isolated sub-environment for each, evaluates it on your own data, and optimizes across the candidate grid — then registers the winning environments as a persistent fleet you can query directly or over MCP.

Status: v1.0 snapshot. This repository is a curated portfolio snapshot of a project that continues to evolve in a private development repository.

📸 Screenshots / demo of the web UI will be added here.

---

The problem

A good RAG setup cannot be reliably predicted up front. MTEB scores do not predict performance in your domain, and the chunking strategy can matter as much as the embedding model. The only trustworthy answer is to measure candidates on your own corpus — but doing that by hand (generating a QA set, building indexes, running ranking metrics, comparing) is slow and expensive.

The solution

RAG Factory closes the loop:

[Recommender]  ->  [Factory]  ->  [Evaluator]
   input            build          measure
     ^                                 |
     +-------- optimization loop ------+

• Recommender (input). A conversational interview plus a data-profile turns your intent and a document sample into a search space of candidate configurations — not a single guessed "truth".
• Factory (build). For a given configuration it builds a sub-environment: parsing → chunking → embedding → index.
• Evaluator (output). It measures each candidate on your own data (MRR, NDCG, faithfulness / completeness), and the result feeds back to select the winner.
• Registry (fleet). Winning environments are persisted and can be served — locally, via a Gradio web UI, or through an MCP server.

What is built here vs. adopted

Honesty about the boundary matters, so it is explicit:

Layer	Decision	Why
Build + eval + optimize core, ranking metrics (MRR/NDCG)	Adopted — AutoRAG	The optimizer and validated metrics are a solved problem; reimplementing them would be worse and slower
QA ground-truth generation	Adopted (engine-provided)	Hand-curating a test set dominates development time
Conversational recommender (input layer)	Built here	No existing tool offers this; the unique, agentic part
Fleet registry / environment manager	Built here	Tools optimize "one best pipeline for one dataset"; they do not manage N persistent environments
MCP server + Gradio web UI + optimization orchestration	Built here	The serving and human-in-the-loop layer wrapping the engine

Principle: reinvent zero metrics and zero optimizer; build exactly what is new.

Tech stack

Python · AutoRAG · LlamaIndex (OpenAI-compatible LLMs) · MCP (FastMCP) · Gradio · pandas / numpy · langdetect · PyYAML · LLM access via OpenRouter.

Package layout

Module	Responsibility
ragfactory/interview.py, profiler.py	Conversational recommender: intent interview + data profiling
ragfactory/recommendation.py, grid.py	Candidate search space & finalist recommendation
ragfactory/corpus.py, solution.py, yaml_builder.py	Corpus prep, QA generation, environment build (AutoRAG YAML)
ragfactory/runner.py, optimize.py, _optimize_worker.py	Build/eval execution and the optimization loop
ragfactory/generation.py, reports.py, emission.py	Answer generation, reporting, artifact emission
ragfactory/registry.py, serving.py	Fleet registry & query serving
ragfactory/mcp_server.py	MCP server (FastMCP, stdio) exposing the fleet
ragfactory/app.py, web_ui.py	Gradio web application

Getting started

# 1. Environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Configure access (copy the template and fill in your key)
cp .env.example .env
#   then edit .env: OPENROUTER_API_KEY, OPENROUTER_BASE_URL, OPENROUTER_MODEL

Run the web application:

python -m ragfactory.app

Run the MCP server (stdio transport):

python -m ragfactory.mcp_server
# optional: RAGFACTORY_REGISTRY_DIR=/path/to/registry python -m ragfactory.mcp_server

Note: some build/evaluation paths run AutoRAG and may require an embedding model and a GPU. A corpus is not bundled — you point RAG Factory at your own documents.

Testing

pytest                    # unit / fast tests (default; costly tests deselected)
pytest -m integration     # slower tests that run AutoRAG / GPU
pytest -m costly          # tests that spend money on live LLM calls (OpenRouter)

The suite is substantial (24 test modules) and mirrors the production package closely. Markers are defined in pytest.ini.

Documentation

• docs/ARCHITECTURE.md — design decisions, the build-vs-adopt rationale, and evaluation methodology.

License

Released under the MIT License © 2026 Dariusz Poślad.