retriEVAL

LLM evaluation as an MCP server. Score your AI's outputs for faithfulness, relevancy, and hallucination from inside any MCP client — no pipeline, no test harness. Every result comes back with a link to a dashboard that keeps the history.

Try it live (no signup) · Watch the 2-minute demo · Dashboard

---

Why

Five customer-support answers, scored on two metrics:

metric	score	passing
answer_relevancy	0.98	5/5
faithfulness	0.70	3/5

Every answer was on-topic and well-written. Two of them contradicted the policy they were supposedly grounded in — one promised free return shipping the policy doesn't offer, another invented a free overnight replacement. Reviewing by eye, you'd sign off on all five.

That gap is the point. Relevancy asks did it answer the question. Faithfulness asks is it actually in the source. You need both, and the second one catches the expensive failures.

Connect

Self-hosted. Clone it, point it at a judge, run it — your data never leaves your machine and there's no service to sign up for.

git clone https://github.com/hcarrillo001/retrieval-mcp
cd retrieval-mcp
pip install -r requirements.txt

export ANTHROPIC_API_KEY=sk-ant-...        # or a local judge, below
python server.py                            # stdio, for Claude Desktop / Cursor

Then just ask:

Score these cases with faithfulness: [{"input": "...", "actual_output": "...", "retrieval_context": ["..."]}]

Pass your cases inline and nothing is stored — one call, no setup step. See Run locally (stdio) for client config, and Deploy as HTTP if you want your own always-on instance with a dashboard.

Want to try it before installing anything? There's a live sandbox at retrieval-mcp.com — no signup, runs on a free judge, nothing saved.

Keeping everything local: set RETRIEVAL_JUDGE_BACKEND=ollama and the judge runs on your machine too, so no data leaves your network at any point. Useful if you're evaluating anything you can't send to a third party.

What you get

• 9 built-in metrics plus custom metrics you author in plain English
• Swappable judges — Anthropic, Groq, Gemini, OpenRouter, or a local Ollama model, so nothing has to leave your network
• Golden sets from files, URLs, inline JSON, JSONL, CSV, or TSV
• Run history in Supabase with shareable permalinks and run comparison
• A spend cap, because a judge-based tool can otherwise run up a bill

Honest limitations

• Judge agreement hasn't been validated against human labels yet, so treat scores as a signal rather than ground truth.
• Golden sets currently hold their own outputs, so re-running one against new model outputs means loading a second set. Splitting them is the next change.
• Golden sets and authored metrics live in the server process and are lost on restart. Runs persist; those don't.

---

Metrics (DeepEval-aligned)

faithfulness · answer_relevancy · contextual_precision · contextual_recall · contextual_relevancy · hallucination · bias · toxicity · summarization — plus authored G-Eval metrics you define in plain language. All are normalized so higher = better (bias/toxicity report the clean fraction), and each reasons before scoring.

Versatile golden sets

load_golden_set accepts a file path (including uploaded files), an http(s) URL, an inline JSON array, or JSONL text, in JSON / JSONL / CSV / TSV. Field names are auto-normalized (question→input, answer→actual_output, ground_truth→expected_output, contexts→context, passages→retrieval_context, …), so most public benchmarks load as-is.

Short replies by default

run_eval scores every case but returns only the 3 lowest-scoring by default (tune with limit), with total_cases/shown and a pointer to show_run_cases(run_id, offset, limit, metric) to page through the rest.

Spend cap (so it can't run up a bill)

Judge spend is metered from real token usage and persisted. Set a hard cap:

export RETRIEVAL_BUDGET_USD=20     # 0/unset = unlimited

Once cumulative spend hits the cap, further Anthropic calls stop and tools return a clear budget_exceeded message. Check/clear with get_budget / reset_budget. (Prices are approximate — override RETRIEVAL_PRICE_IN/OUT $/1M tokens to match current pricing for your model.)

Hybrid setup (local + always-on dashboard)

Run history uses a pluggable store, chosen by env:

• FileStore (default) — JSONL in ~/.retrieval. Zero setup, local only.
• SupabaseStore — when SUPABASE_URL + SUPABASE_SERVICE_KEY are set. Run history lives in Postgres, shared by the local CLI, the deployed MCP, and the website dashboard.

Recommended hybrid flow:

1. Run supabase_schema.sql in Supabase (creates the runs table).
2. Set SUPABASE_URL + SUPABASE_SERVICE_KEY on the MCP (local and/or Railway) so every run is written centrally. Each run records its generator_model and judge_model for cross-model comparison.
3. Deploy web/ to Vercel (set the same Supabase env vars) and map it to retrieval-mcp.com. The dashboard reads history via /api/runs (service key stays server-side) and renders trend-by-model, a model leaderboard, and run history. It shows sample data until Supabase is wired.

Local stays your free sandbox (Ollama judge, file history); the website is the always-on window into the shared history.

Run locally (stdio) — Claude Desktop

pip install -r requirements.txt
export ANTHROPIC_API_KEY=sk-ant-...

{
  "mcpServers": {
    "retrieval": {
      "command": "python",
      "args": ["/ABSOLUTE/PATH/server.py"],
      "env": { "ANTHROPIC_API_KEY": "sk-ant-...", "RETRIEVAL_BUDGET_USD": "10" }
    }
  }
}

Then: "Load examples/rag_golden.jsonl as 'space', run faithfulness, label it v1."

Deploy as HTTP (reach it from anywhere)

export RETRIEVAL_TOKEN=$(openssl rand -hex 24)   # required for a public endpoint
export ANTHROPIC_API_KEY=sk-ant-...
export RETRIEVAL_BUDGET_USD=20
python app.py        # serves $PORT (default 8000); MCP at /mcp, health at /healthz

Deploy to Railway (or any host): the included Dockerfile / Procfile work as-is. Set ANTHROPIC_API_KEY, RETRIEVAL_TOKEN, RETRIEVAL_BUDGET_USD in the host env. Clients connect to https://<host>/mcp with header Authorization: Bearer <token> — add it as a custom connector in claude.ai / Claude Desktop, or point Agent Builder / CI at it. State (golden sets, run history, spend) lives server-side, so it persists across machines.

See it on a real RAG pipeline (demo)

demo/rag_demo.py builds a tiny end-to-end RAG over a small labeled dataset (demo/labeled.json + demo/corpus.json): it retrieves with BM25, computes recall@k against the gold passages (deterministic — the retriever's score), generates an answer, then scores faithfulness (the generator's score). One answer is deliberately hallucinated so you watch the two failure modes separate.

python demo/rag_demo.py            # offline, no key needed
python demo/rag_demo.py --real     # real generation + RetriEval judge (needs ANTHROPIC_API_KEY)

It also writes demo/generated_goldenset.jsonl — load that into the MCP (load_golden_set → run_eval) for the judge-scored version. This is the bridge: your pipeline emits predictions, the dataset supplies the labels, and RetriEval scores retriever and generator independently.

Connecting to a RAG pipeline

• Offline (default): export your pipeline's retrieved context + answer into a golden set and score it — RetriEval never touches your pipeline.
• Live: add a query_rag(question) tool that calls your RAG endpoint or your vector store (Chroma / Supabase pgvector), captures context + answer, and scores in one shot.

Judge backend

export RETRIEVAL_JUDGE_BACKEND=anthropic          # default
export RETRIEVAL_JUDGE_MODEL=claude-sonnet-4-6
# or local, free:
export RETRIEVAL_JUDGE_BACKEND=ollama
export RETRIEVAL_JUDGE_MODEL=deepseek-r1:70b

Tools

Tool	Purpose
list_metrics	built-in + authored metrics
load_golden_set(name, source, fmt)	name a set for reuse (self-host only — shared and lost on restart)
list_golden_sets	what's loaded
author_metric(name, criteria, examples)	plain language → a scorer
run_eval(metrics, cases, golden_set, threshold, outputs, label, limit)	score a set; pass cases inline (JSON/JSONL/CSV/TSV/path/URL) — nothing stored
show_run_cases(run_id, offset, limit, metric)	page the rest
evaluate_case(...)	one-off score
ground_against_url(url, output, question)	check an output's consistency with a web page (no labels — consistency, not correctness)
list_runs(golden_set, last_n)	saved runs
plot_metric_trend / plot_run / compare_runs	inline charts
get_budget / reset_budget	spend cap status / reset

---

License

Apache License 2.0. Built by Hanns Carrillo.