mcp-retrieval-probe

An evaluation harness for MCP-based retrieval servers.

Retrieval servers that sit behind the Model Context Protocol are increasingly the grounding layer for AI assistants: the assistant asks a question, the server returns documents, and the assistant cites them. That makes one property more important than raw relevance. Does the server know when it does not know?

A retrieval server that returns its best guess with a confident-looking score, even when the corpus has nothing useful, will hand an assistant a citation it should never have made. The model did not hallucinate. The grounding layer manufactured the certainty and passed it up. This tool measures that, along with the corpus and data-integrity properties that tend to drift quietly.

It is target-agnostic. Everything server-specific lives in a recipe file, so the same probes run against any MCP retrieval server.

Install

git clone https://github.com/borisknight/mcp-retrieval-probe
cd mcp-retrieval-probe
pip install -r requirements.txt

Use

Write a recipe (start from examples/example.recipe.json), export your own API token, and run a probe:

export MCP_TOKEN="your-own-token"
python -m mcp_retrieval_probe --recipe examples/example.recipe.json calibration
python -m mcp_retrieval_probe --recipe examples/example.recipe.json all --json

The token is read from the environment variable named in the recipe and is never written to disk or logged. Use your own credentials against servers you are permitted to test.

The probes

Probe	Question it answers
calibration	Does the confidence signal mean anything, and does the server ever abstain?
anisotropy	Do similarity scores have enough dynamic range to threshold on?
relevance	Are results on topic, and does a paper retrieve itself by title?
coverage	Does the corpus match its claims, and how much of it is embedded?
citations	Is citation data present and correct, or is "unknown" reported as zero?
conservation	Do two endpoints counting the same thing agree?
discovery	Does the OAuth discovery metadata match the documented endpoint?

Each probe prints a one-line verdict and returns a structured result under --json.

Why these, specifically

calibration sends three buckets of queries: topics the corpus covers, real scholarship it almost certainly does not, and pure gibberish. A calibrated server looks visibly less certain on the last two. One that returns an identical confidence label for all three is not giving its callers a signal at all.

anisotropy explains why a confidence label often fails. In an anisotropic embedding space nearly every pair of vectors is highly cosine similar, so scores bunch into a narrow band and no fixed threshold can separate a strong match from a weak one. The fix is usually normalisation before thresholding, not a new label.

citations separates two things servers routinely conflate: a document with zero citations, and a document whose citation data failed to link. Reporting the second as the first makes landmark work look ignored.

discovery needs no credentials. It compares the canonical resource identifier a server advertises against the endpoint it documents, a mismatch that produces confusing auth failures for spec-compliant clients.

Recipes

A recipe has three sections:

• transport — base_url, protocol_version, and token_env (the name of the environment variable holding the bearer token).
• tools — maps each logical tool a probe needs to the concrete tool the server exposes. Each entry has a name, an args template using $QUERY, $LIMIT, $ID, $CATEGORY, $START, $END, $SOURCE placeholders, and a fields map of dotted response paths. A placeholder whose value is None is dropped from the call, so optional filters can be omitted. Logical tools the recipe leaves out simply cause the probes that need them to skip.
• probes — per-probe configuration: query buckets, category anchors, seeds, the OpenAlex concept to audit, and so on.

Ground truth for the citation probe comes from the public OpenAlex API.

Running it as an MCP server

The probes are also exposed as MCP tools, so an assistant can run an evaluation in conversation and a CI agent can call them directly:

pip install 'mcp>=2.0'

{
  "mcpServers": {
    "retrieval-probe": {
      "command": "python",
      "args": ["-m", "mcp_retrieval_probe.server"],
      "env": {
        "MCP_PROBE_RECIPE": "/path/to/recipe.json",
        "MCP_TOKEN": "your-own-token"
      }
    }
  }
}

Each probe becomes a probe_<name> tool, plus probe_all.

Tests

pip install -r requirements-dev.txt
pytest

The default suite is offline. It needs no network and no credentials, and it covers three layers:

• Transport — the MCP handshake, session id handling, SSE parsing, payload unwrapping, and the 401 and JSON-RPC error paths, against a stubbed HTTP call.
• Probe logic — each probe is shown to fire on a server with the defect it looks for and stay quiet on one without it. A verdict is only worth reporting if you can demonstrate it discriminates.
• CLI — the paths a first-time user actually hits: a missing token, a bad recipe path, --json output, and a probe that raises without aborting the run.

Functional checks against a real server are opt-in, so CI stays credential-free:

export MCP_PROBE_RECIPE=/path/to/recipe.json
export MCP_TOKEN=...          # whatever token_env your recipe names
pytest -m live

Those assert reachability and response shape rather than specific findings, since findings depend on the server under test.

License

MIT. See LICENSE.