corpus-mcp

A local MCP server that gives an agent clean, efficient access to a local knowledge corpus of offline ZIM archives — Wikipedia, medical (MDWiki), developer documentation (DevDocs), and Stack Exchange — through one uniform interface. No internet, no embeddings, no vector database: libzim full-text search plus deterministic, in-server content cleaning.

The public MCP surface is exactly two tools:

search(query, limit?)
fetch(ref, sections?)

The configured corpus is an operator concern, not an agent concern. The agent only ever:

discover  →  search()
select    →  fetch()

Corpus families

Corpus	kind	Document	Section model
Wikipedia, MDWiki	article	article	heading tree (h2+), lead section id ""
DevDocs (C, CMake, Python)	documentation	documentation page	heading tree; in-page TOCs and nav chrome stripped
Stack Exchange	thread	question + answers	synthetic sections: question, accepted-answer, answer-<id>

All corpus identification, routing, ZIM access, HTML interpretation, cleanup, ranking, redirect handling, and normalization remain server responsibilities. The agent is never required to parse HTML, resolve redirects, construct or parse references, or know anything about libzim, ZIM namespaces, or corpus storage internals.

References

search() results carry an opaque ref (e.g. corpus://Wikipedia/Bell_test); fetch() consumes it. The agent must never construct, parse, or modify a ref, nor infer the corpus from one:

search() produces ref      fetch() consumes ref

Architecture

Local agent
    │  MCP / Streamable HTTP  →  http://127.0.0.1:8000/mcp
    ▼
┌──────────────────────────────────────────────┐
│ Corpus MCP Server                            │
│  search()  fetch()                           │
│  ├─ CorpusManager (routing, cache,          │
│  │   bounded-concurrency fan-out)           │
│  ├─ federated ranking (RRF + lexical title  │
│  │   reranking + diversity)                 │
│  ├─ adapters: mediawiki / devdocs /         │
│  │   stackexchange                           │
│  ├─ HTML cleaner → Markdown, section trees  │
│  └─ GlobalRef codec (opaque refs)           │
└─────────────┬────────────────────────────────┘
              ▼
      per-library ZIM service (only libzim touchpoint,
      one search lock per archive)
              ▼
      corpus/  (read-only volume, N .zim archives)
      corpus.toml  (manifest: name, adapter, path)

The MCP layer exposes no libzim concepts: no namespaces, cluster IDs, raw entries, MIME types, or raw HTML.

Prerequisites

• Docker + Docker Compose
• ZIM archives (see below)
• For running the test suite locally: Python 3.12 and uv (or pip)

Corpus layout

The server never downloads archives itself — corpus acquisition is deliberately decoupled from application startup. Default layout:

corpus/
  wikipedia/wikipedia_en_all_nopic_*.zim
  medical/mdwiki_en_all_maxi_*.zim
  devdocs/devdocs_en_cpp_*.zim
  devdocs/devdocs_en_cmake_*.zim
  devdocs/devdocs_en_python_*.zim
  stackexchange/stackoverflow.com_en_all_*.zim
  stackexchange/security.stackexchange.com_en_all_*.zim
  stackexchange/softwareengineering.stackexchange.com_en_all_*.zim
corpus.toml

corpus.toml names each library, its adapter, and its path (relative to the corpus root):

version = 1

[[library]]
name = "Wikipedia"
path = "wikipedia/wikipedia_en_all_nopic_2026-06.zim"
adapter = "mediawiki"

[[library]]
name = "CMake-Docs"
path = "devdocs/devdocs_en_cmake_2026-08.zim"
adapter = "devdocs"

Validation rules: unique names, known adapters, paths must stay inside the corpus root. Verify a corpus before starting the server:

make validate-corpus   # opens every archive, reports metadata
make corpus-list       # list configured libraries

Startup / shutdown

make start           # build + start (docker compose, detached)
make logs            # tail logs
make ps              # container status
make stop            # stop (keep containers)
make down            # stop + remove
make restart
make build

The MCP endpoint is then available at http://127.0.0.1:8000/mcp (Streamable HTTP). The host port is bound to loopback only by default; the container listens on 0.0.0.0:8000 internally.

If any configured ZIM cannot be opened, the server fails to start and names the offending library — there is no partially functional mode.

Tool schemas

search(query: str, limit?: int)

Searches the full-text index of every configured library (bounded concurrency, one worker per archive), fuses the ranked lists with Reciprocal Rank Fusion, reranks tied cross-corpus candidates by lexical title coverage, applies a deterministic diversity pass, and returns clean results. limit defaults to 5; the server enforces a hard maximum (SEARCH_MAX_LIMIT, default 10).

{
  "results": [
    {
      "ref": "corpus://Wikipedia/Bell_test",
      "library": "Wikipedia",
      "kind": "article",
      "title": "Bell test",
      "snapshot": "2026-06",
      "snippet": "To close the detection loophole, an apparatus with a high detection efficiency is needed.",
      "relevant_sections": [
        { "id": "Notable_experiments", "title": "Notable experiments" },
        { "id": "Loopholes", "title": "Loopholes" }
      ]
    }
  ]
}

• ref — opaque global identifier; pass it back to fetch().
• library / kind / snapshot — provenance: which archive, what kind of document, and the corpus snapshot (derived from archive metadata).
• relevant_sections — 0–3 deterministic lexical hints (empty when no section clearly matches). Section IDs are server-derived; the agent must not reconstruct them.

One failing library degrades the search (the others still answer); it never kills it.

fetch(ref: str, sections?: list[str])

Returns the cleaned document as structured Markdown.

• Without sections: the whole document (bounded by MAX_FETCH_CHARS; truncated: true if cut at a section boundary).
• With sections: only those sections (subtrees included). Section IDs come from search() hints or from available_sections. The lead/intro section has id "". For threads, sections are question, accepted-answer, and answer-<id>; their metadata carries score, acceptance, and tags.

{
  "ref": "corpus://Wikipedia/Bell_test",
  "library": "Wikipedia",
  "kind": "article",
  "title": "Bell test",
  "snapshot": "2026-06",
  "sections": [
    { "id": "Loopholes", "title": "Loopholes", "content": "## Loopholes\n\n..." }
  ],
  "available_sections": [
    { "id": "", "title": "Bell test" },
    { "id": "Background", "title": "Background" },
    { "id": "Loopholes", "title": "Loopholes" }
  ],
  "truncated": false
}

Errors are concise and actionable:

{ "error": "invalid_ref", "message": "invalid reference: ..." }
{ "error": "not_found", "message": "Document not found in Wikipedia: Foo_bar" }
{
  "error": "section_not_found",
  "missing_sections": ["Experiments"],
  "available_sections": [ { "id": "Loopholes", "title": "Loopholes" }, "..." ]
}

Example agent workflow

search("Bell experiment loopholes")
    ↓
fetch("corpus://Wikipedia/Bell_test", ["Notable_experiments", "Loopholes"])

Configuration

Environment variables (container defaults shown):

Variable	Default	Meaning
CORPUS_ROOT	/corpus	Corpus root inside the container (required)
CORPUS_CONFIG	/config/corpus.toml	Manifest path inside the container (required)
MCP_HOST	0.0.0.0	Listen address inside the container
MCP_PORT	8000	Listen port inside the container
SEARCH_LIMIT	5	Default limit for search()
SEARCH_MAX_LIMIT	10	Hard maximum for search(limit=…)
MAX_FETCH_CHARS	100000	Output budget for fetched content
SEARCH_WORKERS	8	Concurrent archive searches during fan-out
SEARCH_MAX_CONSECUTIVE	2	Diversity pass: max consecutive results from one library
LOG_QUERIES	true	Log search query text (privacy)
ZIM_CHECK	false	Run libzim's full checksum verification at startup (reads the entire corpus: opt-in, slow on large archives)

Host-side compose variables: CORPUS_ROOT (default ./corpus) and CORPUS_CONFIG (default ./corpus.toml).

The server fails fast on invalid configuration.

Tests

make test     # unit + integration + MCP surface tests (needs .venv)
make lint
make format

Setup for a local test run:

uv venv .venv --python 3.12
uv pip install -e . --python .venv/bin/python
uv pip install --python .venv/bin/python pytest pytest-asyncio ruff
make test

Tests build their own small ZIM fixtures with libzim's writer (one per corpus family); no corpus is required. The MCP surface regression test asserts that the server exposes exactly the two tools search and fetch, and no prompts or resources.

Security posture

Local service by design: host binding is loopback-only by default, the corpus volumes are read-only, the container runs as a non-root user, no privileged mode, no Docker socket, no arbitrary filesystem access, no URL fetching, no shell execution. Neither tool accepts filesystem paths, URLs, commands, or executable content — ref is an opaque corpus identifier only.