corpus-search-mcp

Point it at a folder of markdown and get a local, private, hybrid-search MCP server over your own notes. No cloud API, no per-query cost: chunk your markdown, embed it locally with Ollama, search it with BM25 + vector fusion, and wire it into Claude Code (or any MCP client) as three tools: search, list_sections, get_section.

The hard parts are built and corpus-agnostic: hybrid retrieval, Reciprocal Rank Fusion, an embedding-model-down fallback, fuzzy section lookup, incremental re-indexing, and a runnable quality gate. The only things you tune are a small config.py and, if your corpus is unusual, a chunker.

A tiny example corpus (coffee-brewing notes) is bundled so the eval gate runs out of the box.

How it works

your markdown  ->  chunker  ->  Ollama embeddings + BM25  ->  LanceDB  ->  MCP tools

File	Role
search_core.py	The engine: query expansion, BM25 + vector search, RRF, formatting, and a pure-stdlib keyword index used as the dependency-free fallback
server.py	Generic FastMCP server exposing search / list_sections / get_section
indexer.py	Embeds chunks into LanceDB with a BM25 index; incremental, hash-tracked re-index
config.py	Everything corpus-specific, driven by environment variables
chunkers/	How to split your corpus. h2 (default) and dispatch (advanced) built in
eval.py	Self-contained retrieval quality gate (BM25 keyword mode, no services)
examples/corpus/	A small bundled markdown corpus so the eval runs meaningfully

Install

python -m venv .venv
. .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

You also need Ollama running with an embedding model, for indexing and for vector search at query time:

ollama pull mxbai-embed-large

Search still works without Ollama (BM25 / keyword only); the vector half simply switches off.

Configure

Everything is set by environment variables with sane defaults (see config.py). The one you will always set is the corpus folder:

export CORPUS_DIR=/path/to/your/markdown      # default: examples/corpus
export CHUNKER=h2                             # or "dispatch"; default h2
export OLLAMA_URL=http://localhost:11434/api/embed
export EMBED_MODEL=mxbai-embed-large

Then open config.py and edit SYNONYMS to your domain vocabulary (optional; an empty map disables query expansion).

Index

python indexer.py                 # full build
python indexer.py --changed       # re-index only files whose content changed
python indexer.py --file note.md  # re-index a single file (h2 chunker)

The index and a _index_meta.json of per-file content hashes are written under CORPUS_DIR/.vectordb by default.

Serve

python server.py                  # runs the MCP server over stdio

Wire into Claude Code

Register the server, pointing it at your corpus:

claude mcp add corpus-search \
  -e CORPUS_DIR=/path/to/your/markdown \
  -e CHUNKER=h2 \
  -- python /path/to/corpus-search-mcp/server.py

Use the venv's Python if you installed into one. After it registers, the search, list_sections, and get_section tools are available in your session.

Eval

python eval.py            # exits 0 when Recall@5 / P@1 / MRR meet the floors
python eval.py -v         # also print failing cases

The gate is self-contained: it builds an in-memory BM25 index from the bundled corpus and needs no LanceDB, no embedding model, and no network. Rewrite TEST_CASES (and point CORPUS_DIR at your corpus) to make it a real regression gate for your own material.

Chunkers

How to split your corpus is the one bespoke piece. Two are built in:

• h2 (default) -- split every markdown file at its ## headers. Best for a flat folder of topic notes.
• dispatch (advanced) -- classify each file by the subfolder it lives in (notes/, articles/, references/) and apply a different strategy per type (whole-file, H2, H3). The template for a corpus that mixes document kinds.

To add your own, drop a module in chunkers/ exposing chunk_all(corpus_dir) -> list[dict] (each dict needs id, file, heading, heading_path, line_start, line_end, text, hash) and register it in chunkers/__init__.py.

License

MIT. See LICENSE.