RepoSage — agentic code-Q&A over a codebase

Ask a repository questions in natural language — "where is auth handled?", "what breaks if I change this function?", "show every caller of X" — and get answers grounded in real source, with file:line citations.

RepoSage exists because generic RAG ("chat with your PDF") fails on code: fixed token-window chunking cuts functions in half, and pure vector search ignores the call graph that actually connects code. RepoSage treats retrieval quality as the engineering problem and is exposed as an MCP server, so it plugs straight into Claude Code / Cursor.

Retrieval pipeline

repo ─► AST-aware chunking ─► hybrid index ─► fusion ─► cross-encoder ─► graph ─► cited
        (function/class/        (dense +       (dense +   rerank         expand    answer
         method units,           BM25)          BM25)     (top pool)     (1-hop)
         calls + imports)

• AST-aware chunking — one chunk per function / method / class (never a half-function), plus the calls and imports each definition makes. Zero-dependency (ast stdlib); tree-sitter multi-language is a v2 backend.
• Hybrid retrieval — dense embeddings + BM25 lexical, min-max normalized and fused with a tunable alpha. Beats either alone on code.
• Cross-encoder rerank — retrieve a wide pool via fusion, then reorder it with a (query, code) cross-encoder for precision (the standard "retrieve wide, rerank precise" second stage). Toggleable; see the eval for its measured, honest effect.
• Call-graph expansion — after search finds the best definition, walk one hop along the call graph to pull in what it calls / what calls it. This is the signal generic RAG cannot provide, and it's what makes impact questions answerable.
• Graceful degradation — if sentence-transformers isn't installed, dense retrieval falls back to a deterministic hashed embedding so the whole system still runs end-to-end on a fresh machine.

MCP tools

Tool	Purpose
index_repo(path)	AST-ingest a repo and build/persist the hybrid index
status()	Index size + active embedding backend
search_code(query, k, hybrid)	Ranked definitions with per-signal scores
get_context(query, k, expand_graph)	Cited context bundle for answering
impact_radius(chunk_id)	Blast radius — who calls this definition

Quickstart

uv venv --python 3.12 .venv           # standard CPython (not free-threaded)
uv pip install -e .                    # core: mcp + rank-bm25 + numpy
uv pip install -e ".[embeddings]"      # optional: real semantic embeddings

Register with Claude Code (from this directory):

claude mcp add reposage -- .venv/Scripts/python.exe -m reposage.server

Then in Claude Code: "index this repo with reposage, then ask where auth is handled."

Evaluation

The differentiator is eval/ — labeled, auditable question sets with an ablation that shows what each layer buys. Two corpora: Flask (pallets/flask, 404 chunks, external — the fair test) and this repo's own src/ (53 chunks, dogfood). Every gold label and call edge is verified against the actual source, not guessed.

# Flask (clone once, then run):
git clone --depth 1 https://github.com/pallets/flask .corpora/flask
python -m eval.run --dataset flask

python -m eval.run                    # dogfood on ./src
python -m eval.run --repo PATH        # any repo (with a matching dataset)

Primary result — Flask (19 questions, 404 chunks, equal 8-result budget)

Config	hit@8	MRR
vector-only	0.79	0.563
+ BM25 fusion	0.89	0.576
+ cross-encoder rerank	1.00	0.680
+ graph expansion	0.84	0.568

By category (hit@8):

Category	vector	+ BM25	+ rerank	+ graph
semantic	0.88	0.88	1.00	0.75
keyword (exact identifiers)	1.00	1.00	1.00	1.00
impact	0.40	0.80	1.00	0.80

What the numbers actually say (the honest read, not a rigged monotonic table):

• The cross-encoder reranker is the big win — on a real corpus. It lifts hit@8 from 0.89 → 1.00 and MRR +0.10, helping both semantic (0.88→1.00) and impact (0.80→1.00). Crucially, the same reranker was a wash on the 53-chunk dogfood corpus (see below) — because a tiny corpus has too few distractors for reranking to matter. The lesson: you cannot fairly evaluate a reranker on a toy corpus. Running both corpora is what surfaced that.
• BM25 fusion pays off on the queries it should — it takes impact queries from 0.40 → 0.80 (exact symbol names) and lifts overall hit to 0.89. Dense handles paraphrase; BM25 handles literal identifiers; fusion gets both.
• Graph expansion does not improve ranking — it slightly hurts it (0.89 → 0.84, semantic 0.88 → 0.75), and that's reported rather than hidden. Fusing graph neighbors into the ranked list displaces real hits. The call graph is a context-enrichment / impact-analysis feature, not a ranking-fusion layer — so it's evaluated on its own job instead:

Call-graph quality (Flask) — 8 hand-verified in-repo call edges:

Metric	Value
caller-recall	1.00
avg callers/node (noise proxy)	3.12

The AST-derived graph recovers every labeled caller edge — which is what makes impact_radius trustworthy on real code.

Secondary — dogfood on own src/ (18 questions, 53 chunks)

Config	hit@8	MRR	note
vector-only	0.78	0.452
+ BM25 fusion	0.89	0.615
+ cross-encoder rerank	0.89	0.618	flat — corpus too small to test rerank
+ graph expansion	0.94	0.622

Kept deliberately: the contrast between "rerank is a wash" here and "rerank wins" on Flask is the finding. Small-corpus metrics are also saturated, so treat them as directional only.

Raw metrics: eval/results_flask.json, eval/results.json.

Status

v0.1 — pipeline + MCP server working end-to-end; real embeddings (all-MiniLM-L6-v2) + cross-encoder rerank (ms-marco-MiniLM-L-6-v2) wired; 4-stage eval ablation on two corpora (Flask + dogfood) with call-graph metrics; 9 pipeline tests passing. On Flask the reranker reaches hit@8 = 1.00. Next: a code-domain reranker, smarter graph-aware context assembly, and tree-sitter for non-Python repos.