repolens

Turn any repository into a deterministic, token-budgeted, AST-aware context pack for LLM agents — with a hash-chained provenance row for every file it reads.

repolens is a zero-dependency Python tool (standard library only for the core) and an MCP server. It exists for one job: give an autonomous agent exactly the context it can afford, in a form that is reproducible, auditable, and structurally aware of the code it describes.

A Cognis Digital flagship tool.

---

Why these properties matter

Most "AI code context" tooling optimizes for a compression headline and produces output that is nondeterministic, unbudgeted, structurally blind, and unauditable. When that output feeds an agent that edits your code, four different properties are what actually matter:

1. Reproducibility. The same repository bytes must always yield the same pack — byte for byte. That makes packs diffable in review, cacheable in CI, and safe to reason about. If a pack changes, it is because the repository changed, not because a model or a clock did.
2. An exact token budget. An agent has a finite context window and a finite spend. "Roughly smaller" is not a budget. repolens fits a repository into a precise token ceiling and degrades gracefully when it must.
3. Real structure. Signatures, imports, and call edges are worth far more per token than raw text. A language-aware map lets the pack say what the code is even when there is no room to include all of the code.
4. A tamper-evident record. When something goes wrong downstream, you need to prove precisely what context was produced from which bytes. repolens emits a hash-chained provenance manifest that makes any alteration detectable.

repolens delivers all four.

---

Features

• Deterministic context pack. Stable ordering plus content addressing: the same repository bytes always produce a byte-identical pack. The pack's digest is the head of its provenance chain, so equal digests provably mean equal inputs read in the same order.
• Exact token budgeting with graceful degradation. Give --budget N and repolens fits the repo into N tokens, degrading each file through full body → signatures → names → omit in priority order. A pluggable tokenizer abstraction ships with a deterministic heuristic estimator by default and a character-ratio estimator; real model tokenizers can be registered without touching the pipeline.
• AST-aware maps. Python is parsed with the standard-library ast for exact symbols, signatures, imports, and intra-module call edges. JavaScript/TypeScript, Go, Rust, and Java use lightweight, clearly structured heuristic extractors that report symbols, kinds, signatures, and imports — designed so a full parser can be dropped in behind the same interface later.
• Keyless relevance ranking. A dependency-free BM25 implementation ranks files against a free-text query so the budget is spent on what matters. An optional local-embedding ranker is a clearly marked extension point; BM25 is always available and needs no keys, downloads, or network.
• Hash-chained provenance. Every file read emits a SHA-256 audit row (index, path, content_hash, size), and each row commits to the previous row's hash. Reordering, inserting, deleting, or altering any row breaks the chain — and verify reports the exact first bad row.
• Multiple output formats. Compact text, JSON, Markdown, and XML-tagged prompt blocks. .gitignore-aware discovery, and a secret-redaction pass strips common key/token formats before anything leaves the repository.
• MCP server. A minimal, self-contained JSON-RPC/stdio MCP server exposes a pack tool (and a verify tool). Agents request a budgeted context pack for a path or query and get the manifest back.
• Honest benchmarks. A benchmark harness compares repolens against a naive find + concatenate baseline on bytes, tokens, and wall-clock.

---

Install

pip install .
# or, for development:
pip install -e ".[dev]"

Requires Python 3.11+. The core has zero third-party runtime dependencies; pytest is used only for tests.

---

CLI

# Build a compact text pack that fits in 8000 tokens
python -m repolens pack . --budget 8000

# Rank against a query and emit XML-tagged prompt blocks
python -m repolens pack . --query "auth middleware" --budget 4000 --format xml

# JSON pack (includes the full provenance manifest) written to a file
python -m repolens pack . --format json -o pack.json

# Re-verify a JSON pack's provenance chain
python -m repolens verify pack.json

# Benchmark against the naive find + cat baseline
python -m repolens bench . --budget 8000

# Run the stdio MCP server
python -m repolens mcp

A repolens console entry point is installed as well, so repolens pack . works once installed.

Useful flags

Flag	Effect
-b, --budget N	Exact token budget (omit for unlimited)
-q, --query STR	BM25 relevance query
-f, --format	text (default), json, markdown, xml
-t, --tokenizer	heuristic (default), char, or char:N
--no-gitignore	Do not honor .gitignore
--no-redact	Disable secret redaction
--include-binary	Include binary files
--no-manifest	Omit the provenance manifest from output
--max-file-bytes N	Truncate file bodies larger than N bytes (provenance still hashes full bytes)
--follow-symlinks	Follow symlinks (off by default so a pack cannot escape the repo)
--stats	Print pack statistics to stderr

---

Library API

from repolens import pack_repo, render

pack = pack_repo("path/to/repo", query="database pool", budget=8000)

print(render(pack, "markdown"))
print("digest:", pack.digest)
print("provenance verified:", pack.manifest.verify().ok)
for f in pack.included_files:
    print(f.relpath, f.language, f.level, f.tokens)

Everything is a pure data object: pack_repo(...) returns a ContextPack, and the formatters are pure functions of it.

---

Determinism, precisely

Two runs over the same bytes produce identical output because every stage is deterministic:

• Files are discovered in sorted, forward-slash path order.
• Provenance rows are recorded in that order and chained by SHA-256.
• Ranking ties break by path; budgeting is a deterministic greedy pass.
• JSON is emitted with sorted keys and fixed separators; all formats use \n newlines.

The included test suite proves byte-identical output across repeated runs and across independent copies of the same content.

---

Provenance model

Each row commits to its own fields and the previous row's hash:

row_hash = sha256("repolens-provenance-v1\n" + index + "\n" + path + "\n"
                  + content_hash + "\n" + size + "\n" + prev_hash)

The first row's prev_hash is 64 zero hex characters (the genesis sentinel). The manifest head — the last row's hash — is a single value that commits to the whole read. An auditor can recompute the entire chain from the raw repository bytes.

---

Extending

• Tokenizers. Subclass repolens.tokenizer.Tokenizer, implement count, and call register_tokenizer(name, factory).
• Language extractors. Implement the Extractor interface in repolens.languages.base and register it; the heuristic extractors are structured so a real parser can replace any of them without changing the rest of the system.
• Ranking. embedding_rank is the hook for an optional local-embedding ranker; BM25 remains the keyless default.

---

Testing

python -m pytest -q

The suite covers determinism, token budgeting and graceful degradation, each AST/heuristic extractor, .gitignore semantics, binary/unicode/empty/huge-file edge cases, secret redaction, the provenance chain and its tamper cases, the four output formats, the benchmark harness, the CLI, and the MCP endpoint.

---

License

MIT © 2026 Cognis Digital LLC. See LICENSE.