Distributed Memory System

Cross-project memory for Claude Code — a learning written while working in one project becomes recall-able in every project, without leaking project-specific facts or secrets.

Claude Code's native memory is keyed per repository: what you learn in project A is invisible in project B. This system layers a shared, git-versioned markdown store plus local semantic recall on top — keeping the good per-project notes you already write, while making the reusable ones available everywhere, and enforcing at the write path that secrets and PII never escape their scope.

It is small (617 lines of Python across 8 files), fully local (no API calls at runtime), and the entire index is a disposable cache you can delete and rebuild from the markdown at any time.

---

It works — and here's the receipt

This isn't a demo. The store below is live. It was seeded once by migrating a real engineer's accumulated Claude Code memory (24 project stores + a cross-project engineering knowledge base + global instructions → 203 raw candidates → 150 gated, de-duplicated memories), and has since been growing on its own as that engineer works:

	Count
Total memories	160
— seeded by migration	150
— captured organically since activation (memory_write during real sessions)	10
Global (cross-project)	74
Project-scoped	86 across 32 project scopes
By type	project 70 · reference 61 · procedural 21 · feedback 5 · user 3
By domain	ai 64 · infra 29 · db 23 · design 18 · general 16 · bioinfo 10
By sensitivity	internal 83 · public 75 · secret 2 (both project-scoped, redacted at rest)

The value loop, observed in the wild. In the hours after activation, an engineer rebuilt a Next.js app ("Aurelia") following the built-in sprint workflow. The system captured, unprompted: a deep-research brief, a spec-first test surface, testing gotchas — and five generalizable gotchas it correctly promoted to global scope, e.g.:

drizzle_neon_http_has_no_interactive_transactions_use_neon_serverless — "Drizzle neon-http throws on db.transaction(); use neon-serverless (WebSocket) for atomic writes on Vercel. Local dev = node-postgres; prod = neon-serverless; only the driver import swaps."

That memory was learned in one project. Its body ends with [[nextjs_local_node_postgres_not_neon_http]] — an automatic link to a memory that was migrated from a different project (TubeIntel). Two projects, two points in time, one connected knowledge graph. That is the whole thesis in a single artifact.

There's even a memory the system wrote about itself — use_mcp_memory_db_not_memory_md — a learning that the MCP store, not scattered MEMORY.md files, is now the source of truth.

---

Value propositions

• Cross-project recall. A gotcha, convention, or decision written anywhere is retrievable everywhere — by meaning, not just keywords.
• Security enforced on the write path, not the reader's discretion. Every write is scanned for secrets, PII, and machine-local paths. global scope is provably free of secrets and PII (gate-enforced); project scope may hold sensitive facts, but secrets are redacted at rest. A pre-commit hook is the belt to the gate's suspenders.
• Local and offline. Embeddings run on-device (fastembed, ONNX/CPU). No data leaves the machine; no per-query API cost.
• Markdown + git is the source of truth. Every memory is a plain, diffable, reviewable file. The search index is a derived SQLite cache — delete it and rebuild from the markdown at any time.
• Scoped, not global-by-default. global for reusable-everywhere knowledge; project:<name> for the rest. Promotion (project → global) is propose → approve, with provenance retained.
• Auditable. 617 lines of Python, no framework magic, every decision inspectable.

---

Architecture

Two planes: a source-of-truth plane (markdown in git) and a derived-index plane (SQLite). The index is always reconstructible from the source; the source never depends on the index.

flowchart TB
    subgraph SOT["Source of truth — versioned in git"]
        MD["store/**/*.md<br/>one memory = one file<br/>frontmatter + body"]
    end
    subgraph DERIVED["Derived index — gitignored, rebuildable"]
        FTS["memories_fts<br/>SQLite FTS5 · BM25"]
        VEC["embedding BLOBs<br/>384-dim float32 · normalized"]
    end
    subgraph ENGINE["server/ — 617 LOC"]
        IDX["indexer.py<br/>content-hash cached"]
        SRCH["search.py<br/>hybrid + MMR"]
        GATE["gate.py<br/>sensitivity scanner"]
        MCP["mcp_server.py<br/>5 tools over stdio"]
    end
    CC["Claude Code<br/>(any project)"]

    MD -->|"parse + embed<br/>(only changed files)"| IDX --> FTS & VEC
    CC <-->|"memory_search / memory_write / …"| MCP
    MCP -->|recall| SRCH
    SRCH -->|reads| FTS & VEC
    MCP -->|"write path"| GATE -->|"redacted, scoped"| MD
    MCP -.->|"reindex once after write"| IDX

The write path — where security lives

Secrets are contained at the source, not left to the reader:

flowchart LR
    W["memory_write<br/>(title, body, scope…)"] --> SCAN["gate.scan()<br/>7 secret · 2 PII · 2 path rules"]
    SCAN --> CHK{"scope == global<br/>AND secret/PII?"}
    CHK -->|yes| BLOCK["❌ blocked<br/>'use a project scope'"]
    CHK -->|no| RED["redact_secrets()<br/>mask value, keep structure"]
    RED --> ESC["escalate sensitivity<br/>to match findings"]
    ESC --> FILE["write store/…/slug.md<br/>UTF-8, with frontmatter"]
    FILE --> RI["reindex once"]

• global scope is gate-enforced clean. A write to global containing a secret or PII is rejected, not silently downgraded. (Verified across all 74 global memories in this store: zero secret/PII findings.)
• Redact at rest, keep the shape. A DSN like postgresql://user:pass@host/db is stored as postgresql://user:«REDACTED:db-password»@host/db — the locator survives, the secret doesn't.
• Sensitivity auto-escalates. If you label a memory public but the scanner finds PII, it's promoted to internal before it's written. You can't under-classify by accident.

The read path — hybrid recall

Semantic similarity finds what you meant; keyword search anchors exact terms (library names, error strings, flags). Blended, min-max normalized, then diversified with MMR so the top-k aren't near-duplicates:

score = 0.7 · cosine(query, memory)      # vector — semantic
      + 0.3 · bm25(query, memory)         # FTS5  — lexical

Real query against the live store — "drizzle neon transaction on vercel":

memory	score	vec	bm25
drizzle_neon_http_has_no_interactive_transactions…	1.00	1.00	1.00
pglite_hermetic_drizzle_postgres_integration_tests	0.67	0.76	0.46
tubeintel_stack_and_architecture	0.55	0.60	0.43

The exact learning surfaces first, with a related testing memory and the originating project's architecture right behind — the shape you want for "remind me what I know about X."

---

Design decisions & nuance

The interesting engineering is in the why. Each choice below traded something for something.

1. Markdown + git is truth; SQLite is a cache. Memories are durable, human-reviewable, and diff cleanly in PRs. The index (FTS5 + embedding BLOBs) is gitignored and disposable: rm index.db && python server/indexer.py fully rebuilds it. This means no schema migrations to fear, no lock-in, and a store you can hand-edit or grep. The cost is a rebuild step — paid for by content-hash caching (below).

2. Content-hash caching — only re-embed what changed. Each file's SHA-256 is stored alongside its vector. On reindex, an unchanged file reuses its cached embedding; only new/edited files hit the model. The bulk migration re-embedded 150 files once; every subsequent memory_write re-embeds exactly one.

3. Sensitivity gate on the write path, not the read path. Filtering secrets at read time trusts every reader forever. Gating at write time means the dangerous data never lands in a shareable scope in the first place — and the guarantee is structural, not behavioral. The gate is deliberately layered: the MCP tool calls it, and a portable git pre-commit hook re-runs it on anything added outside the tool.

4. The gate is heuristic — and was hardened by real data. The shipped patterns caught password:-style secrets and /home/ paths. Mining a real corpus exposed two blind spots it would have leaked to global: DSN-embedded passwords (postgresql://user:pass@…) and Windows absolute paths (C:\…). Both are now rules (7 secret · 2 PII · 2 path). This is the honest posture: a gate is a strong default, not a DLP guarantee — so it's designed to be extended, and it was.

5. Scopes + propose→approve promotion. global knowledge is small, clean, and always-on; project:<name> knowledge is abundant and may be sensitive. Promotion is never automatic: memory_promote returns a gate verdict and a proposed move, but a human approves it. Provenance (source_project) is retained so a global memory always remembers where it was learned.

6. Hybrid retrieval + MMR, with fixed, legible weights. 0.7/0.3 vector/BM25 and MMR λ=0.7 are constants, not a tuned model — chosen because they're explainable and good enough, and because a memory system's failure mode should be "returned something slightly off," never "silently mis-ranked by an opaque scorer." Both signals are min-max normalized per query so neither dominates by scale.

7. Local 384-dim embeddings (bge-small-en-v1.5). 384 dimensions (1536 bytes/vector) is the sweet spot for a personal store of hundreds–thousands of memories: strong retrieval, tiny footprint, fast on CPU, no API dependency. Vectors are L2-normalized so cosine similarity is a single dot product.

8. stdio protocol discipline. An MCP stdio server must never write to stdout — it corrupts the JSON-RPC stream. The indexer prints progress, so the server redirects its stdout to stderr around every reindex. Small detail, total protocol failure if missed.

9. UTF-8, always. The store legitimately contains em-dashes, arrows, and the «REDACTED» guillemets. Every file read and write pins encoding="utf-8" — because relying on the platform default (cp1252 on Windows) crashes memory_write the moment a memory contains one of those characters, and corrupts git blobs even when it doesn't.

---

The always-on layer

The store holds hundreds of memories, but only a lean index is ever loaded into a session — full bodies are fetched on demand via memory_search. regen_rules.py generates ~/.claude/distributed-memory.md: the usage protocol plus a one-line entry per global memory, imported by ~/.claude/CLAUDE.md. This keeps every session cheap while making the whole store reachable in one tool call.

## Global memories (74) — one-line index; call `memory_search` for full detail
- drizzle_neon_http_has_no_interactive_transactions… [db] — Drizzle neon-http has no interactive
  transactions; use neon-serverless (WebSocket) for atomic writes on Vercel
- fastapi_spa_mount_order [infra] — Mount StaticFiles(html=True) LAST, after every API/WS route…
- …

---

Schema

One markdown file = one memory. Frontmatter + body. Compatible with Claude Code auto-memory (name / description / metadata.type) and extended for cross-project recall.

---
name: <slug = filename stem>
description: "<one sharp sentence — this is what recall ranks on>"
metadata:
  type: user | feedback | project | reference | procedural
  scope: global | project:<name>
  domain: db | bioinfo | ai | design | infra | general
  sensitivity: public | internal | secret
  source_project: <origin repo>          # provenance, retained through promotion
  origin_session: migration | mcp | …
  created: <YYYY-MM-DD>
---
<body — "Why:" / "How to apply:" encouraged; [[links]] to related memories>

• description is load-bearing: recall ranks on it and the always-on index shows it verbatim.
• scope: global MUST be secret/PII-free (gate-enforced) and should avoid machine-absolute paths.
• sensitivity: secret never leaves project scope.
• [[name]] links reference another memory's slug — this is how the knowledge graph forms.

Average body length in the live store: ~506 characters — sharp and single-fact, not essays.

---

MCP tools

Registered once at user scope, available in every project:

Tool	Purpose
memory_search(query, scope?, domain?, project?, k?, max_sensitivity?)	Hybrid recall by meaning + keywords.
memory_write(title, body, type, domain, sensitivity, scope|project, links?)	Gated write; redacts secrets, escalates sensitivity, reindexes.
memory_promote(name)	Proposal only — returns the gate verdict for a project→global promotion; a human approves.
memory_reindex()	Rebuild the index from the markdown store.
memory_status()	Counts by scope/domain + the embedding model.

MCP stdio servers must never write to stdout — it corrupts the protocol. Keep all logging on stderr.

---

Component map

File	LOC	Role
server/embed.py	21	Local embeddings (fastembed, ONNX/CPU)
server/mem.py	42	Frontmatter + body parsing, content hashing
server/precommit.py	55	Pre-commit sensitivity gate (belt-and-suspenders)
server/regen_rules.py	59	Generate the always-on protocol + global index
server/gate.py	78	Secret / PII / path scanner + redactor
server/indexer.py	80	Build/refresh index.db, content-hash cached
server/search.py	119	Hybrid BM25 + vector recall, MMR-diversified
server/mcp_server.py	163	MCP server: the 5 tools, over stdio
Total	617

---

How the initial corpus was built (a case study in itself)

The 150 seed memories weren't hand-written — they were mined from an engineer's real, scattered memory artifacts, which is a nice demonstration of the multi-agent workflow the system now recommends:

1. Fan-out mining — 6 parallel agents read 40+ source artifacts (24 project MEMORY.md stores, a large cross-project knowledge base, global instructions, in-repo CLAUDE.md files) and extracted 203 candidate memories in a structured schema.
2. Normalization — duplicates merged (e.g. python -m pip appeared 4×), a fragmented user profile consolidated, and a name collision resolved (two distinct projects both called "NEXUS" → separate scopes).
3. Security triage — a shared Postgres credential found across ~10 projects was collapsed into a single secret memory (password omitted, since it was being rotated); ~20 machine paths kept in project scope but stripped from every global.
4. Gated bulk write — every candidate passed through the same gate.py used by memory_write, then a single reindex — with a post-write assertion that no global memory carries a secret or PII.

The full step-by-step process — the six mining agents (with telemetry), the consolidation decisions, the security triage, and verification — is documented in docs/MIGRATION.md.

---

Quick start

git clone <this-repo> && cd claude-memory-system
python -m venv .venv

# Windows:            .venv/Scripts/python -m pip install -r requirements.txt
# macOS/Linux:        .venv/bin/pip install -r requirements.txt

# Build the index from the markdown store (first run downloads the embedding model, then offline)
.venv/Scripts/python server/indexer.py        # or .venv/bin/python

# Try a recall
.venv/Scripts/python server/search.py "postgres transaction on vercel"

# Install the sensitivity pre-commit gate (portable; detects .venv on Windows or POSIX)
cp hooks/pre-commit .git/hooks/pre-commit      # or: ln -sf ../../hooks/pre-commit .git/hooks/pre-commit

# Wire the MCP server into Claude Code (user scope = available in every project)
claude mcp add memory -s user -e PYTHONUTF8=1 -- \
  /abs/path/.venv/Scripts/python.exe /abs/path/server/mcp_server.py

# Generate the always-on protocol + global index, and import it
.venv/Scripts/python server/regen_rules.py

---

Limitations & honest trade-offs

Good engineering names its edges:

• The gate is a strong heuristic, not a DLP system. It catches common secret/PII/path shapes and is easy to extend, but a novel secret format can slip past. global scope is the hard boundary; treat project scope as "may contain sensitive facts."
• Single-user by design. No auth, no concurrency control beyond git. It's a personal/pair store.
• Retrieval weights are fixed defaults, not learned. Legible and good enough; not SOTA ranking.
• The always-on global index grows with the global count. At 74 lines it's cheap; a store with thousands of global memories would want tiering. Project memories don't have this cost — they're fetched only on demand.
• 384-dim embeddings favor footprint and speed over the last few points of retrieval accuracy a larger model might buy.

---

Model

markdown + git = source of truth · SQLite = derived, disposable index · local offline embeddings · hybrid vector + BM25 recall, MMR-diversified · sensitivity gate on the write path · scoped, with human-approved promotion · 617 lines, fully auditable.

License

MIT — see LICENSE.