LNM Brain

A self-hosted, persistent "second brain" for your AI coding assistants and chat tools.

LNM Brain automatically captures every conversation you have with AI assistants (Claude Code, OpenCode, ChatGPT, Google Antigravity, etc.), compresses it into structured, searchable knowledge, and serves it back on demand — so your AI never starts a session from zero again.

It runs entirely on your own infrastructure (Cloudflare Workers + a GitHub repo you own), so your data stays yours.

---

What This Is

Every AI conversation you have is a goldmine of context: decisions made, dead ends avoided, facts learned about your projects. Normally that context dies the moment the chat window closes. LNM Brain fixes that.

It's an implementation of the "LLM Wiki" pattern: instead of re-feeding an LLM your entire raw conversation history (expensive, slow, and eventually impossible as history grows), the system:

1. Captures every conversation as an immutable raw record.
2. Compresses it with an LLM into structured wiki pages (~27x fewer tokens than the raw transcript).
3. Indexes it across multiple retrieval strategies — semantic vectors, keyword search, entity graphs, and fact triples.
4. Serves the most relevant compressed knowledge back to any AI assistant via a simple API or MCP tools, in a fraction of the tokens raw history would cost.

The result: an AI assistant that remembers your projects, your decisions, and your preferences across sessions, tools, and even different AI providers — without you managing any of it manually.

Why This Exists

• Context doesn't survive across sessions. Every new chat starts blank, even about a project you discussed yesterday.
• Context doesn't survive across tools. What you tell Claude Code, ChatGPT doesn't know, and vice versa.
• Re-reading raw history is expensive. Feeding 50,000 tokens of old transcript into every new conversation isn't sustainable or accurate — signal gets buried in noise.
• You want to own your data. This isn't a hosted SaaS memory product; it's your GitHub repo and your Cloudflare account.

Benefits

Cross-session memory	Every AI assistant can recall past decisions, facts, and context instantly.
Cross-platform	Works with Claude Code, OpenCode, ChatGPT, Google Antigravity, and anything that can call an HTTP API or MCP tool.
Massive token savings	~27x fewer tokens than re-reading raw conversation history; graph traversal is ~71x fewer.
Hybrid retrieval	Combines semantic search (vector embeddings), keyword search (D1 FTS5), entity/fact lookups, and knowledge-graph traversal, fused via reciprocal rank fusion — more accurate recall than any single method alone.
You own the data	Everything lives in your own GitHub repo (version-controlled, auditable) and your own Cloudflare account (Workers, KV, Vectorize, D1). No third-party memory service.
Obsidian-compatible	The generated wiki is plain markdown — open it as an Obsidian vault for local browsing, backlinks, and graph visualization.
Free-tier friendly	Designed to run within Cloudflare's free tier (Workers, KV, Vectorize, D1) plus free-tier LLM providers for compression/embedding.

Architecture

                 ┌─── INGEST ────────────────────────────────────┐
client ─▶ /ingest │ GitHub raw + wiki write                       │
                  │ obs:meta + recent:all + recent:{surface}      │ SYNC
                  │ D1 FTS5 keyword index                         │
                  │ ranking triples (fact extraction)             │
                  │ Vectorize multi-vector embeddings             │
                  │   (title | content | summary | entity)        │
                  ├──────────────────────────────────────────────┤
                  │ fact extraction + routing index (async)       │ BACKGROUND
                  └──────────────────────────────────────────────┘

           query
             │
             ▼
   ┌─── /recall (hybrid retrieval) ─────────────────┐
   │  semantic  (Vectorize, all namespaces)   ──┐    │
   │  keyword   (D1 FTS5 → GitHub fallback)   ──┤ RRF│ + recency boost
   │  entity    (structured entity facts)     ──┤    │
   │  triples   (subject-predicate-object)    ──┘    │
   └──────────────────────────────────────────────────┘
                           │
                           ▼
                    top-K results with provenance

Repo layout

raw/              # Immutable source documents (conversations, code, web clippings)
  conversations/  # Verbatim conversation dumps (dated)
  code/           # Code artifacts produced in conversations
  assets/         # Images, screenshots, diagrams
  web/            # Web clippings, articles
wiki/             # LLM-generated structured markdown
  topics/         # Concept pages
  entities/       # People, tools, companies
  projects/       # Active projects
  synthesis/      # Cross-topic analysis
  questions/      # Open questions
  sources/        # Source stubs (leaf nodes)
graph/            # Knowledge graph JSON (nodes + edges)
scripts/          # CLI tools (capture, query, compress, sync, eval)
worker/           # Cloudflare Worker — the capture/query/recall API + MCP server
web/              # Browser-based capture tool
codex-plugin/     # Codex/OpenCode plugin for auto-capture
migrations/       # D1 schema migrations
docs/             # Deep-dive documentation (Karpathy pattern, platform integration)

Note: raw/, wiki/, graph/graph.json, index.md, overview.md, and log.txt are your personal data — this template repo does not ship any of that content. They're generated as you use the system.

How It's Used

1. Deploy the Worker (Cloudflare)

cd worker
npm install
echo 'GITHUB_TOKEN = "your-github-personal-access-token"' > .dev.vars
# Edit wrangler.toml: create your own KV namespace, Vectorize indexes, and D1 database
#   wrangler kv namespace create VECTORS
#   wrangler vectorize create lnm-brain-m3 --dimensions=1024 --metric=cosine
#   wrangler d1 create lnm-brain-fts
wrangler secret put BRAIN_API_KEY   # the key clients will authenticate with
wrangler deploy

2. Point it at your own GitHub repo

Fork or create a repo to hold your captured knowledge, then set:

# worker/wrangler.toml
[vars]
GITHUB_REPO = "your-username/your-second-brain-data"
GITHUB_BRANCH = "main"

3. Capture conversations

• Auto-capture: install the codex-plugin/second-brain plugin for OpenCode/Codex, or use AGENTS.md / CLAUDE.md as system-prompt instructions so the assistant calls the capture API at the end of every session.
• CLI: echo '{"type":"conversation","title":"my-chat","content":"...","tags":["tag1"]}' | python3 scripts/capture.py
• Web: open web/capture.html for a simple browser-based capture form.
• ChatGPT export: use scripts/parse_chatgpt.py to ingest exported conversation JSON.

4. Query the brain

python3 scripts/query.py "your search term"
# or hit the API directly
curl "https://your-worker-subdomain.workers.dev/recall?q=your+search+term&key=$BRAIN_API_KEY"

5. Connect it as an MCP server

Any MCP-compatible client (Claude Code, Claude Desktop, Codex, etc.) can connect directly:

https://your-worker-subdomain.workers.dev/mcp?key=YOUR_API_KEY

Exposed tools include keyword_search, recall_brain, session_context, and more — see PLATFORM_INTEGRATION.md.

6. View the knowledge graph

Open the vault in Obsidian for local graph view and backlinks, or visit the deployed worker's root URL for a web-based graph visualizer.

Documentation

• PLATFORM_INTEGRATION.md — Integrating with Claude Code, OpenCode, OpenAI Code, Google Antigravity, ChatGPT.
• KARPATHY_INTEGRATION.md — The LLM Wiki pattern, RAG, and knowledge-graph design principles behind the system.
• AGENTS.md — Auto-capture behavior spec (read by OpenCode/Codex-style agents).
• CLAUDE.md — Auto-capture + Second Brain integration instructions for Claude Code.
• docs/RITIK-PROCESS-v9.5.0.md — Worked example of the day-to-day operating process.

Token Savings

Method	Tokens per query	Savings
Raw re-read	~50,000	baseline
Wiki pages	~1,850	27x
Graph traversal	~700	71x

Requirements

• A Cloudflare account (free tier is sufficient to start): Workers, KV, Vectorize, D1, and Workers AI.
• A GitHub repo to store captured knowledge (private is recommended — this is your data).
• Python 3.9+ for the CLI tooling (pip install -r requirements.txt).
• Node.js (LTS) + wrangler for deploying the worker.
• Optional: an LLM API key (OpenRouter, NVIDIA NIM, Cloudflare Workers AI is free) for compression and reranking.

Security Notes

• Treat BRAIN_API_KEY / GITHUB_TOKEN as secrets — set them via wrangler secret put or GitHub Actions secrets, never commit them.
• Everything you capture is stored verbatim in your configured GitHub repo — keep that repo private unless you intend your captured conversations to be public.
• This template repo intentionally ships no example captured data (no wiki/, raw/, or graph/graph.json content) — you generate your own as you use it.

License

MIT — see LICENSE.

Contributing

Issues and PRs welcome. This started as a personal tool; contributions that make it more general-purpose or easier to self-host are especially appreciated.