wiki-compiler

A CLI + MCP server that compiles web sources into a structured, growing knowledge base — Obsidian-compatible, local-first, schema-driven.

"There is room here for an incredible new product instead of a hacky collection of scripts." — Andrej Karpathy

Most tools give your AI access to your notes. This one gives your AI a compiled knowledge structure — backlinked wiki pages built from raw sources, organized by a schema you define.

---

How it works

wiki add <url>
    │
    ▼
fetch page → strip HTML → send to Claude
    │
    ▼
Claude extracts: title, summary, key_concepts, categories, tags
    │
    ▼
vault/agent/{slug}.md  ← Obsidian-compatible markdown
    │
    ▼
wiki query "what do I know about X?"
    │
    ▼
Claude reads all vault pages → synthesized answer

Dual vault architecture (kepano's contamination insight):

• vault/agent/ — compiler output, messy, experimental
• vault/personal/ — your curated notes, never touched by the tool

---

Install

git clone https://github.com/Ray0907/wiki-compiler
cd wiki-compiler
uv sync
export ANTHROPIC_API_KEY=sk-ant-...

Usage

# Compile a source into your vault
uv run wiki.py add https://github.com/kepano/obsidian-skills

# Written: vault/agent/kepanoobsidian-skills-agent-skills-for-obsidian.md

# Ask a question against everything you've compiled
uv run wiki.py query "What is the Obsidian CLI used for?"

Then open vault/ as an Obsidian vault — every [[wikilink]] in the compiled pages becomes a node in your knowledge graph.

---

Output format

Each compiled page:

---
title: kepano/obsidian-skills
url: https://github.com/kepano/obsidian-skills
summary: A collection of agent skill files that teach AI agents how to read
  and write Obsidian vaults using Markdown, Bases, JSON Canvas, and CLI.
categories:
  - AI
  - Tools
tags:
  - obsidian
  - agents
  - mcp
date_compiled: '2026-04-03'
---
## Summary
A collection of agent skill files...

## Key Concepts
- [[Obsidian CLI]]
- [[Agent Skills]]
- [[JSON Canvas]]
- [[Model Context Protocol]]
- [[Vault Structure]]

---

Schema

schema.yaml controls what the compiler produces. Edit it to match your domain:

categories:
  - AI
  - Tools
  - Research
  - Knowledge Management
  - Agent Architecture

backlinks:
  min_concepts: 2
  max_concepts: 8
  format: "[[{concept}]]"

output:
  filename_pattern: "{slug}.md"
  frontmatter_fields:
    - title
    - url
    - summary
    - categories
    - tags
    - date_compiled

Schema is re-read on every run — no restart needed.

---

MCP Server

Connect to Claude Desktop, Cursor, or any MCP client:

uv run mcp_server.py

Two tools exposed:

• add_document(url) — compile a URL into the vault
• search_wiki(query) — query the vault

Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "wiki": {
      "command": "/path/to/wiki-compiler/.venv/bin/python",
      "args": ["/path/to/wiki-compiler/mcp_server.py"],
      "cwd": "/path/to/wiki-compiler",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

---

Tests

uv run pytest tests/ -v
# 16 passed in 0.45s

---

Why not an Obsidian plugin?

Obsidian plugins can't run heavy LLM pipelines, are tied to one app, and the Obsidian team is already building native AI. The value here is the compiler — the pipeline that turns raw sources into a coherent, schema-consistent knowledge structure. Obsidian is just a convenient viewer for the output.

The MCP server makes the compiled vault available to any LLM client. As models improve, the compilation gets better — the schema and accumulated knowledge stay yours.

---

Roadmap

• [ ] wiki compile — batch mode, multiple sources at once
• [ ] Human review gate — promote pages from vault/agent/ to vault/personal/
• [ ] wiki report — autonomous mode: one query spawns a team of agents, builds an ephemeral wiki, returns a full report
• [ ] Obsidian CLI integration — use backlink graph for smarter query routing