ytbrain

Turn a YouTube channel into a queryable local knowledge base.

Index every video's captions into a local SQLite/FTS5 brain, then search it with BM25 and ask it questions — answers are sentences quoted from the transcripts, each with a timestamped link that jumps to the exact moment in the video. Works from the CLI and as an MCP server for Claude Desktop, ZCode, Cursor, ...

channel/playlist URL --yt-dlp--> video list --captions--> segments
                                                          |
              ask "why did they drop postgres?"    SQLite + FTS5 (BM25, porter)
                          \                               |
                           `------> quoted sentences -----+
                                    with [MM:SS] + &t= jump links

Where youtube-transcript-mcp answers "what does this one video say?", ytbrain answers "where across this whole channel was X explained?".

Highlights

• Channel-scale, not per-video: ingest a channel, playlist or single video; resumable — interrupted runs continue where they stopped, re-ingest replaces.
• Caption-first: uses YouTube's own manual/auto captions (json3 preferred, vtt fallback with rolling-window dedup). No audio downloads, no whisper, no GPU.
• Zero API keys, fully local: yt-dlp as a library + httpx + SQLite FTS5. The brain is one file you can drop, copy, or delete.
• BM25 search with porter stemming (databases finds database), phrase queries ("write ahead log"), filters by video or channel.
• Extractive Q&A: ask ranks passages, quotes the best-matching sentences and cites each with [MM:SS] + a &t=SECONDSs deep link. Sentences are extracted, never generated — no LLM, no paraphrasing.
• MCP server (5 read-only tools) so any MCP client can query the brain.
• Honest by construction: no-caption videos are reported and skipped, unanswerable questions say so, nothing is invented.

Install

python -m venv .venv                     # Python 3.10-3.13
.venv/Scripts/activate                   # Windows; source .venv/bin/activate elsewhere
pip install -e .                         # or: pip install -e ".[dev]" for pytest

Quickstart

# 1. build a brain (limit optional — omit it for the whole channel)
ytbrain ingest https://www.youtube.com/@Fireship/videos --limit 20

# 2. search it
ytbrain search "open source models"

# 3. ask it
ytbrain ask "what is the problem with robot demos"

# 4. inspect
ytbrain stats
ytbrain videos
ytbrain transcript VIDEO_ID

Verified end-to-end run (3 videos ingested, 567 segments, 0.17 MB database):

$ ytbrain ask "what is the problem with robot demos"
Q: what is the problem with robot demos
A (extractive — sentences quoted from the transcripts):

  "When you read the fine print of virtually any robot demo, you'll find that
   multi-finger dexterity success rates range anywhere from 0% to 90%, and
   that's a big problem because nobody wants to buy a Rosie the Robot maid who"
   — I spent 3 days at MIT... the robot hype is worse than you think [02:35]
     https://www.youtube.com/watch?v=aB5LGrHISqY&t=155s

$ ytbrain search "open source"
1. [02:08] Meta's new model wants "deep access" to your personal life...
   https://www.youtube.com/watch?v=G55HSGpuh1M&t=128s
   and abandon >>open<< >>source<< entirely.

MCP server

Build the brain with the CLI, then serve it read-only to MCP clients:

{
  "mcpServers": {
    "ytbrain": {
      "command": "C:\\path\\to\\ytbrain\\.venv\\Scripts\\ytbrain.exe",
      "args": ["mcp"],
      "env": { "YTBRAIN_DB": "C:\\path\\to\\ytbrain\\data\\ytbrain.sqlite3" }
    }
  }
}

Tool	Purpose
ytbrain_search	BM25 search; FTS5 syntax ("exact phrase", OR, prefix*), video/channel filters.
ytbrain_ask	Extractive Q&A: quoted sentences + timestamped citations.
ytbrain_get_transcript	[MM:SS] text lines with offset/limit pagination.
ytbrain_list_videos	Indexed videos: id, duration, segments, caption source.
ytbrain_stats	Videos, segments, unique terms, channels, talk time, DB size.

The server is deliberately read-only — ingestion is a CLI concern (long-running, needs progress output), querying is what agents do.

CLI reference

Command	Purpose
ytbrain ingest SOURCE [--limit N] [--force] [--languages en de]	Index a channel/playlist/watch URL. Skips already-indexed videos unless --force.
ytbrain search QUERY [--limit N] [--video ID] [--channel NAME]	BM25 hits with snippets and jump links.
ytbrain ask QUESTION [--sentences N]	Extractive answer, max N quoted sentences (default 3).
ytbrain videos [--channel NAME]	List the index.
ytbrain stats	Index statistics.
ytbrain transcript ID [--offset N] [--limit N]	Print a transcript with navigation hints.
ytbrain mcp	Run the MCP stdio server.

Configuration

No .env needed; everything defaults sensibly. Environment variables:

Variable	Default	Notes
YTBRAIN_DB	./data/ytbrain.sqlite3	brain location
YTBRAIN_LANGUAGES	en	comma-separated caption language preference
YTBRAIN_INGEST_SLEEP	0.5	seconds between per-video fetches (politeness)

Performance expectations

• Ingest speed is bounded by YouTube: ~2-4 s per video (metadata + captions) plus the politeness delay. 20 videos ≈ one minute; captions-only means no audio downloads, so it stays cheap and throttle-friendly.
• Search and ask are single-digit milliseconds — the whole brain is one SQLite file with an FTS5 index (3 videos / 567 segments = 0.17 MB; extrapolates to roughly ~35 MB per 1000 talking-hours).
• Re-runs skip indexed videos instantly (= cached lines).

Privacy & security

• Reads public caption tracks via yt-dlp as a library (no shell, no string-built commands); caption files are fetched with httpx and never written to disk.
• Everything stays local: the brain is a SQLite file under data/ (gitignored).
• No API keys, no accounts, no cookies, no telemetry.

Testing

pip install -e ".[dev]"
pytest                              # 46 tests: db/FTS, search, ask, ingest (mocked),
                                    # CLI, MCP tools, json3/vtt parsing
YTBRAIN_DB=data/smoke.sqlite3 python scripts/smoke_mcp.py   # real MCP handshake over stdio

Limitations (honest list)

• Captions only. Videos without captions in your languages are counted and skipped — no local transcription here (that's youtube-transcript-mcp's job).
• ask is extractive, not generative: it quotes, it doesn't compose. If no sentence matches the question terms, it says so instead of guessing.
• Sentence boundaries come from caption punctuation — auto-caption sentences can run long or split mid-thought.
• One best passage region per video per question (deduped); multi-hop synthesis across videos is out of scope.
• Auto-caption vtt fallback dedupes rolling windows best-effort; json3 is preferred whenever YouTube offers it.
• Channel enumeration reflects what YouTube lists (uploads order), not a complete historical archive guarantee.

Project layout

src/ytbrain/
  config.py       env settings, DB path resolution
  db.py           SQLite schema, FTS5 index, search_segments/stats
  youtube.py      yt-dlp wrapper, language/track selection, json3/vtt parsing
  ingest.py       resumable ingest pipeline
  search.py       BM25 search + extractive ask + formatters
  cli.py          argparse CLI (7 subcommands)
  mcp_server.py   FastMCP stdio server (5 tools, read-only)
tests/            46 unit/integration tests (network mocked)
scripts/          smoke_mcp.py (real stdio handshake)

License

MIT