chronicle

Personal event store and retrieval brain. Ingests Telegram, wakapi, dawarich (and whatever comes next) into one timeline, and serves it to agent-runner over MCP so the assistant can actually look things up.

Deploys into homelab-gitops as the chronicle stack.

---

Why this exists

The Telegram archive is 681,331 messages across 487 chats and 457 senders, 2018-12-30 to now. The existing telegram-sync stack embeds every message into Qdrant. Measured against the real corpus:

messages under 20 chars	65.0% (442,954)
messages under 60 chars	94.0%
messages over 200 chars	1.5%
messages with reply_to_id	7.9%

So ~442,000 of those vectors represent ок, ага, +1, да, 😂. They are not retrieval units — they crowd every neighbourhood they land in and bury the 1.5% of messages that carry a proposition.

Chronicle's core move is aggregation before indexing. Events are grouped into episodes by a per-thread fitted time gap: 681k units become ~50k. The index gets ~11× smaller and retrieval gets better at the same time.

Evidence, from SeCom (ICLR 2025), measuring retrieval quality by memory unit on conversational data — at ~30 tokens/turn, where ours are 5–10:

segment-level   71.57   <- what this builds
turn-level      65.58
session-level   63.16
summaries       53.87-56.25   <- worst. Do not build a summary pyramid.

Full reasoning, benchmarks and citations: docs/RESEARCH.md.

It is not a Telegram tool

Telegram is the richest channel, not the only one. 18 adapters, five storage shapes, one Event shape downstream.

tier	sources	why
1 · core	telegram wakapi dawarich calendar	the archive is worth having with only these
2 · behaviour	firefly lastfm forgejo jira	what you did, as opposed to what you said
3 · artifact	immich paperless gmail notion karakeep github linkedin slack	things you made, saved, or were sent
4 · ambient	miniflux owntracks	weak signal; on last, off first if precision drops

Storage shapes, because the homelab is not uniform and assuming it was is how the first version broke: PostgreSQL (telegram, immich, paperless, miniflux, forgejo, dawarich) · MariaDB (firefly) · SQLite (wakapi, karakeep) · flat JSONL files (owntracks) · HTTP/MCP (gmail, calendar, notion, slack, jira, linkedin, lastfm, github).

The policy layer is the point

"Use all possible channels" has a failure mode that looks exactly like the one Chronicle was built to fix. Indexing 681k sub-20-char messages was the wrong unit. Turning on twenty sources without a policy is the wrong source mix — miniflux alone can contribute 100k article rows you never opened, immich has 400 near-identical burst frames per moment, and the archive becomes millions of events that are ~90% chaff. Precision collapses the same way.

So every source declares a density, and density decides how hard the adapter aggregates before anything reaches the episode layer:

• NARRATIVE — deliberate human text. Segmented into episodes.
• DISCRETE — one row really is one thing that happened. Passed through.
• TELEMETRY — meaningful only in aggregate. The adapter rolls it up: wakapi heartbeats → coding sessions, dawarich points → stays, lastfm scrobbles → listening sessions, immich photos → photo sessions.
• AMBIENT — stored, but kept off the default retrieval surface.

chronicle/sources.py also lists the ~40 homelab stacks that are explicitly not sources, so the boundary is documented rather than rediscovered. All 57 stacks is not the goal; monitoring, qdrant and vaultwarden describe the machine, not the life.

make test enforces this: every adapter must have a policy, densities must match, and dawarich/owntracks are flagged as mutually exclusive (same GPS signal — enabling both double-counts every trip and the duplicate reads as corroboration).

Why more channels actually helps

"What was happening before I got depressed" — your stated goal. Telegram tells you what you said. Location tells you whether you stopped leaving the house, wakapi whether you stopped coding, firefly whether spending changed, lastfm what you played at 3am, immich whether you stopped taking photos. The behavioural signals are more honest than the conversational ones, because you don't curate them.

And cross-source corroboration turns a guess into evidence: a trip mentioned in Telegram, confirmed by dawarich coordinates, photographed in immich, and paid for in firefly is a fact you can trust.

Relationship to Hindsight

They are opposites, which is why they compose.

	Hindsight	Chronicle
origin	you decided it mattered	you never chose to save any of it
volume	~5,200 facts	681k events / ~50k episodes
precision	high, curated	low, exhaustive
evidence trail	none	nothing but evidence
shape	a notebook you write in	a recording that ran the whole time

Chronicle does not replace Hindsight and must not flood it. The personal bank is already at 2,726 facts and times out on sync_retain; piping ~50k episodes of extracted facts into it would 40× the bank and make recall useless.

Two narrow flows instead:

• ground (Chronicle → answer), every recall. Hindsight facts are unsourced assertions. Chronicle attaches the conversations behind them — including ones that contradict. This replaces time-based staleness rules (ticket >14d, finance >30d) with a measurement.
• Promotion (Chronicle → Hindsight), rare. v_promotable_facts requires support across ≥3 episodes and ≥2 threads at confidence ≥0.7. Target hundreds per year. Watch get_bank_stats after each run.

Architecture

sources ──► adapters ──► event (immutable, partitioned by year)
                            │
                            ▼
                     SEGMENTATION            per-thread fitted time gap
                     681k ──► ~50k           + caps + reply-edge anchors
                            │
                            ▼
                     episode                 raw_text (returned)
                            │                embed_text (indexed)
                            ▼
                     PostgreSQL              halfvec(1024) exact scan
                     one system              tsvector + trgm + B-tree
                            │
                            ▼
                     MCP ──► agent-runner ──► tg-assistant

Everything is in one PostgreSQL. At ~50k episodes for one user, ANN solves a problem that doesn't exist: exact cosine over ~123 MB is single-digit ms, and it keeps every date filter exact — sidestepping the HNSW percolation failure that bites hardest at the ~1%-cardinality date ranges you query most.

Quick start

git clone <this repo> chronicle && cd chronicle
make test                     # 57 unit tests, no DB or models needed
cp .env.example .env          # fill in, then `make encrypt STACK=chronicle` in homelab
make smoke                    # migrations + every SQL function, throwaway DB

make doctor                   # ← ALWAYS. validates sources before you ingest
make ingest                   # sources -> event -> episode -> embedding
make eval-init && make eval   # chronicle vs ripgrep, on your questions

make doctor is not optional

Two adapter assumptions were already wrong on first contact — wakapi is SQLite not Postgres, firefly is MariaDB with amounts on transactions — and each would have surfaced hours into a backfill, after the worker had written wrong rows. Doctor finds that class of problem in ~10 seconds, read-only.

It checks: driver reachability, timestamps that are actually datetimes (SQLite returns TEXT), ascending order, duplicate ids, unaggregated telemetry, empty narrative text, degenerate thread_key, and import-time-masquerading-as- event-time (the immich createdAt vs EXIF dateTimeOriginal trap). Run it again after upgrading any source stack — an upstream migration is exactly what breaks an adapter quietly.

Deploy: see docs/DEPLOY.md. Short version — clone into /opt/stacks/chronicle, resolve the chronicle-db digest into PINS.md, register 10.211.71.0/24 in NETWORKS.md, make validate in the homelab repo, then docker compose up -d.

Requirements

• PostgreSQL 16 with pgvector ≥ 0.7 — halfvec does not exist in 0.6. The pinned pgvector/pgvector:pg16 image is fine; a distro postgresql-16-pgvector package may not be.
• ~2.5 GB resident (api + db). The worker is restart: "no" and needs 8 GB while it runs — the box is 32 GB with 61.4 GB of mem_limit committed and has hit 96% swap, so it is scheduled, not resident.

Status

Working: segmentation, gap fitting, cross-script entity resolution, intent routing, RRF fusion, bi-temporal facts with deterministic conflict resolution, the full schema, the source-policy layer, 18 adapters across 5 storage shapes, MCP tool surface.

The SQL-backed adapters (telegram, wakapi, dawarich, immich, paperless, firefly, karakeep, miniflux, forgejo) carry real queries against real schemas but have only been run against fixtures — verify each one against your data before trusting its output. The API adapters take an injected fetch_page callable and need wiring to the corresponding MCP tool.

Stubbed: only worker.py enrich — the local-LLM pass for summaries, topics and facts. Everything else runs. Enrichment is deliberately last: retrieval works without it, so ship and measure before spending weeks of CPU there.

Tested: 57 unit tests, plus migrations and every SQL function exercised against real PostgreSQL 16 + pgvector 0.8.0 in CI.