Iris — verifiable BI over federated data

A business user asks a question, gets a number, and decides on it. Usually nobody in that loop — not the user, not the analyst, not the system itself — can tell whether the number came from the data or from the model's imagination.

Iris is an attempt at the other thing: a number you can open.

Q: Tip rate for card payments by borough on rainy versus dry days

  Manhattan  dry   23.54%      ← click it
  Manhattan  wet   23.62%

  displayed 23.54  ·  recomputed from these rows 23.54  ·  ✓ agrees
  3,979,472 contributing rows (card-paid, positive fare)
    fare=13.27  tip=2.00  pickup=2024-01-27 00:38  file=yellow_tripdata_2024-01.parquet

Three sources, three engines, two joins, one question — and a receipt.

Regenerate with .venv/bin/python scripts/make_architecture_gif.py; the row counts are read from data/load_manifest.json, so the diagram cannot drift from the data.

---

Three claims, and the evidence for each

1. The model cannot invent a metric. It never writes SQL. It emits a SemanticQuery naming measures and dimensions from a declared manifest; anything else is rejected before execution, with the valid names in the error. A deterministic compiler turns the validated struct into SQL. → semantic/, tests/test_planner_fake_llm.py

2. Every number opens into its source rows. Click a figure and Iris returns the rows behind it — each naming the vendor file it came from — plus the measure recomputed over exactly those rows. If the two disagree, it says so. → verify/drilldown.py

3. The evaluation can measure things the agent's own machinery can't express. Ground truth is hand-written SQL against the raw views, so a case is not limited to the shapes the system already handles. → eval/fixtures/

Where those claims stop is in Limitations, and the scoring method is in docs/eval_methodology.md.

---

Numbers

Current run — eval/results/latest.json, 15 cases × 3 runs:

Category	Result	What it means
in_manifest	5/5 clean, 100%	questions phrased the way the manifest phrases things
held_out	4/4 clean, 100%	questions whose key terms are absent from the manifest
ambiguous	2/2 clean, 100%	correct behaviour is to ask, not answer
adversarial	4/4 clean, 100%	unanswerable, injected, or empty
grounding	30/30	every figure in every summary appeared in its table

p50 3.4s, $0.35 for all 45 runs. 124 tests, of which 79 need no database, no services and no API key — including the ones covering the no-fabrication guarantee.

Never quote a single-run number from this harness: planning is an LLM call and is not deterministic. Use --repeat; a case that passes sometimes is reported as FLAKY with its rate rather than resolved by whichever way the dice fell.

---

Data — three independent origins

Source	Origin	Lands in	Rows
Yellow-taxi trips, 2024 Q1	NYC TLC	Parquet ("the lake")	9,554,778
Taxi zone lookup	NYC TLC	PostgreSQL	265
GHCN-Daily weather, 4 stations	NOAA	MongoDB	5,204

Nobody designed these to fit together, so their seams are real. Six were measured and are catalogued in docs/join_hazards.md. The important one:

"What is the average tip rate?" has three defensible answers, 9.4 points apart — 17.60%, 22.39%, 27.00% — because cash fares never record a tip (1,330,105 trips, 13.92%, 100% zero-tip). Iris does not pick one. The manifest declares the ambiguity and the agent asks.

Runner-up: NOAA publishes tenths, so a naive loader reports TMAX=83 as 83 °C for a New York January. Plausibly typed, never errors, poisons every aggregate downstream.

---

Quick start

python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
( cd ui && npm install )

cp .env.example .env          # set ANTHROPIC_API_KEY
./run.sh load                 # download + load all three sources (~215 MB, one-off)
./run.sh                      # UI :9300 · API :9301 · MCP :9302

Requires Python 3.12+, Node 18+, a local PostgreSQL, and Docker (for Iris's own MongoDB on 27018 — it will not write into a container it does not own). run.sh refuses to start on a bound port and prints who holds it. See docs/local_setup.md.

.venv/bin/python -m pytest tests/ -q          # 124 tests
.venv/bin/python -m eval.harness --repeat 3   # the eval (needs data + a key)

---

How it works

   question
      │
      ▼
  ┌─────────┐  sees only declared measures and dimensions — never a column
  │  plan   │  emits SemanticQuery  ────────► clarify   (declared ambiguity)
  └────┬────┘                        └──────► refuse    (nothing can answer it)
       │ validated struct
       ▼
  ┌──────────┐  deterministic. Rejections go back to the planner verbatim, once.
  │ compile  │  join-path resolution · fan-out refusal · per-measure FILTER
  └────┬─────┘
       ▼
  ┌──────────┐   DuckDB: Parquet natively · Postgres ATTACH · Mongo → Arrow
  │ execute  │
  └────┬─────┘
       ▼
  ┌──────────┐   join match rates · sentinel impact · drill-down
  │  verify  │
  └────┬─────┘
       ▼
  ┌──────────┐   every numeral must appear in the table, or it is rewritten
  │ narrate  │
  └──────────┘

The manifest is the contract. semantic/manifest.yaml declares sources, entities, joins with cardinality, dimensions with sentinel values, measures with their populations, and ambiguity groups. Unit conversions and metric definitions live there and nowhere else.

Fan-out is refused, not hoped away. Aggregating across a one-to-many join multiplies every additive measure. The compiler checks join direction and declines. A test builds the unsafe weather join by hand and shows the exact 4× inflation it would have produced.

Portability is tested, not asserted. tests/test_portability.py takes an unrelated dataset (library lending), supplies only data and a manifest, and gets correct hand-computable answers. Zero dataset references survive in executable code across federation/ and semantic/; seven remain in docstrings, where they are explaining a hazard rather than driving behaviour. A test tokenises both packages and holds that number, so a stray if borough == cannot appear unnoticed.

Two honest edges on that claim. The foreign dataset is Parquet-only, so it demonstrates dataset portability — a new schema, new manifest, unchanged code — not engine portability; a second Postgres or Mongo deployment is not spun up in tests. And "adding an engine is one class with three methods plus a registry line" is a structural fact about federation/connectors/factory.py, not something a test proves end-to-end; what is tested is that an unknown engine fails with a message naming the three methods.

MCP is an external surface. describe_semantic_model, query_semantic (verified), drill_down, and run_sql — the last returning verified: false with a warning naming what it bypassed. The agent itself talks to the engine in-process.

---

Limitations, owned not buried

The eval validates the compiler, not the definitions. Reference SQL checks that Iris computes what the manifest declares. If avg_tip_rate_card is declared wrongly, the fixture written to match it is wrong the same way. That is exactly why the tip-rate case is a clarify case rather than an answer case — the choice of definition is put to the user, not graded against one.

run_sql has none of the guarantees. It exists for the long tail the manifest doesn't cover. Results carry verified: false and the UI badges them. Anything through that path is ordinary text-to-SQL.

Two of the three sources are live; MongoDB is a snapshot. Parquet is read from disk per query and Postgres is ATTACHed, so both reflect the current state. The Mongo collection is materialised into Arrow once at startup — a document written afterwards is invisible until refresh() or a restart. That is a deliberate choice for a cached third-party feed of 5,204 station-days, not a limit of the bridge, but "federated across three engines" should not be read as "three live connections." tests/test_federation.py proves the boundary by writing to MongoDB and asserting the view does not move.

Pushdown is narrower than federation demos usually imply. Every connector reports only filter and project, and only where EXPLAIN confirms it below the scan node. Joins and aggregates always run in DuckDB. An earlier version of these connectors claimed join and aggregate pushdown; the query plan did not support it, and the test now pins each claim to a real plan.

15 cases is small. All four categories at 100% over 3 runs means no failures observed in 45 runs, not proven correct. Earlier single runs did show flaky failures — that is why --repeat exists and why FLAKY is a distinct verdict.

The planner sees the whole manifest. Fine at 10 measures and 11 dimensions; at hundreds this needs retrieval rather than dumping everything into the prompt. That is the first thing that breaks at scale.

Held-out is a vocabulary test, not a reasoning test. It proves the planner maps unfamiliar wording onto declared names. It does not prove it would handle a genuinely novel analytical shape.

Single dataset. Portability is demonstrated on a synthetic second dataset in tests, not on a second production-scale one.

---

Layout

semantic/     manifest.yaml (the contract) · model · query + validation · compiler
federation/   DuckDB engine + per-ENGINE connectors, built from the manifest
verify/       provenance · drilldown · grounding
agent/        planner (LLM → SemanticQuery) · graph (3 terminal states) · llm
eval/         cases · scorer · harness · fixtures/*.sql (hand-written ground truth)
data_pipeline/  the three loaders — load faithfully, filter in the semantic layer
api/ ui/      FastAPI + Next.js dashboard and Trace Inspector
toolserver/   MCP server (external clients)
docs/         join_hazards · design_decisions (ADRs) · eval_methodology · local_setup

Built with Claude (Sonnet 5 planning, Haiku 4.5 narration) · DuckDB · LangGraph · MCP · FastAPI · Next.js.