Fadlie

The same data lives in four systems. Your catalog knows every path between them. Only one copy is governed.

Fadlie is an MCP agent for DataHub. It finds datasets that hold the same data across different platforms, then finds the governance — owners, domain, descriptions, PII tags, glossary terms — that stopped at one of them. It can level the difference, and it never writes a value it did not read from a dataset it names.

Built for Build with DataHub: The Agent Hackathon, on the contest's own showcase-ecommerce dataset. Every number below is produced by a script in this repository, against a live DataHub instance.

Two minutes and forty seconds, including a live run against the deployed agent: https://youtu.be/YXff0HNRAwU

---

What the measurements found

Run python scripts/mesurer-jumeaux.py to reproduce all of this.

The lineage graph tells you nothing about whether two datasets hold the same data. The graph is a single component: 103 nodes, 161 edges, no isolated dataset. All 88 same-name pairs are connected — at distance 2 or 4. The median distance between two datasets picked at random is also 4. Twins are indistinguishable from strangers.

Names tell you almost as little. Three of fifteen same-name groups are not the same thing: four Tableau datasets called Custom SQL Query share 0 % of their columns; promotions shares 9 %.

And the governance does not travel. Eleven tables exist identically on dbt, Snowflake, Postgres and S3 — 100 % column overlap, customers has the same 22 columns in all four. On the datapack as it ships, dbt/customers has three owners, a domain and a description; its three twins have none, none and none. Twelve identical columns are annotated on one side and bare on the other, and customers.customer_id carries the PII_Data tag on Postgres alone.

Ask that catalog where the personal data is, and it answers with two datasets, with the confidence of something that has looked.

The live demo no longer matches that paragraph, and that is the point. Fadlie has since levelled three of the eighteen groups on the running instance — customers, orders, products — writing 102, 60 and 53 values that it read from their twins. Around 420 gaps remain, on some 46 datasets, so there is plenty left to try. Reload the datapack into a fresh DataHub and the numbers above come back exactly.

How it decides

Structure suggests. A model decides. Nothing is invented.

Layer	What it does	Why it cannot be the last word
catalogue	reads schemas, governance, lineage	—
candidats	2 211 pairs → 97	names and column overlap are wrong 1 time in 5
juge	Amazon Nova Micro, temperature 0	it decides; it never guesses on its own
ecart	what one twin has and another lacks	—

python scripts/mesurer-le-juge.py puts the judge against 16 pairs drawn from the real catalog, chosen to be hard in both directions: replicas that differ in case and column count, reference tables of identical shape, aggregates computed from a table rather than copied from it, and the four homonymous Tableau queries that share only measure names. 16 out of 16. Ten runs of the same pair give the same verdict ten times.

Two rules that shape everything

Fadlie copies; it never writes. No value put into the catalog is produced by a model. Each one comes from a dataset that already carried it, and every proposal names its source — Ecart refuses to exist without one. A description generated by a machine is indistinguishable from a description written by the team that knows the data; six months later nobody can tell which is which.

A disagreement is not a gap. snowflake/ORDER_DETAILS sits in Ecommerce Operations while its four twins sit in Data Platform Team. Fadlie reports it and does not choose. Someone decided, or someone erred, and neither is an agent's call.

A failure must not look like good news

The judge raises rather than returns "different". If it could silently fail, Fadlie would report no duplicates found — a catalog in good order. Nobody audits good news. So the server probes the model before its first verdict, and both real failure modes are verified: a bare model id (Frankfurt requires the regional inference profile) and credentials without Bedrock access.

The tools

Tool	What it returns
catalog_summary	pairs examined, groups found, gaps, disagreements
find_duplicate_datasets	the groups, with the judge's verdict for each pair
governance_gaps	every gap, each naming the dataset it would be copied from
apply_governance	writes them. Dry run by default

No tool takes a user, a token, or an identity — authorisation comes from the Authorization header, and a check enforces it because nothing in the language does.

Running it

cp .env.example .env && $EDITOR .env
set -a && . ./.env && set +a

python -m pytest tests/          # 77 tests, no network, no bill
python -m fadlie check           # reads config, connects to nothing
python -m fadlie report          # the full analysis, printed
python -m fadlie serve           # the MCP server, locally

python scripts/mesurer-jumeaux.py    # every number in this README
python scripts/mesurer-le-juge.py    # the judge against 16 hard pairs
python scripts/verifier-mcp.py       # 24 checks through a real MCP client

What it runs on

• DataHub Core v1.7.0 on EC2, loaded with the contest's showcase-ecommerce datapack — 67 datasets across Snowflake, dbt, Postgres, S3, Tableau, PowerBI and Looker.
• Amazon Bedrock, Amazon Nova Micro as the judge, in eu-central-1.
• AWS App Runner for the MCP server, over HTTPS.

Licence

Apache License 2.0 — see LICENSE.