Veritas

An AI-powered data investigation workspace for Product Managers, combining structured investigation workflows with a collectively maintained three-tier data catalog.

What This Is

Two things working together:

1. Investigation MCP Server -- a custom MCP server that provides investigation lifecycle tools (start, continue, capture, close), three-tier catalog management (local/community/curated), and a self-improving memory system
2. Data Investigation Framework -- a structured workflow for running multi-session analyses with persistent context, findings tracking, and institutional knowledge that compounds over time
3. Agent Memory System -- behavioral learning that enables the AI agent to remember corrections, preferences, and workflow patterns across sessions, with a promotion pipeline for turning recurring patterns into permanent rule improvements

Data access (SQL queries, schema discovery, profiling) is handled by the official Databricks MCP server, which runs alongside the investigation MCP server. This separation means adding new data sources (Dynatrace, Amplitude, etc.) is just adding another MCP server entry.

Quick Start

Setup takes ~5 minutes. See GETTING_STARTED.md for the full guide with screenshots, or follow the quick version:

1. Clone and run setup

git clone https://github.com/hertzcorp/htzd-databricks-mcp.git
cd htzd-databricks-mcp

Mac / Linux: ./setup.sh | Windows (PowerShell): .\setup.ps1

The script checks prerequisites, installs dependencies, then asks you a few questions: your name, email, Databricks HTTP path, and a personal access token. It shows you the exact links to copy each value. One run — no editing files.

2. Restart Cursor and approve MCP servers

Restart Cursor (Mac: Cmd+Q | Windows: Alt+F4, then reopen). When it starts, click "Start" on the two MCP server prompts at the bottom of the screen. This is a one-time approval -- all tools are pre-approved after that.

3. Start asking questions

Open the Chat panel (Mac: Cmd+Shift+I | Windows: Ctrl+Shift+I) and try:

• "What tables do we know about?" -- browse the team's data catalog
• "Start investigation" -- begin a tracked multi-session analysis
• "What data do we have on billing?" -- search the catalog for relevant knowledge
• "Deep investigate: why is contact rate rising?" -- autonomous recursive investigation
• "Deep index" -- crawl Databricks lineage to build the table knowledge base

See docs/ONE-PAGER.md for a quick reference of all features with end-to-end examples.

Project Structure

.
├── src/investigation_mcp/          # Investigation MCP server
│   ├── server.py                   # MCP server entry point (24 tools)
│   └── tools/                      # Tool implementations
│       ├── investigation.py        # Start, continue, capture, close investigations
│       ├── catalog_index.py        # Catalog reading: index, entries, health, context
│       ├── catalog_local.py        # Local entry capture + preferences
│       ├── catalog_sharing.py      # Share local entries to community via PR
│       ├── catalog_curation.py     # Curator tools: review, approve, reject
│       ├── memory.py               # Agent memory: capture, load, reinforce, review, apply
│       ├── _git_helpers.py         # Shared git utilities
│       └── catalog_management.py   # Legacy catalog tools (deprecated)
├── data-catalog/                       # Three-tier data catalog
│   ├── curated/                        # Reviewed, authoritative knowledge (team-shared)
│   ├── community/                      # Unvetted contributions from team members (team-shared)
│   ├── local/                          # Personal notes, auto-captured (gitignored)
│   ├── _deep_index/                    # Pre-computed table knowledge base (team-shared)
│   ├── _schema.yaml                    # Entry type definitions
│   └── _curators.yaml                  # Approved catalog curators
├── investigations/                     # Tracked analyses (gitignored except templates + index)
│   └── _templates/                     # Templates for new investigations
├── docs/
│   ├── hertzcorp-architecture/         # Hertz system architecture references
│   ├── adoption/                       # Adoption strategy and growth plans
│   ├── workshop/                       # Workshop facilitator guide and slide assets
│   ├── ONE-PAGER.md                    # Quick feature reference
│   ├── SPEC.md                         # Technical specification
│   └── PATTERNS.md                     # Architectural patterns
├── .cursor/
│   ├── memory/                         # Agent behavioral memory (gitignored)
│   │   ├── corrections/                # Behavioral corrections (YAML)
│   │   ├── preferences/                # Style and output preferences (YAML)
│   │   ├── patterns/                   # Detected workflow patterns (YAML)
│   │   └── reflections/                # Session retros (Markdown)
│   ├── rules/                          # Cursor rules
│   │   ├── analysis-principles.mdc     # 8 data analysis principles
│   │   ├── memory.mdc                  # Agent memory behavior (when to capture/load/cite)
│   │   ├── pm-guardrails.mdc           # Prevents AI from modifying protected files
│   │   ├── first-run.mdc              # Welcome experience for new users
│   │   └── ...                         # Development workflow rules
│   ├── hooks/
│   │   └── memory-flush.py             # Advisory: remind to capture learnings at session end
│   └── skills/
│       └── data-investigation/         # Investigation workflow skill
├── ideation/                       # Product concepts and drafts (gitignored)
├── scripts/                        # Standalone analysis scripts
├── .github/
│   └── workflows/
│       └── catalog-validation.yml  # CI: validates catalog entries on PR
├── setup.sh                        # Onboarding wizard
├── GETTING_STARTED.md              # PM-friendly setup guide
└── pyproject.toml                  # Package: investigation-toolkit

Three-Tier Data Catalog

The catalog is the institutional knowledge layer. It grows with every investigation.

Tier	Location	Shared?	Trust Level	How It Gets There
Local	data-catalog/local/	No (gitignored)	Personal notes	AI auto-captures during investigations
Community	data-catalog/community/	Yes (via PR)	Unvetted	PM shares via share_to_catalog tool
Curated	data-catalog/curated/	Yes (via PR)	Authoritative	Curator promotes from community

Knowledge lifecycle:

Investigation discovers knowledge
  -> AI auto-captures to local/ (continuous, silent)
  -> PM selects what to share ("share this with the team")
  -> share_to_catalog creates PR to community/ (CI validates, approver merges)
  -> Curator reviews community entries (review_pending)
  -> Curator promotes to curated/ (approve_entry) or rejects (reject_entry)

The AI reads all three tiers with trust ordering: curated (high confidence) > community (hints, verified before use) > local (personal recall).

Investigation Framework

How It Works

Say "start investigation" in Cursor chat to begin a tracked analysis. The framework:

1. Discovery -- loads the catalog, refines the question, finds the right tables
2. Analysis -- runs queries via Databricks MCP, auto-captures discoveries to local catalog
3. Validation -- samples raw records, confirms field semantics, checks system behavior
4. Capture -- saves findings, presents unshared catalog entries for sharing
5. Close -- marks complete, prompts for catalog sharing

Investigations persist across sessions. Say "continue investigation" to resume where you left off.

Agent Memory System

The agent learns from your corrections and preferences over time through two paths:

Soft path (immediate): Corrections and preferences are saved to .cursor/memory/ and loaded into context at the start of future sessions. The agent adjusts its behavior based on past learnings without any structural changes.

Hard path (permanent): When a correction recurs 3+ times across sessions, the agent detects the pattern and proposes a permanent change to the workflow rules or investigation skill. You approve before anything changes. Platform-scope proposals create a PR for team review.

What	How
Correct the agent	It calls capture_memory automatically -- remembered next session
Same correction 3x	Pattern detected, proposed as a rule change at session end
Approve a proposal	apply_proposal edits the rule file (personal) or creates a PR (platform)
Wrong promotion	Agent detects contradiction and proposes rollback
Check system health	memory_stats shows counts, averages, maintenance status

Memory is personal and gitignored. Each PM's agent learns independently. Improvements that benefit everyone flow through the PR-based promotion pipeline.

Analysis Principles

The .cursor/rules/analysis-principles.mdc rule enforces eight principles:

1. Ground truth first -- sample raw records before aggregating
2. Profile before joining -- verify field contents, don't assume from names
3. Ask about system behavior early -- domain knowledge shapes data interpretation
4. Correlation is not causation -- prevalence on a record does not equal cause
5. Explicit assumptions -- every volume estimate needs a stated deflection rate
6. Hypotheses before conclusions -- present as "data suggests" until validated
7. Flag observability gaps -- generic error codes need decomposition, not sizing
8. Signal catalog trust level -- indicate whether knowledge is curated, community, or local

Commands Reference

Investigation Commands

Command	What It Does
start investigation	Create a new tracked investigation
continue [name]	Resume a previous investigation with full context
capture findings	Save findings and offer to share catalog entries
close investigation	Mark complete, prompt for final catalog sharing
list investigations	Show active and recent investigations

Catalog Commands

Command	What It Does
What tables do we know about?	Load catalog index -- shows all entries across tiers
What do we know about [topic]?	Load entries for a specific domain or subject
share this with the team	Share a local catalog entry to community via PR
catalog health	Show catalog metrics: backlog, coverage, stale entries
retry pending shares	Retry shares that failed due to connectivity

Curator Commands

Command	What It Does
review catalog	Show pending community entries grouped by domain
approve [entry]	Promote a community entry to curated (curator-only)
reject [entry]	Remove a community entry with optional reason

MCP Tools Reference

Investigation MCP (Custom)

Tool	Description
start_investigation	Create investigation folder with templates
continue_investigation	Load investigation context and resume
capture_findings	Save findings, prompt for catalog sharing
close_investigation	Mark complete, final catalog check
list_investigations	List active/completed investigations
load_catalog_index	Lightweight index of all catalog entries
get_catalog_entries	Full content for entries matching filter
get_data_context	Read curated domain knowledge
catalog_health	Catalog metrics and health summary
add_local_entry	Capture a discovery to local catalog
share_to_catalog	Share local entry to community via PR
retry_pending_shares	Retry failed shares from offline sessions
review_pending	Summarize unreviewed community entries
approve_entry	Promote community entry to curated (curator-only)
reject_entry	Remove community entry (curator-only)
get_preferences	Read learned preferences
save_preference	Save a preference for future sessions
capture_memory	Capture a correction, preference, or pattern to memory
load_relevant_memories	Load scored, filtered memories for current context
reinforce_memory	Reinforce an existing memory when same correction recurs
learning_review	Generate end-of-session learning summary with proposals
capture_reflection	Write a structured session retro to memory
apply_proposal	Apply approved proposal to rule/skill file (or create PR)
memory_stats	Report memory system health and run maintenance

Official Databricks MCP (Data Access)

Data access tools are provided by the official Databricks MCP server:

Tool	Description
list_catalogs	List Unity Catalog catalogs
list_schemas	List schemas in a catalog
list_tables	List tables in a schema
describe_table	Get column details
execute_sql	Run SQL queries
sample_data	Get sample rows
profile_table	Generate data quality profile

Tips for Better Results

At the start of an investigation:

• The AI loads the catalog automatically. For new domains without curated entries, tell the AI what tables to use.
• Share system context early -- how key systems behave, known data quirks, business rules.
• State a hypothesis: "I think X is caused by Y" is more productive than "Tell me about X."

During analysis:

• Challenge aggregate findings. Ask "Have you looked at actual records?"
• When the AI presents a volume estimate, ask about the assumption: "What deflection rate?"
• The AI auto-captures gotchas, joins, and terms to your local catalog as it discovers them.

At the end of a session:

• Say "capture findings" to save progress. The AI presents unshared catalog entries for sharing.
• Choose which discoveries to share with the team. The AI handles the PR.

For executive deliverables:

• Ask the AI to create a separate brief rather than editing findings.md.
• Push back on oversimplified narratives. Ask for sub-problems.
• Request an assumptions and limitations section.

Extending with Additional MCP Servers

Adding a new data source is just adding an entry to .cursor/mcp.json.template:

{
  "mcpServers": {
    "dynatrace": {
      "command": "npx",
      "args": ["-y", "@dynatrace-oss/dynatrace-mcp-server@latest"],
      "env": {
        "DT_URL": "{{DT_URL}}",
        "DT_TOKEN": "{{DT_TOKEN}}"
      }
    }
  }
}

The investigation MCP and catalog are source-agnostic. The catalog's system field tracks which data source knowledge came from.

Running Scripts

Legacy standalone analysis scripts can be run from the repo root. Note: some older scripts may reference the previous databricks_mcp package which has been renamed to investigation_mcp. See the scripts directory for details.

Troubleshooting

"MCP server not showing in Cursor"

• Run ./setup.sh --test-only to validate configuration
• Restart Cursor completely (Cmd+Shift+P > "Reload Window")
• Check .cursor/mcp.json exists and has correct paths

"Setup fails at connection test"

• Verify your Databricks workspace URL includes https://
• Check that your personal access token hasn't expired
• Ensure your SQL Warehouse is running
• Try ./setup.sh --no-test to skip validation, fix credentials in .env, then ./setup.sh --test-only

"Table not found"

• Check the full table path (catalog.schema.table)
• Use load_catalog_index to see what the team knows about
• Verify you have access to the table in Databricks

Security Notes

• Databricks tokens stored in .env (gitignored, never committed)
• Local catalog entries are gitignored (personal to each user)
• Community catalog entries go through PR review before merging
• CI scans catalog PRs for secrets and PII patterns
• PM guardrails rule prevents AI from modifying protected files (rules, skills, curated catalog) -- exception: apply_proposal during approved learning reviews
• Agent memory files are gitignored (personal behavioral learnings never pushed to remote)
• Memory promotions to rule files require explicit human approval; platform-scope changes go through PR review
• Curator authorization enforced programmatically via _curators.yaml