Claygent Verification Agent

Takes a Clay export CSV and re-checks every Claygent-generated claim against its row's own source URL before that data reaches a live outreach sequence. Ships two interfaces (CLI, MCP server) over one shared verification engine.

Why this exists

Clay's built-in AI research agent, Claygent, hallucinates: it fabricates dates, misattributes facts, and sometimes ignores its own "return JSON only" instruction and returns a narrative sentence instead. I hit this directly building a news/fundraising signal monitor for a client's Clay workflow - Claygent occasionally broke every downstream field extraction on a row by not returning valid JSON, and there was no way to tell a real hallucination from a correct extraction without manually re-checking the source article. Clay has no systematic fix for this; their own support response is "try a different integration."

This tool is that manual re-check, automated: take Claygent's claim, fetch the same source URL a human would click to verify it, and ask an independent model whether the page actually supports the claim. It's built to fail loudly, not quietly - a row with nothing usefully extracted, a source that can't be fetched, or a page that doesn't address the claim all produce their own distinct, reportable outcome (NO_CLAIM_DATA, FETCH_FAILED, UNVERIFIABLE) instead of being silently skipped or forced into a MATCH.

Schema-agnostic by design (and why there's no single "claim column")

There's no fixed Clay export format to conform to, and there's no single "claim column" to point this tool at - a real Clay CSV export never carries the raw Claygent response with data in it; that column always exports blank. The only columns with real values are the ones manually selected from Claygent's side panel (click the response cell, pick which JSON keys become their own table columns). So this tool takes a list of pre-extracted claim columns, whatever names and however many your export happens to have, and treats a row where every one of those columns is blank as the real-world signature of a failed extraction (NO_CLAIM_DATA) - not a JSON parse error, since there's never any raw JSON in the export to parse in the first place. app/csv_mapper.py never hardcodes a specific client's schema.

Architecture

app/
  schema.py            : shared dataclasses passed between every stage
  csv_mapper.py         : deterministic - CSV parsing + configurable multi-column claim mapping
  claim_parser.py        : deterministic - drops blank claim columns, flags NO_CLAIM_DATA if all blank
  fetcher.py              : deterministic - fetch source URL -> page text, with failure taxonomy
  verifiers.py             : ClaimVerifier interface + ClaudeClaimVerifier (the one LLM call)
  model_capabilities.py     : per-model thinking/effort quirks (copied from the sibling
                              ai-personalization-engine project, kept in sync manually)
  engine.py                  : batch orchestration + cost estimation, depends only on
                                the ClaimVerifier interface, never a vendor SDK directly
  cli.py                      : thin CLI, single CSV in, verified CSV out
  mcp_server.py                 : thin MCP tool, same engine.run_batch() call as the CLI
tests/
  test_csv_mapper.py    : multi-column mapping, arbitrary/missing headers
  test_claim_parser.py   : blank-column filtering incl. the all-blank NO_CLAIM_DATA case
  test_fetcher.py          : fetch failure taxonomy (dead link, timeout, paywall, JS shell)
  test_engine.py             : full orchestration against a FakeClaimVerifier double
  fixtures/                    : synthetic CSVs modeled on real Clay export column shapes,
                                  including a blank raw-response column for realism

engine.py never imports anthropic or requests directly - only the ClaimVerifier interface and the deterministic modules above it. The CLI and MCP server both call engine.run_batch() and nothing else, so they can't drift out of sync with each other; adding a third interface (e.g. a future Sheets add-on) means writing a thin wrapper, not new verification logic.

Cost-aware by design

--estimate-only gives a rough pre-flight cost estimate, based on a character-count heuristic, before any API calls are made:

python -m app.cli export.csv --claim-columns funding_series,is_confirmed --source-column Link --estimate-only
# 3 rows, estimated cost for model claude-sonnet-5: ~$0.0071

Every real run also reports actual token usage and dollar cost per batch, same credit-conscious pattern as the other tools in this portfolio.

Running it

python -m venv .venv
.venv/Scripts/activate        # .venv/bin/activate on macOS/Linux
pip install -r requirements.txt

export ANTHROPIC_API_KEY=sk-ant-...   # required for real runs, not for tests

# CLI
python -m app.cli export.csv --claim-columns funding_series,is_confirmed --source-column Link --out verified.csv

# MCP server (wire into a client's .mcp.json, or run standalone)
python -m app.mcp_server

Example .mcp.json entry for Claude Code / Cursor:

{"mcpServers": {"claygent-verifier": {"command": "python", "args": ["-m", "app.mcp_server"]}}}

Input CSV needs, at minimum, one or more pre-extracted claim columns and a source-URL column - any names, any count, pointed to explicitly. The raw Claygent-response column, if your export even has one, is expected to be blank and is simply ignored:

company_name,claygent_extraction,funding_series,is_confirmed,source_url
Acme Robotics,,Series B,CONFIRMED,https://example.com/news/acme-series-b

Output CSV is the same rows plus verification_verdict, verification_detail, verification_confidence_note, and verification_checked_fields columns.

Tests

pip install pytest requests-mock
python -m pytest tests/ -v

33 unit tests, no Anthropic API key, no real network calls (requests_mock intercepts the transport layer for fetcher/engine tests, and raises on any URL that wasn't explicitly registered - so a test can't silently succeed by hitting the real internet). They verify multi-column claim mapping, blank- column filtering (including the all-blank NO_CLAIM_DATA case), the fetch failure taxonomy, and the full engine orchestration against a FakeClaimVerifier double. They do not verify actual LLM judgment quality - whether ClaudeClaimVerifier correctly distinguishes MATCH from MISMATCH on real page text needs a real API key and human review on real claims, which is a manual smoke-testing step, not something covered by the automated suite.

Manual smoke test not yet run in this environment: no ANTHROPIC_API_KEY was available when this was built, so ClaudeClaimVerifier, the CLI's non---estimate-only path, and the MCP tool's real verification call are untested against the live API. Run the CLI once against a real CSV with a real key to close this gap - see "Known limitations" below.

Test data

tests/fixtures/ contains only synthetic data: invented company names, invented URLs, modeled on the real column shapes seen in an actual client Clay export but with no real client data. No Employers/ client CSVs were used in the test suite or in this README, by deliberate choice - see the project's build plan for the reasoning.

Known limitations

• No raw-JSON-blob path. This tool assumes claim data always arrives as N pre-extracted columns, since that's what a real Clay export produces. If some future Clay export config somehow does carry a real JSON blob with data in it, this tool won't parse it - you'd need to select the fields into their own columns first, the same way you already do for a normal Clay export.
• Manual smoke test against the live Claude API hasn't been run yet in this build environment - see "Tests" above. This is the single biggest gap before treating this as demo-ready.
• Cost estimates are heuristics. The pre-flight estimate uses a ~4-characters-per-token approximation, not the API's real tokenizer.
• Model pricing table is a static, manually-maintained reference (MODEL_PRICING_PER_MILLION in app/engine.py). Verify at anthropic.com/pricing before relying on it for real budgeting.
• Paywall/JS-rendered detection is heuristic, not exhaustive: it flags pages with very little extracted article text and, for paywalls, a known marker phrase. A paywall or JS site that doesn't match either signal will likely just extract as thin real text and get judged UNVERIFIABLE by the LLM stage instead of being caught earlier - a reasonable fallback, but not the same as a purpose-built paywall detector.
• One retry on timeout, one vendor implemented. fetcher.py retries a timeout once before giving up; ClaimVerifier has one working implementation (Claude). The interface is vendor-agnostic, but no second implementation has been written.
• No non-technical interface yet. This ships CLI + MCP only. A Sheets/Clay-native interface for GTM operators who aren't in a terminal is planned as later work, not built here.
• No case-study writeup yet. A demo writeup for job-hunting/Portfolio/case-studies/ is planned as later work, once the live-API smoke test above has actually been run and its real output can be shown.