ledger-mcp

I built this over two days to figure out where an MCP server's correctness actually lives — in the tool implementation, or in the schemas the model sees.

The domain is a double-entry ledger, because it has a hard definition of "wrong": debits have to equal credits, or the books don't reconcile. So there's no room to hand-wave that the agent did "something reasonable." Either the reconcile call comes back balanced or it doesn't.

What's in the repo

ledger.py is a small SQLite-backed double-entry ledger. Amounts are integer cents so nothing rounds. test_ledger.py covers it with 7 pytest cases.

server.py wraps the ledger as an MCP server with four tools: create_account, record_transaction, get_balance, reconcile.

Day 1 ended there — server running, the four tools discoverable from Claude Desktop, the sample scenario (record a $250 sale, an $80 supplies purchase on account, pay it off) ran end-to-end.

Day 2 is the eval work.

• evals/cases.py — 10 test cases split across three buckets: five that should just work, three that probe specific design leaks I noticed in Day 1 (dollar-vs-cent ambiguity, no discovery tool for existing accounts, silent account creation on ambiguous prompts), and two restraint cases where the agent shouldn't touch the ledger at all (accounting theory question, advice question).
• evals/harness.py — runs each case through Claude via the Anthropic Messages API with the real MCP server spawned as a subprocess. Each case gets its own SQLite file (server.py reads LEDGER_DB_PATH from the env), so cases are fully isolated. Every tool call is captured, transcripts land in evals/transcripts/.
• evals/inspect_request.py — dry-runs a case and prints the exact request body the harness would POST. No API key, no cost. This turned out to be the most useful thing in the repo.

What the schema audit found

Before running anything against the API I used inspect_request to look at the tool payload the model would actually receive. The docstrings and the JSON schema were telling different stories.

record_transaction.entries at baseline-v1:

"entries": {
  "items": { "additionalProperties": true, "type": "object" },
  "type": "array"
}

The docstring said each entry needs account, amount_cents, and side. The schema said "array of arbitrary objects." Same pattern on side (no debit/credit enum), on account_type (no enum for the five valid values), and on amount_cents — no type constraint anywhere inside entries.items, so nothing prevents the model from passing 42.50 and getting silently truncated to 42 at the SQLite integer column.

Everything I cared about lived in prose.

Full baseline payload: evals/baseline_schemas/tools_payload_baseline.txt.

What I changed

Four edits to server.py, in expected-impact order:

1. Introduced a LedgerEntry pydantic model with account: str, amount_cents: int (gt=0), side: Literal["debit","credit"]. entries.items now $refs a proper nested type in the schema.
2. Literal["asset","liability","equity","revenue","expense"] on account_type. Now visible in the schema as an enum.
3. Rewrote get_balance's docstring to explain the sign convention — positive means "normal balance" for assets and expenses, the reverse for liabilities/equity/revenue. Signal change, not a schema change, but it's semantic detail types can't carry.
4. Added a concrete example call inside record_transaction's docstring.

Fixed payload: evals/baseline_schemas/tools_payload_fixed.txt. The two files are meant to be diffed.

7/7 unit tests still pass.

What I did not do

The harness is real. Running it against claude-sonnet-4-5-20250929 takes ~3 minutes and would cost a few cents. I did not run it against both tags for one honest reason: I hit my Anthropic API credit limit before I could. So this README doesn't quote a pass/fail delta — the schema diff is the evidence here. Anyone with a key can reproduce the behavioral run in one command.

Two things I deliberately left out of the M10 fix pass, even though NOTES.md argues for them:

• A list_accounts discovery tool. Right now the model has no way to check what accounts exist before referencing them.
• Structured error objects instead of raw ValueError strings.

Both are real design improvements, not schema tightening. I kept the fix pass to one thing so the before/after diff would be readable.

Running it

uv sync
uv run pytest -q                                       # unit tests
uv run python -m evals._smoketest                      # verify MCP plumbing, no API
uv run python -m evals.inspect_request simple_sale     # inspect the wire payload

Full agent eval:

export ANTHROPIC_API_KEY="sk-ant-..."
export EVAL_MODEL="claude-sonnet-4-5-20250929"
uv run python -m evals.harness

Per-case transcripts go to evals/transcripts/, and _summary.json collects the pass/fail table.

Tags

• baseline-v1 — Day 1 tools as originally shipped, schemas untouched.
• fixes-v1 — after the M10 schema tightening.

git diff baseline-v1 fixes-v1 -- server.py is the source-level change. The two files in evals/baseline_schemas/ are the wire-level change.

NOTES.md

The raw build log lives in NOTES.md. It reads rough on purpose — it's the actual notes I wrote while building, not a cleaned-up narrative. If anything in this README seems too tidy, that file has the friction.