MCP Eval Demo

A worked example of using evaluations to verify that an LLM agent can actually use an MCP server effectively — not just that the server's code is correct.

Unit tests answer "does delete_note delete a note?" They cannot answer the questions that decide whether an MCP server is any good in practice:

• Does the agent find the right note when the user describes it in words instead of by id?
• Does it notice that a listing preview was truncated, or does it answer from half a note?
• Does it realize update_note overwrites, or does it silently destroy the user's content when asked to "add a line to my grocery list"?
• Does it recover from an error message, or give up?

Those are properties of the tool surface — names, descriptions, schemas, result shapes, error text — and the only way to check them is to run a real agent against the server and grade what it did. That is what this repo is for.

Status

The MCP server, its infrastructure, and the eval harness are all in place.

The server under test: Notes MCP

An in-memory notebook. State lives in the server process and is discarded on exit, so every eval run starts from the same known corpus (see seed.py).

Tool	Behaviour hint	What it does
create_note	write	Creates a note; titles must be unique case-insensitively.
get_note	read-only	Returns one note's full content, by id.
list_notes	read-only	Lists notes newest-updated first, as truncated previews, with an optional substring query.
update_note	destructive	Overwrites the title and/or content of a note.
delete_note	destructive	Permanently removes a note.

Several design choices exist specifically to give the evals something to catch:

• Ids, not titles. Every mutating tool takes a note_id, so an agent asked to change "my grocery list" must look the id up first. This is where agents commonly guess.
• Truncated previews. list_notes returns only the first 120 characters of each note, flagged with content_truncated and content_length. An agent that answers a content question straight from a listing gets it wrong; a good one calls get_note.
• Replace, not append. update_note overwrites. "Add eggs to my grocery list" is therefore a read-modify-write, and an agent that skips the read destroys data.
• Errors that teach. Every failure names the offending value and points at the tool that would resolve it, so an agent has a path forward rather than a dead end.

Layout

src/notes_mcp/
  models.py    Pydantic models — also the tool input/output schemas the agent sees
  store.py     In-memory storage and its error types
  seed.py      Fixed corpus: stable ids and timestamps, so evals are reproducible
  server.py    MCP tool definitions, descriptions, and annotations
  cli.py       `notes-mcp` entry point
evals/
  agent.py       Builds the pydantic-ai agent under test + local trace capture
  task.py        One agent turn against a freshly seeded server — the thing evaluated
  evaluators.py  Custom pydantic-evals evaluators (tool-not-called, argument-contains)
  cases.yaml     The dataset itself: cases that probe specific MCP misuse patterns
  cases.py       Loads cases.yaml — registers the custom evaluators, picks the judge model
  __main__.py    `python -m evals` — runs the dataset against a live model
tests/
  test_store.py    Unit tests for the storage layer
  test_server.py   Protocol-level tests through a real MCP client session
scripts/
  lint.sh    Ruff + pyright + format check
  test.sh    Unit + protocol tests (fast, free)
  evals.sh   Agent-behaviour evals against a live model (slow, costs money)

The tool descriptions are module-level constants in server.py rather than inline docstrings. Description wording is the main thing you tune in response to a failing eval, and keeping it in one place makes those diffs readable.

Getting started

Requires uv and Python 3.12 (pinned in .python-version).

uv sync                       # create .venv and install everything
uv run scripts/test.sh        # unit + protocol tests
uv run scripts/lint.sh        # ruff check, pyright (strict), format check
uv run pre-commit install     # optional: run the same checks on commit

Running the server

uv run notes-mcp                        # stdio, seeded with the sample notes
uv run notes-mcp --empty                # stdio, no notes
uv run notes-mcp --transport streamable-http

.mcp.json registers the stdio server for this project, so an MCP host launched from this directory — Claude Code, for instance — picks up the notes server automatically and you can drive it by hand.

To read the tool surface an agent would see — the thing these evals are really about — without starting an agent at all:

uv run fastmcp list .mcp.json                   # names, signatures, descriptions
uv run fastmcp list .mcp.json --input-schema    # ...with the full JSON schemas
npx @modelcontextprotocol/inspector uv run notes-mcp   # MCP Inspector, for clicking around

On the mcp version: the server is built on the standalone FastMCP library rather than the copy that used to ship inside the mcp SDK as mcp.server.fastmcp — mcp 2.0 removed that module. FastMCP owns which mcp version it needs (3.x resolves mcp 1.x), so pyproject.toml has no hand-written mcp bound. The eval harness arrives at the same library from the other side: pydantic-ai's MCP client is built on FastMCP's Client. Both halves of this repo therefore agree on a version by construction rather than by a pin someone has to maintain. FastMCP 4 is the step that moves both to mcp 2.x, which is why the dependency is capped below it.

Testing approach

Two pytest layers, run by scripts/test.sh:

• test_store.py covers storage semantics — uniqueness, sort order, limits, timestamps. Fast, exhaustive, no protocol involved.
• test_server.py drives the server through an in-process MCP client session (fastmcp.Client, over FastMCP's in-memory transport), so it asserts on what an agent actually receives: the tool list, JSON schemas, behaviour annotations, structured results, and error text. The protocol is real; only the subprocess and socket are not.

Async tests use the anyio pytest plugin rather than pytest-asyncio, because the MCP client holds a cancel scope open for the life of the session and anyio runs fixture setup and teardown in the same task.

A third kind of check — the agent-behaviour evals — calls a live model and costs money, so it isn't part of the pytest suite at all; it has its own runner and script, described next.

The eval harness

evals/ builds a minimal pydantic-ai agent — a generic one-line system prompt, no few-shot examples, no special-cased instructions — and wires its only tools to an in-process Notes MCP server via pydantic_ai.mcp.MCPToolset (agent.py). The system prompt is deliberately bare: these evals exist to check whether the server's own tool names, descriptions, and schemas are enough to guide correct behaviour, not whether prompt engineering can paper over a weak one.

evals/ lives at the top level rather than under src/: it's dev tooling for this repo, not part of the notes-mcp package anyone would install.

pydantic_evals runs that agent over a Dataset of Cases, each targeting one of the four behaviours from the top of this file:

Case	Checks
delete_by_description_looks_up_the_id_first	Asked to delete "my grocery list note," the agent calls list_notes before delete_note, and deletes the right id.
answers_past_the_list_notes_preview_cutoff	A question whose answer is truncated out of the list_notes preview is only answered correctly if the agent calls get_note.
appending_to_a_note_preserves_its_truncated_tail	"Add crackers to my grocery list" must read the full note first — the update_note call is checked for text that only exists past the preview cutoff.
title_conflict_on_create_is_not_silently_lost	Creating a note whose title already exists must not silently drop the new content or claim a duplicate was created.
deleting_a_nonexistent_note_does_not_fabricate_success	A request to delete a note that doesn't exist must not result in a delete_note call with a guessed id, or a reply claiming success.
simple_lookup_answers_from_the_right_note	A sanity-check happy path.

The cases live in cases.yaml rather than in Python — they're data, so adding a case or rewording a rubric doesn't touch code. cases.py is only the loader: it hands the custom evaluators to Dataset.from_file (a YAML file can only name an evaluator the loader registers) and sets the judge model. The YAML's yaml-language-server header points at cases_schema.json, so an editor can complete and validate evaluator names and their arguments; regenerate it after adding or changing a custom evaluator:

uv run python -c "from evals.cases import write_json_schema; print(write_json_schema())"

Evaluators combine pydantic-evals' built-ins (ToolCorrectness, Contains, MaxToolCalls, LLMJudge for the two cases with more than one valid recovery) with two small custom ones in evaluators.py: ToolNotCalled (assert a tool was never invoked — there's no built-in negative check) and ArgumentContains (substring checks on a tool argument, for "the old content must survive" cases where an LLM's exact wording can't be pinned down with an equality or subset-dict match). Both, like the built-ins, read tool-call spans that Agent.instrument_all() plus a local (send_to_logfire=False) logfire.configure() capture — see configure_instrumentation() in agent.py.

__main__.py runs the dataset, prints a full report, and exits non-zero if anything failed — a task error, a crashed evaluator, or a failed assertion. Run it with:

uv run scripts/evals.sh

Configuring a provider

NOTES_MCP_EVAL_MODEL picks both the provider and model, as a pydantic-ai provider:model string, and defaults to anthropic:claude-haiku-4-5-20251001. Copy .env.example to .env and fill in the section for whichever of these three you're using — scripts/evals.sh loads .env automatically (via python-dotenv; it never overrides a variable already set in your shell) and .env is gitignored:

• Anthropic API (default) — needs ANTHROPIC_API_KEY.
• OpenAI — NOTES_MCP_EVAL_MODEL=openai:gpt-5 and OPENAI_API_KEY.
• Amazon Bedrock — NOTES_MCP_EVAL_MODEL=bedrock:<bedrock-model-id>. Authenticates through boto3's normal credential chain, so there's nothing eval-specific to configure beyond the standard AWS SDK variables: set AWS_PROFILE to use a named profile (AWS_DEFAULT_REGION too, if that profile doesn't already set a region — note it must be AWS_DEFAULT_REGION, not AWS_REGION, which boto3's region resolution doesn't check), or leave both unset to use your default profile/region.

No code branches on the provider — eval_model()'s string is handed straight to both the agent and LLMJudge, and pydantic-ai's infer_model resolves the right client and credentials for whichever provider prefix it sees.