evalgate-mcp

An MCP server that lets an agent read its own eval results. Wraps evalgate — run a suite, explain a failure, check for drift, and ask whether the judge was ever calibrated.

Phase 1 is read-only. Nothing here writes a file. run_suite executes the system under test and may bill a provider, but the result comes back in memory rather than landing in .evalgate/result.json. Write access is phase 2, deliberately.

---

Why an agent should be allowed to do this

Handing an agent its own grader is circular. It's also useful, and the circularity is manageable as long as one thing is never hidden: whether anyone measured the grader.

So every result from run_suite carries a judgeStatus block, in one of five states:

state	meaning
no-judge	No judged assertions ran. Nothing to qualify.
calibrated	The judge has a passing stamp bound to its own id. Agreement is published.
uncalibrated	No stamp exists. Scores are unqualified — a judge nobody measured is a random number generator with good manners.
judge-mismatch	A stamp exists but belongs to a different judge. Agreement is withheld, because one judge inheriting another's credibility is worse than no stamp at all.
failed-calibration	The judge last failed calibration. Agreement is reported with the failure.

That block is the point of this server. An agent that can score its own output and can't see the state of its grader will report success it didn't earn.

Tools

tool	cost	what it does
list_suites	free	Load and validate suites — cases, assertions, thresholds.
explain_result	free	Read a result artifact and explain what failed, down to the claim-level grounding breakdown.
check_drift	free	Analyze history.jsonl for slow decline the per-PR gate can't see.
judge_status	free	Read the calibration stamp and report agreement, bias state, and judge binding.
run_suite	executes the SUT; may call providers	Run the suites and return the scored result. Writes nothing.

Bad paths and malformed suites come back as tool errors with the message intact, not as thrown stack traces the model can't act on.

Install

Not on npm. evalgate isn't either — both install from git.

git clone https://github.com/HolisticShort/evalgate-mcp.git
cd evalgate-mcp
npm install          # builds on install via `prepare`
npm run smoke        # end-to-end check against evalgate's example suite

npm run smoke expects an evalgate checkout at ../evalgate, or set EVALGATE_DIR.

Use it with Claude Code

Add to .mcp.json in the project you want to gate:

{
  "mcpServers": {
    "evalgate": {
      "command": "node",
      "args": ["/absolute/path/to/evalgate-mcp/dist/server.js"]
    }
  }
}

Restart Claude Code, then /mcp to confirm the server connected and the five tools are listed.

Phase 2

Write access — letting an agent add cases, adjust thresholds, and write baselines. Held back on purpose. An agent editing the test cases that grade it is a different trust question from an agent reading them, and it deserves its own design rather than arriving as a convenience.

License

MIT