mcp-eval-harness

MCP-based code evaluation harness — sandboxed execution + LLM quality scoring.

An MCP server that exposes 4 evaluation tools (no AI on the server itself — it's just a tool provider). Any MCP-compatible client can connect and use these tools: the included AI agent, Cursor, Claude Desktop, or anything that speaks MCP.

Tools

Tool	Description
run_code	Execute code in a sandboxed Docker container (or local fallback). Returns stdout, stderr, exit code.
run_tests	Run code against test cases. Returns pass/fail for each test with actual vs expected output.
score_answer	LLM-based quality scoring (0-10) against a rubric with breakdown by correctness, efficiency, readability, edge cases.
detect_hallucination	Check if text contains claims not supported by provided context. Returns confidence and unsupported claims.

Architecture

┌─────────────────────┐       MCP (stdio)       ┌─────────────────────┐
│                     │  ◄───────────────────── │                     │
│   MCP Client        │       tool calls         │   MCP Server        │
│   (AI agent)        │  ────────────────────► │   (tool provider)   │
│                     │       results            │                     │
│   Has the brain —   │                          │   No AI here —      │
│   decides what to   │                          │   just runs code,   │
│   evaluate & when   │                          │   executes tests,   │
│                     │                          │   calls scoring APIs │
└─────────────────────┘                          └─────────────────────┘

Server = pure tool provider. Exposes run_code, run_tests, score_answer, detect_hallucination over MCP stdio. No decision-making, no orchestration.

Client = the brain. An AI agent (GPT-4o-mini) that decides which tools to call, in what order, and synthesizes a final evaluation report.

Transport = stdio. The client spawns the server as a subprocess and communicates via stdin/stdout. This is how MCP stdio works — same as how Cursor or Claude Desktop talk to MCP servers.

Use with Any MCP Client

The server isn't locked to our AI agent. You can use it with:

Cursor — Add to your MCP config:

{
  "mcpServers": {
    "eval-harness": {
      "command": "npx",
      "args": ["tsx", "server/index.ts"],
      "cwd": "/path/to/mcp-eval-harness"
    }
  }
}

Claude Desktop — Add to claude_desktop_config.json:

{
  "mcpServers": {
    "eval-harness": {
      "command": "npx",
      "args": ["tsx", "server/index.ts"],
      "cwd": "/path/to/mcp-eval-harness"
    }
  }
}

MCP Inspector (for debugging):

npx @modelcontextprotocol/inspector npx tsx server/index.ts

<!-- Yes, this uses Docker not Firecracker microVMs. It's a demo, not a bank. In prod, use E2B or self-hosted Firecracker. -->

Stack

Package	Purpose
@modelcontextprotocol/sdk	MCP server + client protocol
@ai-sdk/mcp	Vercel AI SDK MCP client adapter
ai	Vercel AI SDK core — generateText, generateObject
@ai-sdk/openai	OpenAI provider (direct API key, NOT Vercel Gateway)
zod	Schema validation for tool inputs and structured outputs
dotenv	Environment variables

Setup

git clone https://github.com/salmankhan-prs/mcp-eval-harness.git
cd mcp-eval-harness
pnpm install
cp .env.example .env
# Add your OpenAI API key to .env

Pull Docker images for sandboxed execution (optional — falls back to local if Docker isn't running):

docker pull node:22-alpine
docker pull python:3.12-alpine

Usage

pnpm eval examples/problems.json

Example Output

Evaluating: Two Sum
Language: javascript
============================================================

Test Results: 3/3 passed

Quality Score: 8.5/10
  Correctness:  9/10
  Efficiency:   9/10
  Readability:  8/10
  Edge Cases:   8/10

Hallucination Check: No hallucinations detected

Verdict: PASS

Project Structure

mcp-eval-harness/
├── README.md
├── package.json
├── tsconfig.json
├── .env.example
├── .gitignore
├── server/
│   ├── index.ts                     # MCP server — registers tools, starts stdio transport
│   └── tools/
│       ├── run-code.ts              # Docker-sandboxed code execution (hacky but works)
│       ├── run-tests.ts             # Test runner — executes code against test cases
│       ├── score-answer.ts          # LLM-based quality scoring against rubric
│       └── detect-hallucination.ts  # LLM-based hallucination detection
├── client/
│   └── index.ts                     # AI agent that orchestrates evaluation via MCP
└── examples/
    └── problems.json                # Sample coding problems with test cases

How Each Tool Works

run_code — Detects if Docker is available. If yes, runs code in a container with --network=none, memory/CPU/PID limits. If Docker is unavailable, falls back to local child_process with a 10s timeout. In production you'd swap this for a Firecracker microVM pool.

run_tests — Wraps the candidate solution + each test input into a runnable script. Calls run_code for each test case and compares stdout to expected output. Convention: solution must define a solution() function.

score_answer — Sends the problem + solution to GPT-4o-mini with a structured output schema (Zod). Returns an overall score and per-criterion breakdown. This is what ReasonCore's expert evaluators do, but automated.

detect_hallucination — Sends a claim + context to GPT-4o-mini. Returns whether unsupported claims exist, a confidence score, and specific unsupported claims.

Prerequisites

• Node.js 22+
• Docker Desktop running (optional — falls back to local execution)
• OpenAI API key in .env