CodePecker

An MCP server that reviews a piece of code across four dimensions — security, standards, production readiness, sustainability — automatically fixes the issues, verifies the fix by running the code's tests, and reports what it did. Any MCP-capable agent (Claude Code, Codex, Copilot) can call it as a tool; there's also a CLI for local demos.

review → remediate → run tests → repeat (bounded)   →   findings + fixed code + diff + citations

How it works

For each of the four dimensions, CodePecker gathers findings two ways:

• Deterministic checks (regex/code) for rules that must be caught reliably — hardcoded secrets, eval/unsafe deserialization, bare except, missing tests. No LLM, so they never "forget".
• An LLM judge for the nuanced rules (input validation, logging, timeouts, N+1 queries, …), with guardrails: it may only cite rules from the batch it was given, any evidence it quotes must appear in the code, and severity/dimension come from the rule metadata — hallucinated findings are dropped in code.

Each rule lives in a markdown file in knowledge_base/ (RAG), tagged deterministic: true|false so it's enforced by exactly one path. A hand-written, bounded agent loop then asks the model to remediate and re-runs the tests — a fix that resolves a finding but breaks the tests is not accepted.

The full "why" behind every decision is in DECISIONS.md.

Setup

Requires Python 3.10+.

git clone <this repo> && cd CodePecker
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e . --no-deps            # makes `codepecker` + `python -m codepecker.*` work
cp .env.example .env                  # then add your key (below)

Keys — the default needs just one. Text runs on Groq (fast Llama 3.3 70B) and embeddings run locally (no key). Put your Groq key in .env:

GROQ_API_KEY=...        # free key at https://console.groq.com/keys

Provider-agnostic via LiteLLM — switch model or provider with no code change, e.g. CODEPECKER_TEXT_MODEL=openai/gpt-4o (one key does both), or go fully offline with CODEPECKER_TEXT_MODEL=ollama/llama3.1. See .env.example.

Usage

CLI (local demo)

codepecker examples/sample_bad_code.py
# or:  python -m codepecker.cli examples/sample_bad_code.py

Prints the findings, the remediated code, a unified diff, the rules cited, and a metrics summary. Each run is appended to metrics.jsonl.

Reliable live demo: the loop makes many LLM calls, so a free tier's tokens-per-minute cap can throttle a full run. The loop is resilient — a mid-run rate limit is recorded and the review still completes (with degraded coverage noted) rather than crashing. For a smooth end-to-end demo, use a higher-limit tier or run the text model locally: CODEPECKER_TEXT_MODEL=ollama/llama3.1 (no key, no limits).

MCP server (in an agent)

Run it over stdio:

python -m codepecker.server

Connect it to an MCP client with this config (Claude Desktop / Codex / Copilot use the same shape):

{
  "mcpServers": {
    "codepecker": {
      "command": "/absolute/path/to/.venv/bin/python",
      "args": ["-m", "codepecker.server"],
      "env": { "GROQ_API_KEY": "your-key" }
    }
  }
}

For Claude Code:

claude mcp add codepecker -- /absolute/path/to/.venv/bin/python -m codepecker.server

The server exposes one tool, review_and_remediate(code, language).

Evaluation

python eval/run_eval.py

Runs CodePecker over the labeled golden set (eval/golden/) and reports precision/recall/F1 per dimension, remediation resolution + test-pass rates, and mean iterations/latency; writes eval/report.json. This is the "how do I know it's good?" evidence and is meant to run in CI. (It drives the full loop over every sample, so use a decent rate-limit tier.)

Design decisions (the short "why")

Decision	Why
MCP server, not a bot/CI check	Reusable across agents, and reviews in the loop rather than post-hoc
Hand-written loop, no LangChain	Bounded task; transparent and testable control flow
RAG over fine-tuning for rules	Rules stay editable, auditable, and citable (markdown files)
Deterministic secrets/eval/except vs LLM for nuance	Reliability where it's non-negotiable, flexibility where it's fuzzy
Tests gate success	A fix that breaks behavior is a failure, not a fix
Judge guardrails (constrained citations + evidence grounding)	Hallucinated findings are dropped by code, not trusted
One LLM seam (LiteLLM behind LLMClient)	Swapping provider — or going offline — is a config change
Sandboxed test run (subprocess + timeout)	Executing untrusted code is a security boundary

Full detail — every alternative considered and every trade-off — in DECISIONS.md.

Project layout

src/codepecker/
  config.py            env-driven model IDs + tuning constants
  types.py             LLM Protocols (DIP/ISP) + the Finding type
  llm_client.py        the only module that talks to a provider (LiteLLM)
  vector_store.py      ChromaDB adapter (RAG index)
  knowledge/loader.py  parse + embed the markdown knowledge banks
  tools/
    deterministic_checks.py   code checks, keyed by rule id
    judge.py                  batched, guardrailed LLM judge
    run_tests.py              sandboxed pytest runner
  agent.py             review_and_remediate() — the bounded loop
  metrics.py           append-only metrics log + summary
  cli.py               local demo runner
  server.py            FastMCP server (stdio)
knowledge_base/        the rules: security/ standards/ readiness/ sustainability/
eval/                  golden samples + run_eval.py
tests/                 the test suite

Testing

pytest -q                       # 85 offline tests (local embeddings, faked LLM)
pytest -m "live or not live"    # + the 1 live acceptance test (needs GROQ_API_KEY)

The default suite is fully offline and deterministic; the one live test is opt-in.

Non-goals / next steps

MVP simplifications, called out honestly (see DECISIONS.md):

• Sandbox is a subprocess + timeout, not a container — production wants gVisor/a microVM with no network and resource limits.
• Deterministic checks are regex-based — production would use AST analysis.
• Local embeddings (all-MiniLM-L6-v2) trade recall for zero keys — swap in a hosted embedder for higher-quality retrieval at scale.
• Not yet: metadata-routed retrieval for very large rule sets, a metrics dashboard, real GitHub integration, runtime energy profiling, remote HTTP/Cloud Run deploy.