veriloop

Verifier-scored agent runtime. A minimal ReAct loop you can fully inspect: every step is budgeted, scored by a verifier function, and written to a replayable JSONL trace. The tools are exposed once over MCP so they plug into any framework. No LangChain, no LangGraph, no CrewAI — just the primitives, typed.

The problem

Agent demos hide their failure modes. A polished screencast shows the run that worked; it does not show the runs that looped, hallucinated a tool name, sent malformed arguments, or burned forty steps on a two-step task. Framework-first builds make this worse: when the loop belongs to someone else's abstraction, you can't see why a run went wrong, only that it did.

The position of this repo: the first agent worth building is a minimal loop you fully understand. Frameworks earn their place later, for durable stateful orchestration — not as a substitute for knowing what your agent actually does at each step.

veriloop makes every decision inspectable and scoreable:

• Hard step budget — the loop terminates, provably (there's a test for it).
• Verified tool calls — args are schema-checked before execution; bad calls are blocked, logged, and fed back so the model can self-correct.
• A verifier score on every step — verifier functions return scored judgments (reward-function discipline), logged alongside the step, not bolted on after.
• Full JSONL trace — every run is a replayable, diffable artifact. Traces and scores are first-class outputs, not debug noise.
• Kill switch — a threading.Event stops the run cleanly from outside.
• Tools over MCP — the same three-tool registry the loop executes is served over MCP stdio, so any MCP-capable client reuses it unchanged.

Approach

The loop is the classic ReAct cycle — think → act → observe — kept deliberately small (a few hundred lines of typed Python across loop.py, verifiers.py, trace.py, tools.py, llm.py):

1. Think — an LLMClient (protocol; any model plugs in — a deterministic FakeLLM ships for tests/CI, an OpenRouterClient with per-call token/cost accounting and a hard cost cap ships for live runs) looks at the task and the step history and emits a Decision: a thought plus exactly one of tool_call / answer.
2. Verify — pre-act verifiers judge the decision. The built-in SchemaVerifier validates tool args against the tool's pydantic schema; a failing judgment blocks execution.
3. Act — the tool runs (calculator, sandboxed file_read, offline web_fetch stub), or the blocked call becomes an error observation.
4. Observe & score — post-act verifiers score the step (the built-in BudgetVerifier scores remaining headroom). Decision + judgments + observation are appended to the JSONL trace as one step record.
5. Repeat until the model answers, the budget is exhausted, the kill switch fires, or too many consecutive failures trip the fallback stop.

Retry policy is budget-honest: a rejected step consumes a step and its error is fed back as the observation — there are no free retries, so traces never lie about cost.

Verifiers are the extension point: implement the Verifier protocol (a name, a phase, and judge(ctx) -> Judgment) to add task-specific checks, and their scores land in the same trace.

Evaluation

The eval plan is 30 cases; 10 seed cases are committed in eval/cases.seed.jsonl (arithmetic, sandboxed file tasks, and recovery/adversarial cases: malformed args, unknown tools, sandbox escapes, budget traps). The remaining 20 follow the same schema: 10 more multi-step arithmetic/file compositions and 10 more adversarial cases.

Metrics, measured by eval/harness.py (scripted) and eval/run_live.py (live):

Metric	Definition	Scripted (CI)	Live: openai/gpt-4o-mini (2026-07-19)
Task completion rate	runs ending completed	9/10	9/10
Expected-outcome pass rate	all of a case's checks pass	10/10	5/10
Mean steps-to-completion	steps used, completed runs only	2.33	1.89
Mean verifier score	all judgments, all steps	0.883	0.945
Verifier-blocked steps	tool calls blocked pre-execution	2 (scripted by design)	0
Budget-exhaustion rate	runs ending budget_exhausted	1/10 (by design)	0/10
Total cost	from OpenRouter usage accounting	$0	$0.0016 (8,066 in / 666 out tokens)

Honesty note: the live column is one run of the 10-case seed set — single repetition, temperature 0, max 512 tokens/call, ≤6 steps/case, via OpenRouter. The raw artifacts for that exact run — per-case JSONL traces, summary.json with per-case token/cost accounting, and a failure analysis — are committed at eval/results/live-gpt4omini-2026-07-19/. The 5/10 live pass rate is signal, not embarrassment: the misses are adversarial cases where the model behaved reasonably (refused a sandbox-escape without calling the tool, declined an infinite-loop task, never produced the malformed calls the recovery cases script for) — dissected case-by-case in results.md. Cases were not tuned to make the model pass.

Scripted mode exercises the runtime's plumbing deterministically (CI reruns it; output goes to the gitignored eval/results.md); live mode measures model capability, and its numbers come only from committed run artifacts.

Sample trace

Illustrative format example — not output from a recorded run. Generate a real one with uv run python -m veriloop (written to traces/demo.jsonl).

{"type":"run_start","task":"What is 17 * 23?","max_steps":8,"ts":1789700000.01}
{"type":"step","step":0,"decision":{"thought":"Arithmetic; use the calculator.","tool_call":{"tool":"calculator","args":{"expression":"17 * 23"}},"answer":null},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"args match calculator schema"},{"verifier":"budget","phase":"post_act","score":1.0,"passed":true,"reason":"step 1/8; headroom 1.00"}],"observation":{"ok":true,"content":"391"},"ts":1789700000.02}
{"type":"step","step":1,"decision":{"thought":"The observation has the product.","tool_call":null,"answer":"17 * 23 = 391"},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"final answer step; no tool call to validate"},{"verifier":"budget","phase":"post_act","score":0.88,"passed":true,"reason":"step 2/8; headroom 0.88"}],"observation":null,"ts":1789700000.03}
{"type":"run_end","status":"completed","answer":"17 * 23 = 391","steps_used":2,"mean_score":0.97,"error":null,"ts":1789700000.03}

Run it

uv run python -m veriloop              # scripted demo; prints the trace it wrote
uv run pytest                          # real tests: budget, verifiers, trace round-trip, tools
uv run ruff check .                    # lint
uv run python eval/harness.py          # scripted eval; writes eval/results.md
OPENROUTER_API_KEY=... uv run python eval/run_live.py   # live eval (gpt-4o-mini); writes eval/results/live-*/

Docker one-liner:

docker build -t veriloop . && docker run --rm veriloop

MCP server

The tool registry is served over MCP stdio:

uv run python -m veriloop.mcp_server

Plug it into any MCP client — e.g. Claude Code:

{
  "mcpServers": {
    "veriloop": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/veriloop", "python", "-m", "veriloop.mcp_server"]
    }
  }
}

Same tools, same schemas, zero duplication: the loop and the MCP server share one ToolRegistry.

Limitations

• Scripted CI numbers measure plumbing, not intelligence. FakeLLM replays fixed decision paths. The live column is measured but thin: one model, one run, 10 cases — not a benchmark.
• Verifiers are heuristic functions, not learned reward models — dense signal, but only as good as the checks you write.
• web_fetch is an offline stub by default; enabling live fetches without an allowlist is an SSRF risk (flagged in tools.py).
• Single-threaded, one tool call per step — no parallel tool fan-out, no streaming.
• Traces are replayable but the loop is not resumable — replay reconstructs what happened; it does not restart a run mid-flight.
• The sandbox fences file_read only; the calculator and fetch stub have their own guards, but there is no process-level isolation.

Layout

src/veriloop/     loop.py  verifiers.py  trace.py  tools.py  llm.py  mcp_server.py
eval/             cases.seed.jsonl  harness.py  sandbox/
tests/            budget, verifier, trace round-trip, tool-safety tests
docs/             DECISIONS.md
ARCHITECTURE.md   state machine, verifier contract, MCP layering

MIT — see LICENSE.