mcp-incident-copilot

An MCP server that gives AI agents guarded, audited, read-only access to ops tooling (alerts, metrics, logs, deploys, runbooks), plus a triage agent that diagnoses an injected Kafka-lag incident end to end in 7 tool calls, verified against ground truth in CI, with the full transcript committed.

What this solves

• Agents helping with incidents need access to logs and metrics, and handing them raw credentials is how "AI-assisted ops" fails security review; here every capability is a schema-validated, read-only MCP tool with an audit trail, and the agent has no side door by construction (ADR-001).
• Post-incident "how did we conclude that" is usually archaeology; here the transcript of every tool call, with the agent's stated reasoning per step, is the committed artifact of the diagnosis.
• Correlation-based triage is easy to get confidently wrong; this repo's war story is its own agent blaming a deploy 35 minutes late because its inflection detector found where the metric grew fastest instead of where the regime changed, caught by a ground-truth test and fixed with CUSUM.

Why this exists

Two things are true at once: agents are genuinely useful in incident triage (they read faster, correlate wider, and never forget to check deploy markers), and no responsible platform team will hand an agent Grafana admin and kubectl. The resolution is a governed tool surface. This repo builds one over MCP: five read-only tools (list_alerts, list_deploys, query_metric, search_logs, get_runbook), each with a declared schema, argument validation that rejects unknown and missing arguments as typed JSON-RPC errors, result shapes designed for a context window, and an audit trail that becomes part of the incident record.

On top of it runs a triage agent that speaks only MCP, one call at a time. The local planner is a deterministic policy encoding the generic-triage runbook (alerts first, then the timeline of deploys and metric inflections, then mechanism evidence from logs, then the prescribed action); LangGraph over Azure OpenAI is the production planner behind the same interface (ADR-002). Determinism is what buys the strongest claim in the repo: the incident environment is seeded with ground truth (a checkout-consumer deploy at minute 60 introduces a slow deserializer; lag ramps; payment errors follow; pages fire at 74 and 81), and CI fails the build if the agent's diagnosis names the wrong service. The whole loop, transcript, and diagnosis are reproducible from one command.

The incident and the diagnosis

The recorded triage, step by step (this is the actual committed out/transcript.json rendered):

Tech stack

Technology	Role in this project	Why chosen here
MCP (JSON-RPC 2.0)	The only door to the tools	Governance is the transport: schemas, typed errors, audit, identical for in-repo and external agents
Python 3.11 asyncio	Server, tools, agent loop	The loop is IO-shaped tool calls; the whole triage runs in milliseconds locally
NumPy	Metric summarization + CUSUM onset detection	The inflection detector is the analytical heart, and its failure mode is the war story
Seeded telemetry environment	Ground truth	The diagnosis is assertable, which is what made the detector bug catchable in CI
LangGraph + Azure OpenAI (prod adapter)	Production planner	Same next-action interface; swaps in without touching tools, guardrails, or transcript
structlog	JSON logs	The CLI's summary line is the machine-readable triage record
pytest + pytest-cov	Suite	10 tests: protocol guardrails, audit, detector stability across seeds, ground-truth diagnosis; 91 percent measured
GitHub Actions	CI	Lint, tests, and a triage smoke that asserts the correct root cause on every push

Quickstart

Prerequisites: Python 3.11+, git.

git clone https://github.com/<you>/mcp-incident-copilot.git
cd mcp-incident-copilot
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pytest --cov=copilot          # includes the ground-truth diagnosis test

python -m copilot.cli         # run the triage; writes out/transcript.json + out/diagnosis.json
cat out/diagnosis.json

Point any MCP client at McpOpsServer to use the same tools interactively; the server speaks standard initialize / tools/list / tools/call.

The numbers

Metric	Value
Tool calls to correct diagnosis	7 (transcript committed)
Onset detection vs injected fault	lag onset found at minute 59 (injected 60); error-rate at 60; stable within ±3 across seeds 1, 7, 42, 99 (tested)
Pre-fix detector error	blamed a deploy 35 minutes late (notifications@95 vs checkout-consumer@60)
Guardrail tests	unknown tool, unknown argument, missing argument: all typed errors; audit trail order asserted
Coverage	91 percent (206 stmts), 10 tests

Architecture decisions

ADR-001: the in-repo agent goes through MCP too; a governance layer with a side door is decoration. ADR-002: the boring choice of a deterministic runbook policy locally with LangGraph/Azure OpenAI as the prod planner, and why that is what makes the diagnosis CI-assertable.

Intentionally out of scope

• Write actions (restarts, rollbacks, scaling). Read-only is a security posture, not a limitation; a write tool needs approvals and idempotency design of its own. Trigger: a human-in-the-loop approval flow.
• Live Prometheus/Loki adapters. The tool interfaces are shaped for them (range summaries, capped log results); adapters are mechanical and listed first in future work.
• LLM-planner demo recordings. A canned LLM transcript would demo nothing verifiable; the deterministic transcript is complete and honest (ADR-002).

Security and compliance

The agent holds no credentials to any underlying system: it can only call declared MCP tools, all read-only. Argument validation rejects anything outside the schema. Every call is audited in order, and the audit is part of the incident artifact. Log results are capped and pre-structured; nothing streams raw production data into a model context unbounded.

Failure modes

Failure	Detection	Behaviour	Recovery
Agent asks for an unknown metric	Tool returns the available list instead of an error dead-end	Planner can re-select; transcript shows the miss	By design
Malformed tool call	Typed -32602 with the missing/unknown fields	Step fails visibly in the transcript	Planner retries with corrected arguments
Flat/noisy metric queried for inflection	inflection_significant: false	Planner must not build a timeline on it	The strength gate exists because of the war story's adjacent failure mode
Two plausible deploys near one onset	Correlation window picks the nearest; transcript shows both	Ambiguity is visible, not hidden	Human reads the transcript; tightening to causal probes is future work
Wrong diagnosis regression	CI triage smoke compares against ground truth	Build fails	The detector fix landed exactly this way

Hardest problem solved

The agent's first full run produced a confident, well-evidenced, wrong answer: it blamed the notifications deploy at minute 95. Ground truth was the checkout-consumer deploy at minute 60. Every step in the transcript looked reasonable; the failure was one number: query_metric reported the lag inflection at minute 92.

The detector computed inflection as the argmax of the smoothed slope. For a metric that goes flat-then-ramp, the slope after onset is roughly constant with noise, so the largest smoothed slope lands anywhere inside the ramp, and this replay put it 32 minutes late, right next to an innocent deploy. The correlation logic then did its job correctly on a wrong input, which is what made the output convincing.

The fix (commit fix(tools): detect metric regime onset with CUSUM, not max slope) reframes the question: not "where does the series move fastest" but "where does the slope regime change". CUSUM over the centered first differences answers that: the cumulative sum falls while the slope is below its mean and rises after, so its extremum is the onset. Measured: lag onset 59, error-rate onset 60, against an injected fault at 60, stable within ±3 minutes across four seeds (tested). A strength score gates flat series as insignificant so the planner cannot build a timeline out of noise. The general lesson: in incident tooling, the analytical primitive underneath the reasoning, not the reasoning itself, is where confident wrongness comes from, and ground-truth tests are how you catch it.

Future work

• Prometheus and Loki adapters behind the existing tool interfaces.
• A human-approval write tool (propose_rollback) that files the action for confirmation rather than executing.
• LangGraph planner wiring as the prod profile with per-step token budgets.
• Multi-incident environments (concurrent faults) to stress the timeline logic.
• First metric to watch in adoption: transcript length distribution. Rising call counts per diagnosis means the tool results are not carrying enough signal per call.

License

MIT