OpsPilot MCP — AI DevOps Agent with a Custom MCP Server

After building DeployMate AI as a RAG diagnostic assistant, this project explores the next layer: how agents actually act on real systems, not just reason about text. OpsPilot is a real MCP (Model Context Protocol) server exposing Kubernetes-style diagnostic tools to an LLM agent, plus a from-scratch agent loop that connects to it exactly the way Claude Code connects to any MCP server under the hood.

No real Kubernetes cluster is needed — fake_cluster.json mocks cluster state, so the whole thing runs anywhere in a couple of minutes, and every piece of the MCP/agent mechanics has been verified against the real, official mcp Python SDK.

---

1. System architecture

+-----------------------------------------------------------------------+
|                              YOU                                      |
|              "Why is payment-api failing?"                            |
+----------------------------------+------------------------------------+
                                   |
                                   v
+-----------------------------------------------------------------------+
|                       agent.py (MCP CLIENT)                           |
|                                                                         |
|   1. Connects to server.py over stdio (subprocess)                    |
|   2. Discovers available tools                                        |
|   3. Sends question + tool list to Claude                             |
|   4. Executes whatever tool Claude asks for                           |
|   5. Feeds the result back, repeats until Claude gives a final answer |
+----------------------------------+------------------------------------+
                                   |  MCP protocol (stdio, JSON-RPC under the hood)
                                   v
+-----------------------------------------------------------------------+
|                       server.py (MCP SERVER)                          |
|                                                                         |
|   Read-only tools:                Gated (destructive) tool:           |
|   - get_pod_status()              - restart_pod()                     |
|   - get_logs()                       -> returns PENDING, does not     |
|   - analyze_deployment()                execute                       |
|   - run_health_check()            - approve_action()                  |
|   - create_incident_report()         -> executes a pending action     |
+----------------------------------+------------------------------------+
                                   |  reads
                                   v
                        +----------------------+
                        |  fake_cluster.json    |   (stand-in for a real
                        |  (mock state)         |    Kubernetes API / kubectl)
                        +----------------------+

  Claude is the actual reasoning engine -- the MCP server has no
  intelligence of its own, it only executes exactly what it's told.

The one-sentence version: agent.py is a thin loop that lets Claude reason; server.py is a thin layer that lets Claude act; neither one is smart on its own — the intelligence is entirely in Claude's reasoning, and MCP is just the standardized wire format connecting the two.

---

2. The agent loop, step by step, for one real query

This is the exact Reason -> Act -> Observe pattern behind every coding agent, made fully visible in this project's terminal output:

 TURN 1
 -----------------------------------------------------------------
 User: "Why is payment-api failing?"
 Claude reasons: "I don't know yet -- I should check pod status first."
 Claude calls:    get_pod_status(namespace="production")
 -----------------------------------------------------------------
                              |
              agent.py executes the real tool call against
              server.py, gets back:
              payment-api -> CrashLoopBackOff, 14 restarts
                              |
                              v
 TURN 2
 -----------------------------------------------------------------
 Claude reasons: "CrashLoopBackOff -- I need the logs to see why
                  it's crashing."
 Claude calls:    get_logs(pod="payment-api")
 -----------------------------------------------------------------
                              |
              Result: "...OutOfMemoryError... exit code 137"
                              |
                              v
 TURN 3
 -----------------------------------------------------------------
 Claude reasons: "OOM -- let me check the configured memory limit
                  to confirm it's undersized."
 Claude calls:    analyze_deployment(deployment="payment-api")
 -----------------------------------------------------------------
                              |
              Result: memory_limit: "256Mi"
                              |
                              v
 TURN 4 (final)
 -----------------------------------------------------------------
 Claude has enough evidence. No more tool calls.
 Returns a final diagnosis: OOMKilled due to a 256Mi memory limit
 that's too low for the workload -- recommends raising it to
 512Mi, citing exit code 137 and the configured limit as evidence.
 -----------------------------------------------------------------

Notice Claude decided the order of investigation itself — nothing in the code hardcodes "always check logs after pod status." That sequencing is the actual reasoning the model is doing, turn by turn, driven by what each tool result reveals.

---

3. The approval gate — a state diagram

This is the security-relevant part of the design, and worth being able to draw from memory:

                    restart_pod(pod, reason) called
                                 |
                                 v
                    +------------------------+
                    |    PENDING_APPROVAL     |
                    |  (nothing has happened   |
                    |   to the real system     |
                    |   yet -- this is just a  |
                    |   recorded request)      |
                    +------------+-------------+
                                 |
              approve_action(id) called with the SAME id
                                 |
                  +--------------+---------------+
                  |                              |
            valid, unused id              invalid or already-used id
                  |                              |
                  v                              v
          +---------------+              +----------------+
          |    EXECUTED     |              |     ERROR       |
          | (action runs,   |              | (fails clearly,  |
          |  removed from   |              |  server does NOT |
          |  pending list -- |              |  crash, action   |
          |  can't be        |              |  is NOT          |
          |  approved again) |              |  re-executed)    |
          +---------------+              +----------------+

The key property, verified by test_scenarios.py: an action can transition from PENDING_APPROVAL to EXECUTED exactly once. A second approval attempt on the same id — or an approval attempt on a made-up id — fails gracefully rather than silently re-running a destructive action or crashing the server.

---

4. What's in this repo

File	Role
server.py	The MCP server — exposes 7 tools, reads fake_cluster.json
agent.py	The MCP client + agent loop — connects to the server, talks to Claude, executes tool calls
fake_cluster.json	Mock cluster state standing in for a real Kubernetes API
test_scenarios.py	8 scenarios validating server behavior directly, no API key needed
run_all_scenarios.py	Runs the real Claude-powered agent across multiple demo questions

5. Test scenarios — what each one proves

#	Scenario	What it proves
1	CrashLoopBackOff / OOM (payment-api)	The full diagnostic chain returns correct, consistent data across all four read tools
2	ImagePullBackOff (notification-worker)	The system handles a different failure mode, not just one hardcoded case
3	Healthy service (auth-service)	The system correctly reports "nothing's wrong" — a good agent shouldn't invent a problem just because it was asked to investigate
4	Querying a pod/deployment that doesn't exist	Errors are handled gracefully — a clear message, not a crash
5	Querying an empty/unknown namespace	Returns an empty result, not an error, for a namespace with no matches
6	Approval gate happy path	restart_pod → pending → approve_action → executed, confirming the gate works end to end
7	Approving an unknown action id	Fails gracefully instead of crashing the server
8	Re-approving an already-executed action	An action can't be executed twice — proves the gate isn't just cosmetic

Run all 8 free, with no API key needed:

python3 test_scenarios.py

Run the real agent

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
export ANTHROPIC_API_KEY=your_key_here

python3 agent.py "Why is payment-api failing?"
python3 agent.py "Is anything wrong with notification-worker?"
python3 agent.py "Is auth-service healthy right now?"

# or run several demo questions in one go:
python3 run_all_scenarios.py

Watch the terminal — it prints tool discovery, every tool call with its exact arguments, every result returned, and the final diagnosis. Nothing is hidden; this is the whole loop made visible.

6. Design decisions

Why MCP instead of just writing custom Python functions Claude could call directly? MCP standardizes tool discovery and execution the same way regardless of what's behind it — a Kubernetes cluster today, a GitHub API or a database tomorrow — without rewriting the agent loop each time. It's also exactly the mechanism real tools like Claude Code use, so building on it directly is the most honest way to demonstrate this skill.

Why mock the cluster instead of connecting to a real one? The goal was to demonstrate the agent and protocol mechanics correctly and testably, not to stand up Kubernetes infrastructure. Swapping fake_cluster.json reads for real Kubernetes Python client calls inside each tool function is a contained change — the MCP layer and the agent loop wouldn't need to change at all.

Why is restart_pod gated behind human approval, but the diagnostic tools aren't? This is a deliberate least-privilege design. The agent can gather unlimited information because reads are safe — but write/destructive actions against production are exactly the category the JD's "security and compliance" concerns are about. Rather than trusting the agent's judgment about when it's safe to restart something, the system makes it structurally impossible to do so without an explicit human step — the same "instruction vs. enforcement" principle behind hooks in Claude Code, applied to an MCP tool boundary instead.

Why does create_incident_report exist as a tool at all, if it's just formatting text? It forces the agent to commit to a structured, explicit root cause and evidence, in a fixed format, rather than describing a vague diagnosis in prose. That structure is what makes it easy to spot when the agent's reasoning is weak (e.g. it can't fill in a specific evidence field) versus genuinely well-grounded.

What's mocked vs. real

• Real: the MCP server/client protocol itself, tool discovery, tool execution, the Claude tool-calling agent loop, the approval-gate logic — all verified directly against the real mcp Python SDK
• Mocked: the underlying data source (fake_cluster.json instead of a live Kubernetes API or GitHub API)

Status

A working prototype. All 8 test scenarios pass against the real MCP server mechanics with no API key required. The live agent loop (agent.py, run_all_scenarios.py) requires an ANTHROPIC_API_KEY and has been run successfully end to end.