<div align="center">

k8s-dr-mcp

!!! This is MVP version only !!! Running on production on your own risk and responsibility! !!!

Kubernetes disaster recovery for AI agents — over SSH, for when the API server is already dead.

</div>

---

Every Kubernetes MCP server on the market talks to the Kubernetes API. That is exactly the thing that is gone when you actually need help at 3am.

k8s-dr-mcp gives an AI agent a way in through the back door: SSH to the nodes themselves. systemctl, etcdctl, kubeadm, crictl, journalctl, openssl — the tools you'd reach for by hand, wrapped in 63 MCP tools with a three-gate safety model so the agent can't wreck a cluster that's already having a bad day.

┌───────────────────────────── CLUSTER HEALTHY ──────────────────────────────┐
│                                                                            │
│   LLM ──► kubectl MCP ──► kube-apiserver ──► etcd               ✓ works    │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

┌────────────────────────── CONTROL PLANE IS DOWN ───────────────────────────┐
│                                                                            │
│   LLM ──► kubectl MCP ──► kube-apiserver  ✗                     ✗ dead     │
│                              nothing left to talk to                       │
│                                                                            │
│   LLM ──► k8s-dr-mcp  ──► SSH ──► the node itself               ✓ works    │
│                                      systemctl · etcdctl ·                 │
│                                      kubeadm · crictl · journalctl         │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

Run it alongside a normal kubectl-based MCP server. The agent picks the right one on its own: API for healthy clusters, SSH for broken ones.

---

Contents

Quick start · Disaster playbook · Tool reference	Safety model · Configuration · Kubespray
MCP client setup · Docker · Context budget	How it works · Development · Security

---

Highlights

• 63 tools, 8 groups — diagnostics, etcd, certificates, control plane, kubelet, runtime, network, and orchestrated recovery workflows
• Works when nothing else does — no API server, no kubeconfig, no CNI required
• Read-only by default — 43 of the 63 tools can't change anything; the other 20 need two explicit unlocks
• Three-gate safety — mode gate → confirmation gate → automated pre-flight checks, then an auto-backup
• Parallel by design — AsyncSSH connection pool fans out across every control-plane node at once
• Bastion-native — tunnels into private node networks through a jump host
• Audit trail — one structured JSON line per operation, to stderr or a file
• Kubespray-aware — host-mode etcd, .pem cert layouts, /etc/etcd.env, script-based cert renewal

---

Quick start

1. Install

pip install -e .

2. Describe your cluster

Copy the example that matches your topology:

cp config.example.yaml            cluster.yaml   # generic kubeadm cluster
cp cluster.stacked.example.yaml   cluster.yaml   # Kubespray: stacked etcd + Calico
cp cluster.host-etcd.example.yaml cluster.yaml   # Kubespray: host etcd + Cilium

Minimum viable config — node names, addresses, and how to SSH in:

cluster:
  name: my-cluster
  etcd_deployment: stacked        # stacked | external | host
  container_runtime: containerd   # containerd | crio
  cni_plugin: calico

  ssh_defaults:
    user: root
    key_path: ~/.ssh/id_ed25519
    sudo: true

  # Optional — only if the nodes live on a private network
  # bastion:
  #   host: bastion.example.com
  #   user: ubuntu
  #   key_path: ~/.ssh/bastion_key

  control_plane:
    - { name: cp-1, host: 10.0.0.11, etcd_member_name: cp-1 }
    - { name: cp-2, host: 10.0.0.12, etcd_member_name: cp-2 }
    - { name: cp-3, host: 10.0.0.13, etcd_member_name: cp-3 }

  workers:
    - { name: worker-1, host: 10.0.0.31 }
    - { name: worker-2, host: 10.0.0.32 }

cluster.yaml, .env, .mcp.json and everything under ssh/ are gitignored — they hold your real hostnames, addresses and keys. Only the *.example.* files are meant to be committed.

3. Wire it into your agent

// .mcp.json in your project root
{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "k8s-dr-mcp",
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "./cluster.yaml"
      }
    }
  }
}

Other clients (Claude Desktop, Cursor, Windsurf, VS Code, Docker) → MCP client setup.

4. Ask

You     The cluster is down — kubectl just hangs.

Claude  → dr_cluster_overview()
        → dr_check_etcd_health()
        → dr_check_etcd_alarms()

        etcd on all three nodes has a NOSPACE alarm — the database hit the
        2 GiB default quota. Fix is defrag + disarm. That's destructive and
        the server is read-only. Want me to walk you through, or restart in
        readwrite mode?

You     Restart in readwrite. Go ahead.

Claude  → dr_etcd_defrag(node="cp-1", confirm=True)        ↳ backup first
        → dr_etcd_alarm_disarm(node="cp-1", confirm=True)
        → …cp-2, cp-3…
        → dr_check_etcd_health()

        All three members healthy, quorum restored, API server answering.

---

Disaster playbook

What to say, and what the agent will reach for.

Symptom	Say something like	Tools it uses
x509: certificate has expired	"Certs look expired — check and renew across the control plane."	dr_certs_check_expiration → dr_certs_backup → dr_certs_renew_all ⚠ → dr_cp_restart_all ⚠
etcdserver: mvcc: database space exceeded	"etcd is out of space."	dr_check_etcd_alarms → dr_etcd_defrag ⚠ → dr_etcd_alarm_disarm ⚠
2 of 3 control-plane nodes gone	"We lost quorum. One survivor left."	dr_etcd_member_list → dr_recover_quorum_loss ⚠ (force-new-cluster on the survivor, then re-add members)
API server won't come back after a manifest edit	"apiserver won't start — compare cp-1 against the others."	dr_cp_validate_manifests → dr_cp_diff_manifests → dr_cp_get_manifest → dr_cp_restart_component ⚠
Node stuck NotReady	"worker-3 is NotReady, find out why."	dr_kubelet_status → dr_kubelet_logs → dr_runtime_status → dr_net_cni_config
Pods stuck ContainerCreating	"CNI looks broken on this node."	dr_net_cni_config → dr_net_cni_binaries → dr_net_cilium_status → dr_net_dns
Restored from a snapshot, unsure what's next	"Give me a recovery plan before we touch anything."	dr_generate_recovery_plan (read-only — diagnoses every node and writes out the steps)
Total loss, need it back now	"Diagnose and recover the whole cluster."	dr_recover_full_cluster ⚠ (diagnose → backup → repair → verify)

⚠ = requires K8S_DR_MODE=readwrite and confirm=True.

---

Tool reference

63 tools across 8 groups. 43 are read-only. 20 are destructive (⚠) and pass all three safety gates before running. Every tool is prefixed dr_.

Group	Tools	Destructive	Purpose
diagnostics	10	0	Health, logs, ports, disk — always loaded
etcd	10	7	Snapshots, restore, members, defrag, alarms
certificates	7	2	Expiry, inspection, renewal, PKI backup
control-plane	7	2	Static pod manifests, component restarts
kubelet	8	3	Kubelet state, config, systemd services
runtime	6	1	containerd / CRI-O, containers, images
network	9	0	CNI, interfaces, firewall, DNS, Cilium, Hubble
recovery	6	5	Multi-step orchestrated recovery workflows

<details id="diagnostics"> <summary><b>diagnostics</b> — 10 tools, all read-only</summary>

Always registered, even if you leave it out of K8S_DR_TOOL_GROUPS.

Tool	What it does
dr_cluster_overview	Unified health snapshot of every control-plane node
dr_check_node	Detailed system health for one node
dr_check_etcd_health	etcd cluster health, member status, quorum
dr_check_etcd_alarms	Active etcd alarms (NOSPACE, CORRUPT)
dr_check_api_health	kube-apiserver health on a node
dr_check_ports	Whether critical Kubernetes ports are listening
dr_check_services	systemd unit status
dr_get_logs	journald logs for a specific unit
dr_check_static_pods	List and validate /etc/kubernetes/manifests/
dr_check_disk_usage	Disk usage on the paths that matter

</details>

<details id="etcd"> <summary><b>etcd</b> — 10 tools, 7 destructive</summary>

Tool	What it does
dr_etcd_snapshot	Take a snapshot and verify its integrity
dr_etcd_snapshot_status	Inspect an existing snapshot file
dr_etcd_member_list	List cluster members
dr_etcd_restore ⚠	Restore from a snapshot via etcdutl
dr_etcd_force_new_cluster ⚠	Bootstrap a single-member cluster from a survivor — total quorum loss
dr_etcd_member_add ⚠	Add a member
dr_etcd_member_remove ⚠	Remove a member
dr_etcd_alarm_disarm ⚠	Disarm active alarms
dr_etcd_defrag ⚠	Defragment storage to reclaim space
dr_etcd_compact ⚠	Compact history to a revision

</details>

<details id="certificates"> <summary><b>certificates</b> — 7 tools, 2 destructive</summary>

Tool	What it does
dr_certs_check_expiration	Expiry dates for all Kubernetes certificates
dr_certs_inspect	openssl inspection of one certificate
dr_certs_inspect_all	Inspect everything under the PKI directories
dr_certs_check_connectivity	Verify the TLS chain to a service
dr_certs_backup	Tarball the whole PKI directory
dr_certs_renew_all ⚠	kubeadm certs renew all
dr_certs_renew_single ⚠	Renew one certificate

</details>

<details id="control-plane"> <summary><b>control-plane</b> — 7 tools, 2 destructive</summary>

Tool	What it does
dr_cp_get_manifest	Read a static pod manifest
dr_cp_validate_manifests	YAML-validate every manifest
dr_cp_backup_manifests	Back up the manifests directory
dr_cp_check_images	Are the referenced images present locally?
dr_cp_diff_manifests	Diff manifests between two nodes — finds the odd one out
dr_cp_restart_component ⚠	Restart one component by cycling its manifest
dr_cp_restart_all ⚠	Restart all components in dependency order

</details>

<details id="kubelet"> <summary><b>kubelet</b> — 8 tools, 3 destructive</summary>

Tool	What it does
dr_kubelet_status	systemd status for kubelet
dr_kubelet_config	Read the kubelet config file
dr_kubelet_flags	Read startup flags (kubeadm-flags.env)
dr_kubelet_kubeconfig	Read the kubelet kubeconfig, secrets masked
dr_kubelet_logs	kubelet journal logs
dr_kubelet_restart ⚠	Restart kubelet
dr_service_start ⚠	Start any systemd unit
dr_service_stop ⚠	Stop any systemd unit

</details>

<details id="runtime"> <summary><b>runtime</b> — 6 tools, 1 destructive</summary>

Tool	What it does
dr_runtime_status	containerd / CRI-O service status
dr_runtime_config	Read the runtime config file
dr_runtime_containers	List containers via crictl
dr_runtime_container_logs	Logs from one container
dr_runtime_images	Cached images on the node
dr_runtime_restart ⚠	Restart the runtime

</details>

<details id="network"> <summary><b>network</b> — 9 tools, all read-only</summary>

Tool	What it does
dr_net_cni_config	CNI configuration files
dr_net_cni_binaries	Installed CNI plugin binaries
dr_net_interfaces	Interfaces and routing table
dr_net_connectivity	TCP reachability between two nodes
dr_net_firewall	Firewall rules
dr_net_dns	DNS config and resolution
dr_net_cilium_status	Cilium agent health
dr_net_cilium_connectivity	Quick Cilium connectivity test
dr_net_hubble_status	Hubble relay health and recent flows

</details>

<details id="recovery"> <summary><b>recovery</b> — 6 orchestrated workflows, 5 destructive</summary>

Each of these chains many low-level tools into one guarded, multi-step procedure.

Tool	What it does
dr_generate_recovery_plan	Diagnose every node and produce a plan — read-only, start here
dr_recover_expired_certs ⚠	Back up PKI → renew on all CP nodes in parallel → restart → verify
dr_recover_etcd_single ⚠	Restore etcd from a snapshot on one node
dr_recover_etcd_cluster ⚠	Restore etcd from a snapshot across all CP nodes
dr_recover_quorum_loss ⚠	Rebuild the cluster from a surviving member
dr_recover_full_cluster ⚠	Auto-diagnose, then run whichever recovery fits

</details>

---

Safety model

The server is read-only until you say otherwise, and even then every destructive tool has to clear three gates and take a backup before it touches anything.

Claude calls: dr_etcd_defrag(node="cp-1", confirm=True)
       │
       ▼
┌────────────────────┐
│  GATE 1 · MODE     │  K8S_DR_MODE == "readwrite"?
│                    │
│  readonly  ───────►│──► BLOCKED  "Set K8S_DR_MODE=readwrite"
│  readwrite ───────►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  GATE 2 · CONFIRM  │  confirm == True?
│                    │
│  confirm=False ───►│──► PREVIEW  what it would do + pre-flight results,
│                    │             nothing executed
│  confirm=True  ───►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  GATE 3 · PREFLIGHT│  ✓ SSH reachable      ✓ required binary present
│                    │  ✓ service in the     ✓ target dir writable
│                    │    expected state     ✓ enough disk space
│  any check failed ►│──► BLOCKED  with the specific failure
│  all passed    ───►│──► pass
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  AUTO-BACKUP       │  etcd data dir · PKI · static pod manifests
└─────────┬──────────┘  (path returned in the result, and logged)
          ▼
┌────────────────────┐
│  EXECUTE           │  run it over SSH
└─────────┬──────────┘
          ▼
┌────────────────────┐
│  AUDIT             │  one JSON line → stderr and/or K8S_DR_AUDIT_LOG
└────────────────────┘

Gate 2 is the useful one in practice. Calling a destructive tool with confirm=False is a dry run: you get a description of the operation plus the pre-flight results, and nothing happens. Agents can safely explore what a fix would involve before proposing it.

Audit entries look like this:

{"timestamp":"2026-07-28T09:14:22.881Z","tool":"dr_etcd_defrag","node":"cp-1",
 "mode":"readwrite","result":"success","confirm":true,
 "backup_path":"/var/lib/etcd-backup/etcd-20260728-091422.tar.gz","duration_ms":4130}

---

Configuration

Cluster file

Full schema, with defaults shown. Only name and control_plane are required.

cluster:
  name: my-cluster
  etcd_deployment: stacked        # stacked | external | host
  container_runtime: containerd   # containerd | crio
  cni_plugin: calico
  cert_management: kubeadm        # kubeadm | script

  ssh_defaults:
    user: root
    key_path: ~/.ssh/id_ed25519
    port: 22
    sudo: true
    connect_timeout: 30
    command_timeout: 120

  bastion:                        # omit entirely for direct access
    host: ""
    user: ""
    key_path: ""
    port: 22

  control_plane:                  # required, at least one
    - name: cp-1
      host: 10.0.0.11
      ip: 10.0.0.11
      etcd_member_name: cp-1
      ssh_user: ubuntu            # optional per-node override
      ssh_key_path: ~/.ssh/cp1    # optional per-node override

  etcd_nodes: []                  # only when etcd_deployment: external
  workers: []                     # optional, for kubelet/runtime/network tools

  paths:                          # override only what differs
    etcd_data_dir:      /var/lib/etcd
    etcd_backup_dir:    /var/lib/etcd-backup
    k8s_manifests:      /etc/kubernetes/manifests
    k8s_pki:            /etc/kubernetes/pki
    k8s_pki_etcd:       /etc/kubernetes/pki/etcd
    kubelet_config:     /var/lib/kubelet/config.yaml
    kubelet_kubeconfig: /etc/kubernetes/kubelet.conf
    kubelet_flags:      /var/lib/kubelet/kubeadm-flags.env
    kubelet_service:    /etc/systemd/system/kubelet.service
    kubelet_dropin:     /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
    containerd_config:  /etc/containerd/config.toml
    crio_config:        /etc/crio/crio.conf
    cni_bin:            /opt/cni/bin
    cni_config:         /etc/cni/net.d
    etcd_cert_dir:      /etc/kubernetes/pki/etcd
    etcd_ca_cert:       ca.crt
    etcd_client_cert:   server.crt     # supports a {node} placeholder
    etcd_client_key:    server.key     # supports a {node} placeholder
    etcd_env_file:      ""             # Kubespray: /etc/etcd.env

Environment variables

All prefixed K8S_DR_. Also readable from a .env file in the working directory.

Variable	Default	Description
K8S_DR_MODE	readonly	readonly or readwrite
K8S_DR_CLUSTER_CONFIG	./cluster.yaml	Path to the cluster file
K8S_DR_TOOL_GROUPS	all 8 groups	Comma-separated groups to load
K8S_DR_LOG_LEVEL	INFO	DEBUG · INFO · WARNING · ERROR
K8S_DR_AUDIT_LOG	(stderr only)	Also append audit JSON to this file
K8S_DR_SSH_USER	root	Default SSH user
K8S_DR_SSH_KEY_PATH	~/.ssh/id_ed25519	Default private key
K8S_DR_SSH_PORT	22	Default SSH port
K8S_DR_SSH_TIMEOUT	30	Connect timeout, seconds
K8S_DR_SSH_KNOWN_HOSTS	~/.ssh/known_hosts	none disables host key verification
K8S_DR_BASTION_HOST	(none)	Jump host address
K8S_DR_BASTION_USER	(none)	Jump host user
K8S_DR_BASTION_KEY_PATH	(none)	Jump host key
K8S_DR_BASTION_PORT	22	Jump host port

Values in cluster.yaml win over the environment for anything both can express — the env vars are there for container deployments where mounting one file is easier than templating two.

---

Kubespray clusters

Kubespray doesn't lay things out the way kubeadm does: etcd runs as a systemd unit rather than a static pod, certificates are .pem files in /etc/ssl/etcd/ssl, and renewal goes through a script instead of kubeadm certs renew. Tell the server about it and every tool adapts:

cluster:
  etcd_deployment: host           # systemctl, not a static pod
  cert_management: script         # not kubeadm

  paths:
    k8s_pki:           /etc/kubernetes/ssl
    k8s_pki_etcd:      /etc/ssl/etcd/ssl
    etcd_cert_dir:     /etc/ssl/etcd/ssl
    etcd_ca_cert:      ca.pem
    etcd_client_cert:  "admin-{node}.pem"     # {node} → the node's name
    etcd_client_key:   "admin-{node}-key.pem"
    etcd_env_file:     /etc/etcd.env
    kubelet_config:    /etc/kubernetes/kubelet-config.yaml
    kubelet_flags:     /etc/kubernetes/kubelet.env

Two ready-made starting points ship with the repo: cluster.stacked.example.yaml (stacked etcd + Calico) and cluster.host-etcd.example.yaml (host etcd + Cilium).

---

MCP client setup

<details> <summary><b>Claude Code</b></summary>

.mcp.json in the project root, or ~/.claude.json for every project:

{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "k8s-dr-mcp",
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

Without installing the entry point, use "command": "python", "args": ["-m", "k8s_dr_mcp"].

Verify with claude mcp list. A copyable starting point lives in .mcp.json.example, including a second server for the healthy-cluster case.

</details>

<details> <summary><b>Claude Desktop</b></summary>

macOS ~/Library/Application Support/Claude/claude_desktop_config.json, Windows %APPDATA%\Claude\claude_desktop_config.json:

{
  "mcpServers": {
    "k8s-dr": {
      "command": "python",
      "args": ["-m", "k8s_dr_mcp"],
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

</details>

<details> <summary><b>Cursor · Windsurf · VS Code</b></summary>

Cursor — .cursor/mcp.json; Windsurf — ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "k8s-dr": {
      "command": "python",
      "args": ["-m", "k8s_dr_mcp"],
      "env": {
        "K8S_DR_MODE": "readonly",
        "K8S_DR_CLUSTER_CONFIG": "/absolute/path/to/cluster.yaml"
      }
    }
  }
}

VS Code — .vscode/mcp.json, same shape but the top-level key is "servers" and each entry needs "type": "stdio".

</details>

<details> <summary><b>Run both DR and kubectl MCP servers together</b> — recommended</summary>

The agent chooses based on what's actually broken: kubectl while the API answers, SSH once it doesn't.

        ┌──────────┐
        │  Claude  │  sees every tool from both servers
        └────┬─────┘
             │
       ┌─────┴───────────────────┐
       ▼                         ▼
┌──────────────┐        ┌──────────────────┐
│  k8s-dr-mcp  │        │  kubectl MCP     │
│  63 DR tools │        │  server          │
│              │        │                  │
│  SSH → nodes │        │  K8s API         │
│              │        │                  │
│  when the    │        │  when the        │
│  API is DOWN │        │  cluster is OK   │
└──────────────┘        └──────────────────┘

See .mcp.json.example for a working two-server config, including the SSH port-forward wrapper for reaching a private API endpoint through a bastion.

</details>

Enabling write operations

"env": { "K8S_DR_MODE": "readwrite" }

That only opens gate 1. Every destructive tool still needs confirm=True and still has to pass pre-flight checks. Running read-only day to day and restarting in readwrite when you've decided on a fix is a reasonable habit.

---

Docker

docker build -t k8s-dr-mcp .

docker run --rm -i \
  -v ./cluster.yaml:/config/cluster.yaml:ro \
  -v ./ssh:/config/ssh:ro \
  -e K8S_DR_MODE=readonly \
  k8s-dr-mcp

Note: the image sets K8S_DR_MODE=readwrite in its own ENV so it works out of the box for recovery. Pass -e K8S_DR_MODE=readonly (as above) — or edit the Dockerfile — if you'd rather the container default to safe.

The image runs as a non-root user, expects the cluster file at /config/cluster.yaml, and reads SSH keys from wherever your config points — /config/ssh/<key> if you mount the repo's ssh/ directory. From an MCP client:

{
  "mcpServers": {
    "k8s-dr": {
      "type": "stdio",
      "command": "docker",
      "args": [
        "run", "--rm", "-i",
        "-v", "./cluster.yaml:/config/cluster.yaml:ro",
        "-v", "./ssh:/config/ssh:ro",
        "-e", "K8S_DR_MODE=readonly",
        "k8s-dr-mcp:latest"
      ]
    }
  }
}

---

Context budget

63 tool definitions is a lot of context to hand an LLM before it has done anything. Load only the groups the situation calls for:

K8S_DR_TOOL_GROUPS=diagnostics,etcd,certificates

  all groups · 63 tools              selected · 27 tools
┌───────────────────────┐          ┌───────────────────────┐
│ diagnostics      10   │          │ diagnostics      10   │ ✓ always on
│ etcd             10   │          │ etcd             10   │ ✓
│ certificates      7   │   ────►  │ certificates      7   │ ✓
│ control-plane     7   │          │                       │
│ kubelet           8   │          │  ~57% fewer tool      │
│ runtime           6   │          │  definitions in the   │
│ network           9   │          │  context window       │
│ recovery          6   │          └───────────────────────┘
└───────────────────────┘

diagnostics is always registered whether you list it or not — you can't triage without it.

---

How it works

                        ┌──────────────────────────┐
                        │   MCP client             │
                        │   Claude Code · Desktop  │
                        │   Cursor · Windsurf      │
                        └────────────┬─────────────┘
                                     │ JSON-RPC over stdio
                        ┌────────────▼─────────────┐
                        │   k8s-dr-mcp  (FastMCP)  │
                        │                          │
                        │   tool router            │ ← K8S_DR_TOOL_GROUPS
                        │   safety gates           │ ← mode · confirm · preflight
                        │   auto-backup            │
                        │   audit logger           │ → JSON, stderr or file
                        └────────────┬─────────────┘
                                     │ AsyncSSH connection pool
                                     │ cached · auto-reconnect · parallel
                        ┌────────────▼─────────────┐
                        │   bastion / jump host    │  optional
                        └────────────┬─────────────┘
                                     │ tunneled SSH
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
     ┌────────▼────────┐    ┌────────▼────────┐    ┌────────▼────────┐
     │  cp-1           │    │  cp-2           │    │  cp-3           │
     │  etcd           │    │  etcd           │    │  etcd           │
     │  kube-apiserver │    │  kube-apiserver │    │  kube-apiserver │
     │  scheduler      │    │  scheduler      │    │  scheduler      │
     │  controller-mgr │    │  controller-mgr │    │  controller-mgr │
     │  kubelet        │    │  kubelet        │    │  kubelet        │
     └─────────────────┘    └─────────────────┘    └─────────────────┘

       systemctl · etcdctl · kubeadm · crictl · journalctl · openssl · ss · ip

The SSH pool keeps one connection per node, opens the bastion tunnel once and reuses it, reconnects on drop, and fans out across all control-plane nodes concurrently — so dr_cluster_overview on a three-node cluster costs about as long as querying one node.

Everything the server does is an ordinary shell command over SSH. Nothing is installed on the nodes, and nothing depends on the cluster being able to schedule a pod.

Layout

src/k8s_dr_mcp/
├── server.py          FastMCP instance, SSH pool lifespan
├── config.py          env settings + cluster YAML loading
├── models/            pydantic models — cluster topology, tool results
├── ssh/               AsyncSSH connection pool, command executor
├── safety/            gates · preflight checks · auto-backup · audit log
└── tools/             the 8 tool groups

---

Development

pip install -e ".[dev]"

pytest tests/ -v            # test suite
ruff check src/ tests/      # lint
mypy src/                   # type check (advisory — see below)

Adding a tool: write it in the right tools/*.py module, register it in that module's register(), and if it changes anything on a node, route it through run_safety_gates() with a confirm: bool parameter and a pre-flight check.

What CI enforces

.github/workflows/ci.yml runs on every push and pull request:

Job	Gate
lint	ruff check — blocking
test	pytest on Python 3.10 / 3.11 / 3.12 / 3.13 — blocking
build	wheel + sdist, twine check — blocking
docker	image builds and still defaults to K8S_DR_MODE=readonly — blocking
secrets	no private-key material, no tracked .env / cluster.yaml / *.pem, nothing in ssh/ but its README — blocking
types	mypy src/ — advisory

Type-checking is advisory on purpose. FastMCP tools receive their context as ctx: Context = None — the framework injects it at call time, but mypy reads the None default as an implicit Optional and flags every such signature. Annotating Context | None just relocates the complaint to each ctx. use site, so the honest options are a large signature rework or an advisory job. If you make the type errors go away without weakening the checks, that's a very welcome PR.

Tech

• Python 3.10+, asyncio throughout
• FastMCP — MCP server framework
• AsyncSSH — non-blocking SSH; parallel node fan-out without a thread pool
• Pydantic — config validation and structured tool results

---

Security

This server executes privileged shell commands on your Kubernetes nodes, usually with sudo. Treat it as the sensitive thing it is:

• Keep it read-only unless you are actively recovering something.
• Use a dedicated SSH key for the server rather than your personal one, and scope it as tightly as your setup allows.
• Never commit keys or real inventories. cluster.yaml, .env, .mcp.json and ssh/* are gitignored for exactly this reason — if one slips through, rotate the key; deleting the file isn't enough.
• Turn on host key verification in production. K8S_DR_SSH_KNOWN_HOSTS=none disables it, which is convenient for a lab and a bad idea anywhere else.
• Point K8S_DR_AUDIT_LOG at a real file when running against production, so there's a record of what ran and when.

Found a vulnerability? Please report it privately rather than opening a public issue.

Status

Alpha. The tooling is complete and covered by tests, but it has been exercised against a limited set of cluster topologies (kubeadm and Kubespray, containerd, Calico and Cilium). Try it on a throwaway cluster before you trust it with a real outage — ideally rehearse your recovery with it, so the first time you use it in anger isn't the first time you use it.

Reports about other topologies — CRI-O, external etcd, flannel, k3s — are especially welcome.

Contributing

Issues and pull requests are welcome. Useful contributions, roughly in order of value:

1. Support for cluster layouts that don't work yet
2. New recovery workflows for failure modes you've actually hit
3. Pre-flight checks that would have caught something the hard way

Please run ruff and the tests before opening a PR, and keep destructive tools behind the three gates.

License

MIT.

---

<div align="center"> _{<b>Rehearse your disaster recovery before you need it.</b><br> The best time to find out this works on your cluster is a Tuesday afternoon, not a Saturday night.} </div>