Cluster Health

Run each check below in order. After all checks, print a summary table: area | status (✅/⚠️/❌) | one-line note. Only show ⚠️ or ❌ rows in the summary unless everything is green.

Run independent checks in parallel where possible.

1. Nodes

kubectl get nodes -o wide

Flag: any node not Ready, any pressure condition, version mismatch across nodes.

2. Flux

flux get all -A

Flag: any resource where READY != True. Ignore SUSPENDED=True resources.

3. Pods

kubectl get pods -A --no-headers | grep -vE '\s(Running|Completed|Succeeded)\s'
kubectl get daemonsets -A --no-headers | awk '$2 != $4 {print}'

Flag: CrashLoopBackOff, Error, OOMKilled, stuck Pending. Daemonsets where DESIRED != READY.

4. Rook-Ceph

kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph.health}'
kubectl get pvc -A --no-headers | grep -v Bound

Flag: anything other than HEALTH_OK. Any unbound PVCs.

4b. Stuck CSI unpublish (silent stale RBD mappings)

for p in $(kubectl get pods -n rook-ceph -l app=rook-ceph.rbd.csi.ceph.com-nodeplugin -o name); do
  n=$(kubectl get $p -n rook-ceph -o jsonpath='{.spec.nodeName}')
  l=$(kubectl logs $p -c csi-rbdplugin -n rook-ceph --since=15m 2>/dev/null | grep 'directory not empty')
  echo "$n ENOTEMPTY=$(echo -n "$l" | grep -c .) last=$(echo "$l" | tail -1 | awk '{print $2}')"
done

Flag: any node with a non-zero count. The window is 15m and the loop retries every ~2 min, so a live loop always shows ≥3; last= is there to confirm recency (log timestamps are UTC — compare against date -u, not local time). Don't widen the window: with a 1h lookback this check keeps reporting a node as broken for an hour after it's actually fixed. Nothing else in this skill catches it — the volume keeps working for its current pod, so pods, PVCs and Ceph health all stay green while the RBD image is silently pinned to that node forever.

What it means: NodeUnpublishVolume is failing with remove /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount: directory not empty and retrying every ~2 min. A leftover directory sits in the pod's mount/ dir on local disk (app wrote at the volume root while unmounted). Unpublish never completes, so NodeUnstageVolume never runs, so the image stays krbd-mapped. The damage only surfaces later, when the pod reschedules to another node and hits FailedMount ... rbd image ... is still being used.

Fix — pull the pod UID and PV from the error line, confirm nothing is mounted there, then remove the leftover dir; kubelet finishes cleanup and GCs the pod dir within ~2 min:

kubectl exec <nodeplugin-pod> -c csi-rbdplugin -n rook-ceph -- grep -c '<pod-uid>' /proc/mounts   # MUST be 0
kubectl exec <nodeplugin-pod> -c csi-rbdplugin -n rook-ceph -- rmdir <mount-dir>/<leftover-dir>

If a pod is already stuck mounting elsewhere, also clear the stale mapping on the old node — see [[rwo-ceph-forcedelete-hazard]] for the umount/unmap sequence. Never ceph osd blocklist add.

5. miroir

kubectl get pods -n miroir-system --no-headers | grep -v Running
kubectl get miroirnodes
kubectl get miroirvolumes -A --no-headers | grep -v Ready

Flag: any pod not Running. Any MiroirNode missing its default pool, showing no CAPACITY, or with a blank DRBD version — that node's r-miroir partition is missing or its agent has not claimed it, and it silently stops being a placement candidate. Any MiroirVolume not Ready, or reporting fewer replicas than its StorageClass asks for (1/1 for miroir-local, 2/2 for miroir-replicated).

6. Kopiur Backups

kubectl get snapshotschedule.kopiur.home-operations.com -A -o json \
  | jq -r '.items[]
      | "\(.metadata.namespace)/\(.metadata.name) cron=\(.spec.schedule.cron) last=\((.status.lastSchedule.at // "never")[0:19]) next=\((.status.nextSchedule.at // "-")[0:19]) fails=\(.status.consecutiveFailures) suspended=\(.spec.suspend // false)"'
kubectl get snapshots.kopiur.home-operations.com -A -o json \
  | jq -r '[.items[] | select(.status.phase != "Succeeded")
            | "\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"] | .[]'
for k in snapshotpolicy snapshotschedule restore; do
  echo "$k: $(kubectl get $k.kopiur.home-operations.com -A --no-headers | wc -l)"
done
date -u +%Y-%m-%dT%H:%M:%SZ

Flag: any schedule that is SUSPENDED, has consecutiveFailures > 0, or whose last= is older than the interval its own cron= implies (read the cron off the object — do not assume a period). next= in the past by more than one interval means the scheduler has stopped firing. Also flag any Snapshot stuck in a non-Succeeded phase across two runs — a single Running/Deleting is just a run in flight.

The three counts should match each other (one policy, schedule and restore per stateful app); the absolute number tracks how many apps use components/persistence, so compare them to each other rather than to a fixed number.

Snapshots are retained on a GFS schedule, so a large total is expected — see [[kopiur-retention-design]] before diagnosing accumulation.

Transient churn is normal. Each run creates a VolumeSnapshot, an ephemeral PVC and a mover pod, then tears them down, which produces VolumeFailedDelete (PV deleted before its VolumeAttachment detaches), FailedScheduling (mover waiting on the ephemeral PVC) and MissingDependency (waiting for the VolumeSnapshot to become readyToUse) in check 17. Expect roughly one of each per schedule per run. Only treat them as a fault if PVs are stuck Released/Failed or a schedule's consecutiveFailures is climbing.