Must Gather Analyzer

Must-Gather Analyzer Skill

Comprehensive analysis of OpenShift must-gather diagnostic data with helper scripts that parse YAML and display output in oc-like format.

Overview

This skill provides analysis for:

• ClusterVersion: Current version, update status, and capabilities
• Cluster Operators: Status, degradation, and availability
• Pods: Health, restarts, crashes, and failures across namespaces
• Nodes: Conditions, capacity, and readiness
• Network: OVN/SDN diagnostics and connectivity
• Events: Warning and error events across namespaces
• etcd: Cluster health, member status, and quorum
• Storage: PersistentVolumes and PersistentVolumeClaims status

Must-Gather Directory Structure

Important: Must-gather data is contained in a subdirectory with a long hash name:

must-gather/
└── registry-ci-openshift-org-origin-...-sha256-<hash>/
    ├── cluster-scoped-resources/
    │   ├── config.openshift.io/clusteroperators/
    │   └── core/nodes/
    ├── namespaces/
    │   └── <namespace>/
    │       └── pods/
    │           └── <pod-name>/
    │               └── <pod-name>.yaml
    └── network_logs/

The analysis scripts expect the path to the subdirectory (the one with the hash), not the root must-gather folder.

Instructions

1. Get Must-Gather Path

Ask the user for the must-gather directory path if not already provided.

• If they provide the root directory, look for the subdirectory with the hash name
• The correct path contains cluster-scoped-resources/ and namespaces/ directories

2. Choose Analysis Type

Based on user's request, run the appropriate helper script:

ClusterVersion Analysis

./scripts/analyze_clusterversion.py <must-gather-path>

Shows cluster version information similar to oc get clusterversion:

• Current version and update status
• Progressing state
• Available updates
• Version conditions
• Enabled capabilities
• Update history

Cluster Operators Analysis

./scripts/analyze_clusteroperators.py <must-gather-path>

Shows cluster operator status similar to oc get clusteroperators:

• Available, Progressing, Degraded conditions
• Version information
• Time since condition change
• Detailed messages for operators with issues

Pods Analysis

# All namespaces
./scripts/analyze_pods.py <must-gather-path>

# Specific namespace
./scripts/analyze_pods.py <must-gather-path> --namespace <namespace>

# Show only problematic pods
./scripts/analyze_pods.py <must-gather-path> --problems-only

Shows pod status similar to oc get pods -A:

• Ready/Total containers
• Status (Running, Pending, CrashLoopBackOff, etc.)
• Restart counts
• Age
• Categorized issues (crashlooping, pending, failed)

Nodes Analysis

./scripts/analyze_nodes.py <must-gather-path>

# Show only nodes with issues
./scripts/analyze_nodes.py <must-gather-path> --problems-only

Shows node status similar to oc get nodes:

• Ready status
• Roles (master, worker)
• Age
• Kubernetes version
• Node conditions (DiskPressure, MemoryPressure, etc.)
• Capacity and allocatable resources

Network Analysis

./scripts/analyze_network.py <must-gather-path>

Shows network health:

• Network type (OVN-Kubernetes, OpenShift SDN)
• Network operator status
• OVN pod health
• PodNetworkConnectivityCheck results
• Network-related issues

Events Analysis

# Recent events (last 100)
./scripts/analyze_events.py <must-gather-path>

# Warning events only
./scripts/analyze_events.py <must-gather-path> --type Warning

# Events in specific namespace
./scripts/analyze_events.py <must-gather-path> --namespace openshift-etcd

# Show last 50 events
./scripts/analyze_events.py <must-gather-path> --count 50

Shows cluster events:

• Event type (Warning, Normal)
• Last seen timestamp
• Reason and message
• Affected object
• Event count

etcd Analysis

./scripts/analyze_etcd.py <must-gather-path>

Shows etcd cluster health:

• Member health status
• Member list with IDs and URLs
• Endpoint status (leader, version, DB size)
• Quorum status
• Cluster summary

Storage Analysis

# All PVs and PVCs
./scripts/analyze_pvs.py <must-gather-path>

# PVCs in specific namespace
./scripts/analyze_pvs.py <must-gather-path> --namespace openshift-monitoring

Shows storage resources:

• PersistentVolumes (capacity, status, claims)
• PersistentVolumeClaims (binding, capacity)
• Storage classes
• Pending/unbound volumes

Monitoring Analysis

# All alerts.
./scripts/analyze_prometheus.py <must-gather-path>

# Alerts in specific namespace
./scripts/analyze_prometheus.py <must-gather-path> --namespace openshift-monitoring

Shows monitoring information:

• Alerts (state, namespace, name, active since, labels)
• Total of pending/firing alerts

3. Interpret and Report

After running the scripts:

1. Review the summary statistics
2. Focus on items flagged with issues
3. Provide actionable insights and next steps
4. Suggest log analysis for specific components if needed
5. Cross-reference issues (e.g., degraded operator → failing pods → node issues)

Output Format

All scripts provide:

• Summary Section: High-level statistics with emoji indicators
• Table View: oc-like formatted output
• Issues Section: Detailed breakdown of problems

Example summary format:

================================================================================
SUMMARY: 25/28 operators healthy
  ⚠️  3 operators with issues
  🔄 1 progressing
  ❌ 2 degraded
================================================================================