Resource Analysis

Resource Analysis Skill

This skill provides detailed guidance for analyzing system resource usage from sosreport archives, including memory, CPU, disk space, and process information.

When to Use This Skill

Use this skill when:

• Analyzing the /sosreport:analyze command's resource analysis phase
• Investigating performance issues or resource bottlenecks
• Identifying resource exhaustion problems
• Correlating resource usage with system failures

Prerequisites

• Sosreport archive must be extracted to a working directory
• Path to the sosreport root directory must be known
• Understanding of Linux resource management

Key Resource Data Locations in Sosreport

1. Memory Information:

   • sos_commands/memory/free - Memory usage snapshot
   • proc/meminfo - Detailed memory statistics
   • sos_commands/memory/swapon_-s - Swap usage
   • proc/buddyinfo - Memory fragmentation

2. CPU Information:

   • sos_commands/processor/lscpu - CPU architecture and features
   • proc/cpuinfo - Detailed CPU information
   • sos_commands/processor/turbostat - CPU frequency and power states (if available)
   • uptime - Load averages

3. Disk Information:

   • sos_commands/filesys/df_-al - Filesystem usage
   • sos_commands/block/lsblk - Block device information
   • sos_commands/filesys/mount - Mounted filesystems
   • proc/diskstats - Disk I/O statistics

4. Process Information:

   • sos_commands/process/ps_auxwww - Process list with details
   • sos_commands/process/top - Process snapshot (if available)
   • proc/[pid]/ - Per-process information

Implementation Steps

Step 1: Analyze Memory Usage

1. Parse free command output:

   # Check if free output exists
   if [ -f sos_commands/memory/free ]; then
     cat sos_commands/memory/free
   fi

2. Extract memory metrics:

   # Parse /proc/meminfo for detailed stats
   if [ -f proc/meminfo ]; then
     grep -E "^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree|Dirty|Slab):" proc/meminfo
   fi

3. Calculate memory usage percentage:

   • Total memory = MemTotal
   • Used memory = MemTotal - MemAvailable
   • Usage percentage = (Used / Total) * 100
   • Parse from free output or calculate from meminfo

4. Check for memory pressure indicators:

   # Look for OOM events in logs
   grep -i "out of memory\|oom killer" sos_commands/logs/journalctl_--no-pager 2>/dev/null
   
   # Check swap usage
   if [ -f sos_commands/memory/swapon_-s ]; then
     cat sos_commands/memory/swapon_-s
   fi

5. Identify memory issues:

   • Memory usage > 90% → Critical
   • Memory usage > 80% → Warning
   • Heavy swap usage (>50% swap used) → Performance issue
   • OOM killer events → Critical memory exhaustion

Step 2: Analyze CPU Usage

1. Extract CPU information:

   # Get CPU count and model
   if [ -f sos_commands/processor/lscpu ]; then
     grep -E "^(CPU\(s\)|Model name|Thread|Core|Socket|CPU MHz):" sos_commands/processor/lscpu
   fi

2. Check load averages:

   # Parse uptime for load averages
   if [ -f uptime ]; then
     cat uptime
   fi
   
   # Or from proc/loadavg
   if [ -f proc/loadavg ]; then
     cat proc/loadavg
   fi

3. Interpret load averages:

   • Load average format: 1-min, 5-min, 15-min
   • Compare with CPU count from lscpu
   • Load > CPU count → System overloaded
   • Load >> CPU count (2x or more) → Critical overload

4. Check for CPU throttling:

   # Look for thermal throttling in logs
   grep -i "throttl\|temperature\|thermal" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

5. Identify CPU issues:

   • 1-min load > 2x CPU count → Critical
   • 5-min load > CPU count → Warning
   • Thermal throttling present → Hardware/cooling issue

Step 3: Analyze Disk Usage

1. Parse df output for filesystem usage:

   if [ -f sos_commands/filesys/df_-al ]; then
     # Skip header and special filesystems, show only regular filesystems
     grep -v "^Filesystem\|tmpfs\|devtmpfs\|overlay" sos_commands/filesys/df_-al | grep -v "^$"
   fi

2. Identify full or nearly-full filesystems:

   # Extract filesystems with usage > 85%
   if [ -f sos_commands/filesys/df_-al ]; then
     awk 'NR>1 && $5+0 >= 85 {print $5, $6, $1}' sos_commands/filesys/df_-al | grep -v "tmpfs\|devtmpfs"
   fi

3. Check disk I/O errors:

   # Look for I/O errors in logs
   grep -i "i/o error\|read error\|write error\|bad sector" var/log/dmesg 2>/dev/null
   grep -i "i/o error\|read error\|write error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

4. Analyze block devices:

   if [ -f sos_commands/block/lsblk ]; then
     cat sos_commands/block/lsblk
   fi

5. Identify disk issues:

   • Filesystem > 95% full → Critical
   • Filesystem > 85% full → Warning
   • I/O errors present → Hardware issue
   • Root filesystem full → System stability risk

Step 4: Analyze Process Information

1. Parse ps output:

   if [ -f sos_commands/process/ps_auxwww ]; then
     # Show header
     head -1 sos_commands/process/ps_auxwww
   fi

2. Find top CPU consumers:

   # Sort by CPU usage (column 3), show top 10
   if [ -f sos_commands/process/ps_auxwww ]; then
     tail -n +2 sos_commands/process/ps_auxwww | sort -k3 -rn | head -10
   fi

3. Find top memory consumers:

   # Sort by memory usage (column 4), show top 10
   if [ -f sos_commands/process/ps_auxwww ]; then
     tail -n +2 sos_commands/process/ps_auxwww | sort -k4 -rn | head -10
   fi

4. Check for zombie processes:

   # Look for processes in Z state
   if [ -f sos_commands/process/ps_auxwww ]; then
     grep " Z " sos_commands/process/ps_auxwww || echo "No zombie processes found"
   fi

5. Count processes by state:

   # Count processes by state (R=running, S=sleeping, D=uninterruptible, Z=zombie, T=stopped)
   if [ -f sos_commands/process/ps_auxwww ]; then
     tail -n +2 sos_commands/process/ps_auxwww | awk '{print $8}' | cut -c1 | sort | uniq -c
   fi

6. Identify process issues:

   • Zombie processes present → Parent process not reaping children
   • Many processes in D state → I/O bottleneck
   • Single process using >80% memory → Memory leak or expected behavior
   • Many processes using high CPU → CPU contention