System Config Analysis

System Configuration Analysis Skill

This skill provides detailed guidance for analyzing system configuration from sosreport archives, including OS information, installed packages, systemd services, and SELinux/AppArmor settings.

When to Use This Skill

Use this skill when:

• Analyzing the /sosreport:analyze command's system configuration phase
• Investigating service failures or misconfigurations
• Verifying package versions and updates
• Checking security policy settings (SELinux/AppArmor)
• Understanding system state and configuration

Prerequisites

• Sosreport archive must be extracted to a working directory
• Path to the sosreport root directory must be known
• Understanding of Linux system administration

Key Configuration Data Locations in Sosreport

1. System Information:

   • uname - Kernel version
   • etc/os-release - OS distribution and version
   • uptime - System uptime
   • proc/uptime - Uptime in seconds
   • sos_commands/release/ - Release information

2. Package Information:

   • installed-rpms - RPM packages (RHEL/Fedora/CentOS)
   • installed-debs - DEB packages (Debian/Ubuntu)
   • sos_commands/yum/ - Yum/DNF information
   • sos_commands/rpm/ - RPM database queries

3. Service Status:

   • sos_commands/systemd/systemctl_list-units - All units
   • sos_commands/systemd/systemctl_list-units_--failed - Failed units
   • sos_commands/systemd/systemctl_status_--all - Detailed service status
   • sos_commands/systemd/systemctl_list-unit-files - Unit files

4. SELinux:

   • sos_commands/selinux/sestatus - SELinux status
   • sos_commands/selinux/getenforce - Current enforcement mode
   • sos_commands/selinux/selinux-policy - Policy information
   • var/log/audit/audit.log - SELinux denials

5. AppArmor (if applicable):

   • sos_commands/apparmor/ - AppArmor configuration
   • etc/apparmor.d/ - AppArmor profiles

6. System Configuration Files:

   • etc/ - System-wide configuration
   • etc/sysctl.conf or etc/sysctl.d/ - Kernel parameters
   • etc/security/limits.conf - Resource limits

Implementation Steps

Step 1: Analyze System Information

1. Check OS version and distribution:

   if [ -f etc/os-release ]; then
     cat etc/os-release
   fi

2. Get kernel version:

   if [ -f uname ]; then
     cat uname
   elif [ -f proc/version ]; then
     cat proc/version
   fi

3. Check system uptime:

   if [ -f uptime ]; then
     cat uptime
   elif [ -f proc/uptime ]; then
     # Parse uptime from proc/uptime (seconds)
     awk '{printf "%.2f days\n", $1/86400}' proc/uptime
   fi

4. Extract key system details:

   • OS name and version
   • Kernel version
   • System architecture (x86_64, aarch64, etc.)
   • Uptime (days)

5. Check for outdated kernel or OS:

   • Compare kernel version with current stable
   • Note if system hasn't been rebooted in a very long time (>365 days)
   • Identify if OS version is EOL

Step 2: Analyze Installed Packages

1. List installed packages:

   # For RPM-based systems
   if [ -f installed-rpms ]; then
     cat installed-rpms
   fi
   
   # For DEB-based systems
   if [ -f installed-debs ]; then
     cat installed-debs
   fi

2. Extract key package versions:

   # Important system packages
   grep -E "^(kernel|systemd|glibc|openssh|openssl)" installed-rpms 2>/dev/null
   
   # Or use awk to parse package name and version
   awk '{print $1}' installed-rpms | head -20

3. Check for known problematic versions:

   • Security vulnerabilities (if known CVEs)
   • Buggy package versions
   • Compatibility issues

4. Identify package manager issues:

   # Check yum/dnf logs for errors
   if [ -d sos_commands/yum ]; then
     grep -i "error\|fail" sos_commands/yum/* 2>/dev/null
   fi

5. Count packages and categorize:

   • Total packages installed
   • Key package versions (kernel, systemd, glibc, etc.)
   • Recently updated packages (if timestamps available)

Step 3: Analyze Service Status

1. List all systemd units:

   if [ -f sos_commands/systemd/systemctl_list-units ]; then
     cat sos_commands/systemd/systemctl_list-units
   fi

2. Identify failed services:

   if [ -f sos_commands/systemd/systemctl_list-units_--failed ]; then
     cat sos_commands/systemd/systemctl_list-units_--failed
   elif [ -f sos_commands/systemd/systemctl_list-units ]; then
     grep "failed" sos_commands/systemd/systemctl_list-units
   fi

3. Check service details:

   # Parse detailed status for failed services
   if [ -f sos_commands/systemd/systemctl_status_--all ]; then
     # Extract service names and their status
     grep -E "●|Active:" sos_commands/systemd/systemctl_status_--all | head -50
   fi

4. Count services by state:

   # Count running, failed, inactive services
   if [ -f sos_commands/systemd/systemctl_list-units ]; then
     awk '{print $4}' sos_commands/systemd/systemctl_list-units | sort | uniq -c
   fi

5. Identify critical service failures:

   • System services (systemd-*, dbus, NetworkManager)
   • Application services (httpd, nginx, postgresql, etc.)
   • Custom services

6. Extract failure reasons from logs:

   # For each failed service, find related log entries
   grep -i "failed to start\|service.*failed" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

Step 4: Analyze SELinux Configuration

1. Check SELinux status:

   if [ -f sos_commands/selinux/sestatus ]; then
     cat sos_commands/selinux/sestatus
   fi

2. Get SELinux mode:

   if [ -f sos_commands/selinux/getenforce ]; then
     cat sos_commands/selinux/getenforce
   fi

3. Check for SELinux denials:

   # Look for AVC denials in audit log
   if [ -f var/log/audit/audit.log ]; then
     grep "avc.*denied" var/log/audit/audit.log | head -50
   fi
   
   # Or in journald logs
   grep -i "selinux.*denied\|avc.*denied" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -20

4. Parse denial information:

   • Extract denied operations (read, write, execute, etc.)
   • Identify source and target contexts
   • Note which services are affected

5. Check for SELinux booleans:

   if [ -f sos_commands/selinux/getsebool_-a ]; then
     cat sos_commands/selinux/getsebool_-a
   fi

6. Identify SELinux issues:

   • SELinux in permissive mode (may hide errors)
   • SELinux disabled (security concern)
   • Frequent AVC denials (policy may need adjustment)
   • Context mismatches

Step 5: Check System Configuration

1. Review kernel parameters:

   # Check sysctl settings
   if [ -f sos_commands/kernel/sysctl_-a ]; then
     cat sos_commands/kernel/sysctl_-a
   elif [ -d etc/sysctl.d ]; then
     cat etc/sysctl.d/*.conf 2>/dev/null
   fi

2. Check resource limits:

   if [ -f etc/security/limits.conf ]; then
     grep -v "^#\|^$" etc/security/limits.conf
   fi
   
   # Check limits.d directory
   if [ -d etc/security/limits.d ]; then
     cat etc/security/limits.d/*.conf 2>/dev/null
   fi

3. Review boot parameters:

   if [ -f sos_commands/boot/grub2-editenv_list ]; then
     cat sos_commands/boot/grub2-editenv_list
   elif [ -f proc/cmdline ]; then
     cat proc/cmdline
   fi

4. Check systemd configuration:

   # Look for systemd configuration overrides
   if [ -d etc/systemd/system ]; then
     find etc/systemd/system -name "*.conf" 2>/dev/null
   fi