Logs Analysis

Logs Analysis Skill

This skill provides detailed guidance for analyzing logs from sosreport archives, including journald logs, system logs, kernel messages, and application logs.

When to Use This Skill

Use this skill when:

• Analyzing the /sosreport:analyze command's log analysis phase
• Investigating specific log-related errors or warnings in a sosreport
• Performing deep-dive analysis of system failures from logs
• Identifying patterns and root causes in system logs

Prerequisites

• Sosreport archive must be extracted to a working directory
• Path to the sosreport root directory must be known
• Basic understanding of Linux log structure and journald

Key Log Locations in Sosreport

Sosreports contain logs in several locations:

1. Journald logs: sos_commands/logs/journalctl_*

   • journalctl_--no-pager_--boot - Current boot logs
   • journalctl_--no-pager - All available logs
   • journalctl_--no-pager_--priority_err - Error priority logs

2. Traditional system logs: var/log/

   • messages - System-level messages
   • dmesg - Kernel ring buffer
   • secure - Authentication and security logs
   • cron - Cron job logs

3. Application logs: var/log/ (varies by application)

   • httpd/ - Apache logs
   • nginx/ - Nginx logs
   • audit/audit.log - SELinux audit logs

Implementation Steps

Step 1: Identify Available Log Sources

1. Check for journald logs:

   ls -la sos_commands/logs/journalctl_* 2>/dev/null || echo "No journald logs found"

2. Check for traditional system logs:

   ls -la var/log/{messages,dmesg,secure} 2>/dev/null || echo "No traditional logs found"

3. Identify application-specific logs:

   find var/log/ -type f -name "*.log" 2>/dev/null | head -20

Step 2: Analyze Journald Logs

1. Parse journalctl output for error patterns:

   # Look for common error indicators
   grep -iE "(error|failed|failure|critical|panic|segfault|oom)" sos_commands/logs/journalctl_--no-pager 2>/dev/null | head -100

2. Identify OOM (Out of Memory) killer events:

   grep -i "out of memory\|oom.*kill" sos_commands/logs/journalctl_--no-pager 2>/dev/null

3. Find kernel panics:

   grep -i "kernel panic\|bug:\|oops:" sos_commands/logs/journalctl_--no-pager 2>/dev/null

4. Check for segmentation faults:

   grep -i "segfault\|sigsegv\|core dump" sos_commands/logs/journalctl_--no-pager 2>/dev/null

5. Extract service failures:

   grep -i "failed to start\|failed with result" sos_commands/logs/journalctl_--no-pager 2>/dev/null

Step 3: Analyze System Logs (var/log)

1. Check messages for errors:

   # If file exists and is readable
   if [ -f var/log/messages ]; then
     grep -iE "(error|failed|failure|critical)" var/log/messages | tail -100
   fi

2. Check dmesg for hardware issues:

   if [ -f var/log/dmesg ]; then
     grep -iE "(error|fail|warning|i/o error|bad sector)" var/log/dmesg
   fi

3. Analyze authentication logs:

   if [ -f var/log/secure ]; then
     grep -iE "(failed|failure|invalid|denied)" var/log/secure | tail -50
   fi

Step 4: Count and Categorize Errors

1. Count errors by severity:

   # Critical errors
   grep -ic "critical\|panic\|fatal" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0"
   
   # Errors
   grep -ic "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0"
   
   # Warnings
   grep -ic "warning\|warn" sos_commands/logs/journalctl_--no-pager 2>/dev/null || echo "0"

2. Find most frequent error messages:

   grep -iE "(error|failed)" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \
     sed 's/^.*\]: //' | \
     sort | uniq -c | sort -rn | head -10

3. Extract timestamps for error timeline:

   # Get first and last error timestamps
   grep -i "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \
     head -1 | awk '{print $1, $2, $3}'
   grep -i "error" sos_commands/logs/journalctl_--no-pager 2>/dev/null | \
     tail -1 | awk '{print $1, $2, $3}'

Step 5: Analyze Application-Specific Logs

1. Identify application logs:

   find var/log/ -type f \( -name "*.log" -o -name "*_log" \) 2>/dev/null

2. Check for stack traces and exceptions:

   # Python tracebacks
   grep -A 10 "Traceback (most recent call last)" var/log/*.log 2>/dev/null | head -50
   
   # Java exceptions
   grep -B 2 -A 10 "Exception\|Error:" var/log/*.log 2>/dev/null | head -50

3. Look for common application errors:

   # Database connection errors
   grep -i "connection.*refused\|connection.*timeout\|database.*error" var/log/*.log 2>/dev/null
   
   # HTTP/API errors
   grep -E "HTTP [45][0-9]{2}|status.*[45][0-9]{2}" var/log/*.log 2>/dev/null | head -20