Managing Incidents

Incident Management

Provide end-to-end incident management guidance covering detection, response, communication, and learning. Emphasizes SRE culture, blameless post-mortems, and structured processes for high-reliability operations.

When to Use This Skill

Apply this skill when:

• Setting up incident response processes for a team
• Designing on-call rotations and escalation policies
• Creating runbooks for common failure scenarios
• Conducting blameless post-mortems after incidents
• Implementing incident communication protocols (internal and external)
• Choosing incident management tooling and platforms
• Improving MTTR and incident frequency metrics

Core Principles

Incident Management Philosophy

Declare Early and Often: Do not wait for certainty. Declaring an incident enables coordination, can be downgraded if needed, and prevents delayed response.

Mitigation First, Root Cause Later: Stop customer impact immediately (rollback, disable feature, failover). Debug and fix root cause after stability restored.

Blameless Culture: Assume good intentions. Focus on how systems failed, not who failed. Create psychological safety for honest learning.

Clear Command Structure: Assign Incident Commander (IC) to own coordination. IC delegates tasks but does not do hands-on debugging.

Communication is Critical: Internal coordination via dedicated channels, external transparency via status pages. Update stakeholders every 15-30 minutes during critical incidents.

Severity Classification

Standard severity levels with response times:

SEV0 (P0) - Critical Outage:

• Impact: Complete service outage, critical data loss, payment processing down
• Response: Page immediately 24/7, all hands on deck, executive notification
• Example: API completely down, entire customer base affected

SEV1 (P1) - Major Degradation:

• Impact: Major functionality degraded, significant customer subset affected
• Response: Page during business hours, escalate off-hours, IC assigned
• Example: 15% error rate, critical feature unavailable

SEV2 (P2) - Minor Issues:

• Impact: Minor functionality impaired, edge case bug, small user subset
• Response: Email/Slack alert, next business day response
• Example: UI glitch, non-critical feature slow

SEV3 (P3) - Low Impact:

• Impact: Cosmetic issues, no customer functionality affected
• Response: Ticket queue, planned sprint
• Example: Visual inconsistency, documentation error

For detailed severity decision framework and interactive classifier, see references/severity-classification.md.

Incident Roles

Incident Commander (IC):

• Owns overall incident response and coordination
• Makes strategic decisions (rollback vs. debug, when to escalate)
• Delegates tasks to responders (does NOT do hands-on debugging)
• Declares incident resolved when stability confirmed

Communications Lead:

• Posts status updates to internal and external channels
• Coordinates with stakeholders (executives, product, support)
• Drafts post-incident customer communication
• Cadence: Every 15-30 minutes for SEV0/SEV1

Subject Matter Experts (SMEs):

• Hands-on debugging and mitigation
• Execute runbooks and implement fixes
• Provide technical context to IC

Scribe:

• Documents timeline, actions, decisions in real-time
• Records incident notes for post-mortem reconstruction

Assign roles based on severity:

• SEV2/SEV3: Single responder
• SEV1: IC + SME(s)
• SEV0: IC + Communications Lead + SME(s) + Scribe

For detailed role responsibilities, see references/incident-roles.md.

On-Call Management

Rotation Patterns

Primary + Secondary:

• Primary: First responder
• Secondary: Backup if primary doesn't ack within 5 minutes
• Rotation length: 1 week (optimal balance)

Follow-the-Sun (24/7):

• Team A: US hours, Team B: Europe hours, Team C: Asia hours
• Benefit: No night shifts, improved work-life balance
• Requires: Multiple global teams

Tiered Escalation:

• Tier 1: Junior on-call (common issues, runbook-driven)
• Tier 2: Senior on-call (complex troubleshooting)
• Tier 3: Team lead/architect (critical decisions)

Best Practices

• Rotation length: 1 week per rotation
• Handoff ceremony: 30-minute call to discuss active issues
• Compensation: On-call stipend + time off after major incidents
• Tooling: PagerDuty, Opsgenie, or incident.io
• Limits: Max 2-3 pages per night; escalate if exceeded

Incident Response Workflow

Standard incident lifecycle:

Detection → Triage → Declaration → Investigation
  ↓
Mitigation → Resolution → Monitoring → Closure
  ↓
Post-Mortem (within 48 hours)

Key Decision Points

When to Declare: When in doubt, declare (can always downgrade severity)

When to Escalate:

• No progress after 30 minutes
• Severity increases (SEV2 → SEV1)
• Specialized expertise needed

When to Close:

• Issue resolved and stable for 30+ minutes
• Monitoring shows all metrics at baseline
• No customer-reported issues

For complete workflow details, see references/incident-workflow.md.

Communication Protocols

Internal Communication

Incident Slack Channel:

• Format: #incident-YYYY-MM-DD-topic-description
• Pin: Severity, IC name, status update template, runbook links

War Room: Video call for SEV0/SEV1 requiring real-time voice coordination

Status Update Cadence:

• SEV0: Every 15 minutes
• SEV1: Every 30 minutes
• SEV2: Every 1-2 hours or at major milestones

External Communication

Status Page:

• Tools: Statuspage.io, Instatus, custom
• Stages: Investigating → Identified → Monitoring → Resolved
• Transparency: Acknowledge issue publicly, provide ETAs when possible

Customer Email:

• When: SEV0/SEV1 affecting customers
• Timing: Within 1 hour (acknowledge), post-resolution (full details)
• Tone: Apologetic, transparent, action-oriented

Regulatory Notifications:

• Data Breach: GDPR requires notification within 72 hours
• Financial Services: Immediate notification to regulators
• Healthcare: HIPAA breach notification rules

For communication templates, see examples/communication-templates.md.

Runbooks and Playbooks

Runbook Structure

Every runbook should include:

1. Trigger: Alert conditions that activate this runbook
2. Severity: Expected severity level
3. Prerequisites: System state requirements
4. Steps: Numbered, executable commands (copy-pasteable)
5. Verification: How to confirm fix worked
6. Rollback: How to undo if steps fail
7. Owner: Team/person responsible
8. Last Updated: Date of last revision

Best Practices

• Executable: Commands copy-pasteable, not just descriptions
• Tested: Run during disaster recovery drills
• Versioned: Track changes in Git
• Linked: Reference from alert definitions
• Automated: Convert manual steps to scripts over time

For runbook templates, see examples/runbooks/ directory.

Blameless Post-Mortems

Blameless Culture Tenets

Assume Good Intentions: Everyone made the best decision with information available.

Focus on Systems: Investigate how processes failed, not who failed.

Psychological Safety: Create environment where honesty is rewarded.

Learning Opportunity: Incidents are gifts of organizational knowledge.

Post-Mortem Process

1. Schedule Review (Within 48 Hours): While memory is fresh

2. Pre-Work: Reconstruct timeline, gather metrics/logs, draft document

3. Meeting Facilitation:

• Timeline walkthrough
• 5 Whys Analysis to identify systemic root causes
• What Went Well / What Went Wrong
• Define action items with owners and due dates

4. Post-Mortem Document:

• Sections: Summary, Timeline, Root Cause, Impact, What Went Well/Wrong, Action Items
• Distribution: Engineering, product, support, leadership
• Storage: Archive in searchable knowledge base

5. Follow-Up: Track action items in sprint planning

For detailed facilitation guide and template, see references/blameless-postmortems.md and examples/postmortem-template.md.