Incident Response Lifecycle

Incident Response Lifecycle

Structured process management for network incidents from detection through post-incident review. This skill covers the organizational coordination layer: severity classification, escalation, role assignment, stakeholder communication, recovery coordination, and root cause analysis. It does not cover technical evidence collection, device forensics, or containment execution — use the incident-response-network skill for network-level evidence gathering and forensic analysis.

The procedure follows the operational lifecycle shape: detect and classify the incident, triage and escalate to the right people, coordinate the investigation across teams, manage communications to all audiences, drive resolution and recovery, then conduct a blameless post-incident review.

See references/communication-templates.md for notification templates by audience and severity level. See references/rca-framework.md for the 5-whys methodology, fishbone diagram guidance, and post-mortem document structure.

When to Use

• Service-affecting incident declared — a P1 or P2 event requires formal incident management with role assignment and communications
• Escalation decision needed — determining who to notify at what severity level and when to engage vendor support or management
• Multi-team coordination required — investigation spans network, security, application, and infrastructure teams needing a single command structure
• Customer or regulatory notification required — incident has external communication obligations (SLA breach, data exposure, regulatory reporting)
• Post-incident review facilitation — scheduling, structuring, and running blameless post-mortems with 5-whys root cause analysis
• Incident metrics reporting — collecting MTTD, MTTI, MTTR, and recurrence data for continuous improvement

Prerequisites

• Incident management authority — the person initiating this process must have authorization to declare incidents and assign roles within the organization
• Contact directory — current on-call rosters, escalation contacts for management, vendor TAC numbers, and regulatory notification contacts must be accessible
• Communication channels — bridge call infrastructure (conference line or collaboration tool), status page access, and email distribution lists for each stakeholder group must be established
• Incident tracking system — a ticketing system to record the incident, track actions, and maintain the timeline of events
• Defined severity criteria — organizational agreement on what constitutes P1 through P4 severity (see Threshold Tables below for a reference framework)

Procedure

Follow these six steps in sequence. Steps 3 and 4 run in parallel once roles are assigned — investigation coordination and communication management proceed simultaneously. Each step references templates from references/communication-templates.md and methodology from references/rca-framework.md where applicable.

Step 1: Detection and Classification

Classify the incident by severity, type, and scope to determine the appropriate response level.

Severity assignment — apply the P1–P4 taxonomy from the Threshold Tables section. Base severity on the highest-impact criterion met. When multiple criteria apply at different levels, the highest governs.

Incident type classification — categorize as outage (service unavailable), degradation (reduced capacity), security (unauthorized access or data exposure), or data loss (corruption or deletion).

Scope determination — assess whether the incident affects a single device, a network segment, an entire site, or multiple sites. Scope drives staffing, communication breadth, and recovery complexity.

Initial impact assessment — estimate affected user count, impacted services and their business criticality, data at risk, and revenue impact per hour. Record estimates in the incident ticket.

Step 2: Triage and Escalation

Assign roles, notify stakeholders, and set response timeline expectations based on the severity classification from Step 1.

Role assignment — every P1 or P2 incident needs four named roles: Incident Commander (IC) — owns the incident end-to-end and makes escalation decisions; Technical Lead — coordinates diagnostics and synthesizes findings; Communications Lead — drafts stakeholder notifications and manages the status page; Scribe — maintains the real-time timeline and records bridge call decisions. For P3, IC and Technical Lead may be combined. P4 uses normal operations workflows.

Escalation matrix execution — notify by severity: P1 — all four roles plus engineering management, VP/director on-call, vendor TAC if vendor equipment is involved, executive notification within 30 minutes. P2 — all four roles plus engineering management within 1 hour. P3 — Technical Lead plus team lead within 4 hours. P4 — assigned engineer via normal ticket queue.

Response timeline expectations: P1 — bridge in 15 minutes, first update in 30 minutes, then every 30 minutes. P2 — bridge in 30 minutes, first update in 1 hour, then every 2 hours. P3 — initial assessment in 4 hours, daily updates. P4 — acknowledgment within 1 business day.

Vendor engagement criteria — engage vendor TAC when the incident involves hardware failure, software defects requiring patches, or when internal triage has not identified root cause within the severity time window.

Step 3: Investigation Coordination

Coordinate the technical investigation across teams and evidence sources. For network-level evidence collection (device state, routing tables, interface data, log retrieval), reference the incident-response-network skill — this step focuses on organizing the investigation, not executing forensic commands.

Evidence collection tasking — assign team members to collect evidence from relevant domains: network devices (via incident-response-network procedures), application logs, infrastructure metrics, and security tooling alerts. Each assignee reports findings to the Technical Lead.

Parallel investigation streams — for complex incidents, run multiple investigation threads simultaneously. Common parallel tracks: (1) symptom analysis — what is failing and for whom, (2) change correlation — what changed recently (deployments, config modifications, maintenance), (3) external factors — upstream provider issues, DDoS, DNS resolution failures.

Hypothesis tracking — maintain a running list of hypotheses with current status (investigating, confirmed, ruled out). Each hypothesis should have an owner and a validation method. Update the list on every bridge call.

Timeline of events (ToE) — the Scribe maintains a running chronological log of when events occurred, when they were detected, what actions were taken, and what was discovered. The ToE becomes the foundation for the post-incident review in Step 6.

Subject matter expert engagement — when investigation stalls or enters an unfamiliar domain, escalate to specialists. Define clear handoff: what has been tried, what data is available, and what specific question needs answering.

Step 4: Communication Management

Manage stakeholder communications throughout the incident. Use the templates in references/communication-templates.md for consistent messaging across audiences.

Stakeholder notification by audience — executive summary (business impact, estimated resolution, customer exposure — no technical detail), technical detail (root cause hypothesis, diagnostics, remediation plan — delivered on bridge call), customer-facing (service impact, workaround if available, estimated resolution — via status page), regulatory (formal notification per compliance framework when required). Use templates from references/communication-templates.md.

Status update cadence — follow severity-based cadence from Step 2. Each update includes: current status, progress since last update, next planned action, and revised time-to-resolution estimate.

Bridge call management — the IC runs calls with a fixed agenda: (1) technical status from Tech Lead, (2) communication status from Comms Lead, (3) hypothesis updates, (4) decisions needed, (5) action items with owners and deadlines. Keep calls focused — park side discussions as action items.

External notification requirements — track regulatory reporting deadlines, law enforcement notification when criminal activity is suspected, customer SLA breach notification per contractual terms, and vendor escalation for ongoing support.

Step 5: Resolution and Recovery

Drive service restoration through validated recovery steps with monitoring to confirm the fix holds.

Recovery validation criteria — before declaring resolved, confirm: (1) service health checks return normal for all affected components, (2) monitoring dashboards show green for at least 15 minutes (P1) or 30 minutes (P2), (3) no new related alerts during observation, (4) affected users confirm restoration (sample check for large populations).

Phased restoration — for multi-layer network incidents, restore in order: core infrastructure → distribution layer → access layer → end-to-end verification. Verify each phase before proceeding. Do not restore all layers simultaneously — cascading failures during recovery are worse than a phased approach.

Back-out plan execution — if the fix causes new issues, execute the pre-defined rollback. Every remediation action should have a documented rollback method before execution.

Enhanced monitoring period — maintain heightened monitoring after resolution: P1 for 24 hours, P2 for 12 hours, P3 through the next business day. This means reduced alert thresholds on affected systems, active watch by on-call, and immediate re-escalation if symptoms recur.

Incident closure — send closure notification to all stakeholders (template in references/communication-templates.md). Update the ticket with resolution summary, total duration, and final impact. Schedule the post-incident review.