Webex Incident Workflow

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "Container",
      "style": "attention",
      "items": [
        {
          "type": "TextBlock",
          "text": "INCIDENT DECLARED -- Network Outage",
          "weight": "Bolder",
          "size": "Large",
          "color": "Attention"
        }
      ]
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Severity:", "value": "P1 -- Service Impacting" },
        { "title": "Detected:", "value": "2024-02-21 14:32 UTC" },
        { "title": "Reporter:", "value": "NetClaw (automated)" }
      ]
    },
    {
      "type": "TextBlock",
      "text": "**Symptoms:**\n- R1 unreachable (ping 0%)\n- 47 downstream routes lost\n- 3 OSPF adjacencies down\n- BGP peer to ISP: IDLE",
      "wrap": true
    },
    {
      "type": "TextBlock",
      "text": "**Impact:**\n- Site A has no WAN connectivity\n- Estimated affected users: ~200",
      "wrap": true
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Incident Commander:", "value": "Awaiting claim" },
        { "title": "ServiceNow:", "value": "CR/INC pending" }
      ]
    },
    {
      "type": "ActionSet",
      "actions": [
        {
          "type": "Action.Submit",
          "title": "Claim IC Role",
          "data": { "action": "claim_ic", "incident_id": "INC-001" }
        }
      ]
    }
  ]
}

Room Name: INC-001 -- R1 Network Outage -- 2024-02-21

**Incident Team Formed**
**IC:** John Smith (claimed at 14:35 UTC)
**NetClaw:** Automated investigation assistant

**Triage Checklist:**
- [x] Alert generated and posted
- [x] Incident declared (P1)
- [x] IC assigned -- John Smith
- [ ] ServiceNow incident created
- [ ] Upstream device checked
- [ ] Blast radius confirmed
- [ ] Customer communication sent

_NetClaw beginning automated investigation..._

**Automated Investigation -- Step 1/4**
_Checking upstream device R2 for connectivity to R1..._

**Investigation Results -- Step 1/4**
**Ping from R2 to R1 (10.1.1.1):** 0% success -- R1 unreachable from upstream

**Investigation Results -- Step 2/4**
**R2 interface Gi1 (toward R1):** up/up, 0 CRC errors, last input 4 min ago
> Physical layer looks OK from R2 side

**Investigation Results -- Step 3/4**
**R2 OSPF neighbors:** R1 missing from neighbor table (was FULL)
> OSPF adjacency lost, DR election may be in progress

**Investigation Results -- Step 4/4**
**R2 logs (last 30 min):**

**Analysis:** R2 saw Gi1 flap at 14:31. Line protocol came back up but OSPF hasn't re-converged. Likely physical issue on R1 side causing interface bounce.

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "TextBlock",
      "text": "Status Update -- 14:50 UTC (18 min elapsed)",
      "weight": "Bolder",
      "size": "Medium"
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Status:", "value": "Investigating" },
        { "title": "Finding:", "value": "R1 appears to have reloaded unexpectedly. R2 sees the link recover but R1 is not responding to OSPF hellos yet." },
        { "title": "Next Step:", "value": "Waiting for R1 to complete boot sequence. Checking console access." },
        { "title": "ETA:", "value": "Unknown -- dependent on R1 recovery" }
      ]
    },
    {
      "type": "TextBlock",
      "text": "_ServiceNow INC0012345 updated_",
      "isSubtle": true
    }
  ]
}

{
  "type": "AdaptiveCard",
  "version": "1.3",
  "body": [
    {
      "type": "Container",
      "style": "good",
      "items": [
        {
          "type": "TextBlock",
          "text": "INCIDENT RESOLVED",
          "weight": "Bolder",
          "size": "Large",
          "color": "Good"
        }
      ]
    },
    {
      "type": "FactSet",
      "facts": [
        { "title": "Duration:", "value": "34 minutes (14:32 -- 15:06 UTC)" },
        { "title": "Resolution:", "value": "R1 experienced a software crash. Device auto-reloaded and recovered. All OSPF adjacencies re-established. Full routing restored." },
        { "title": "Root Cause:", "value": "Software crash -- Traceback found in logs indicating bug CSCxx12345. TAC case recommended." }
      ]
    },
    {
      "type": "TextBlock",
      "text": "**Post-Resolution Verification:**\n- R1 reachable: 100% ping success\n- OSPF neighbors: 3/3 FULL\n- BGP peer: Established\n- Route count: 47 routes (matches baseline)\n- Connectivity: 100% to all targets",
      "wrap": true
    },
    {
      "type": "TextBlock",
      "text": "_ServiceNow INC0012345 resolved | GAIT session abc123 closed_",
      "isSubtle": true
    }
  ]
}

**Post-Incident Review -- Scheduled**
**Incident:** Network Outage -- R1 crash
**Date:** 2024-02-22 10:00 UTC
**Space:** This thread

**Review Artifacts (attached):**
1. Timeline of events
2. R1 show logging output
3. R1 show version (confirms reload reason)
4. GAIT audit trail (full session)
5. Pre/post health check comparison

**Discussion Topics:**
- Was detection fast enough?
- Was automated investigation helpful?
- What monitoring gaps exist?
- Should R1 be upgraded to patched version?
- Do we need redundant path for this link?