Runbook Gen

## 5. War Room Setup

### Communications
- **Slack channel**: `#inc-sev1-[incident-name]`
- **Zoom bridge**: [link] (permanent war room)
- **Shared doc**: Google Doc or Notion page for live notes
- **Status page**: Update immediately on detection
- **Customer comms**: Template below

### Status Page Updates

### Customer Communication

## 6. Post-Mortem Template

## Error Handling

| Scenario | Behaviour | Guidance |
|----------|-----------|----------|
| User doesn't know the service/platform | Ask clarifying questions one at a time (not a list) | Start with "what service is this for?" |
| User says "just give me a template" | Output a blank Runbook Template with placeholder brackets | Skip Discovery, fill later |
| User asks for both runbook + post-mortem at once | Generate runbook first, then offer post-mortem | "I'll write the runbook now. Do you have an incident in mind for the post-mortem?" |
| Service has no existing monitoring | Flag the gap, offer to add basic health-check instructions | Note alerts must be configured separately |
| Runbook for a third-party/SaaS dependency | Include vendor status page check; mark resolution as "vendor fixes" | Focus on detection + escalation, not resolution |
| User says "I don't know the escalation contacts" | Use generic roles (Primary on-call, Secondary, EM, VP Eng) | Leave bracketed placeholders |
| Multiple teams involved | Add a RACI section to the runbook | Clarify who decides vs who executes |
| Runbook already exists, user wants update | Treat as revision: read current, diff, propose changes | Flag deprecations, test outdated commands |

## 8. Production Checklist

Before marking a runbook complete, verify every item:

- [ ] **Commands are copy-paste ready** — no unsubstituted variables (`[ns]` only in section headers/notes, never in commands)
- [ ] **Every step has a time-box** — engineer knows when to escalate
- [ ] **Decision tree is shallow** — max 3-4 levels; deeper means split into sub-runbooks
- [ ] **Multiple resolution paths** — at least 2 common failure modes covered
- [ ] **One runbook per incident type** — don't combine "DB failover" and "deployment rollback"
- [ ] **Verification step after every resolution** — how to confirm the fix worked
- [ ] **Escalation contacts listed** — primary + backup, both Slack and phone
- [ ] **Status page templates included** — Investigating / Monitoring / Resolved
- [ ] **Runbook tested in a drill** — within the current quarter, not when incident strikes
- [ ] **README section added** — link to runbook from team's on-call repo
- [ ] **Secrets checked** — no passwords, API keys, or tokens in commands; use env vars
- [ ] **Reviewed by secondary on-call** — fresh eyes catch assumptions

## Anti-Patterns

| Anti-pattern | Why It Fails | Fix |
|-------------|-------------|-----|
| Steps that say "fix the issue" | Too vague under pressure | Tell HOW, not what — exact commands |
| Commands with unsubstituted variables | Copy-paste fails, engineer wastes time retyping | Every command must run as-is |
| No verification step | Fix might not work, but nobody knows | Add check after every resolution |
| Runbooks >6 months stale | Commands rot, trust erodes, people ignore them | Schedule quarterly review in team calendar |
| Runbooks nobody tested | First test happens during the actual incident | Require one drill per quarter per runbook |
| Too many steps (>15) | Cognitive overload during Sev1 | Split into sub-runbooks, keep shallow |
| No time-boxes | Engineer doesn't know when to escalate | Every diagnosis/resolution step has a max time |
| Single resolution path | Assumes first idea works | Always have Plan B in the same runbook |
| Post-mortem blames people | Blame culture kills incident reporting | "What broke the system?" not "Who broke it?" |
| Skipping customer comms | Stakeholders hear "we're down" from social media | Draft customer email as part of template |

## Incident Timeline Template

## Post-Mortem Template

## Sources

- Google SRE Book: "Monitoring Distributed Systems"
- Google SRE Workbook: Incident Response
- PagerDuty incident response documentation
- Atlassian post-mortem best practices
- Microsoft SRE practices
- AWS Well-Architected Framework: Operational Excellence

## Checklist

- [ ] Skill loads without errors in the AI agent
- [ ] YAML frontmatter is valid (description, compatibility, audience)
- [ ] Workflow section provides clear step-by-step instructions
- [ ] Error handling section covers common failure modes
- [ ] All referenced files (references/, scripts/, assets/) exist
- [ ] Skill triggers correctly for intended use cases
- [ ] No broken links or missing resources