Rescue

Production Incident Commander

When production is down, every minute costs trust. This skill runs an incident like a principal SRE — fast triage, clear decision-making, structured recovery, and prevention so it never happens again.

Severity Classification

Before doing anything, classify the incident:

Severity determines urgency. SEV-1/2: restore first, investigate later. SEV-3/4: investigate first, then fix.

Process

Phase 1: Gather Context

Ask for:

1. Production URL (if not already known)
2. What's happening? (down, slow, errors, specific feature broken)
3. When did it start? (narrows the commit search window)
4. What changed recently? (deploy, config change, dependency update, traffic spike)

If the user is panicking, skip questions and use whatever info is available. Speed > completeness for SEV-1.

Phase 2: Run Diagnostics

node ${CLAUDE_PLUGIN_ROOT}/tools/incident-commander.mjs <project-directory> --url=<production-url>

Parse the JSON output.

If a Sentry MCP server is connected (check your available tools — search for sentry tools), pull live production errors before guessing at code: list the most recent / most frequent issues since the incident window, read the top stack traces, and map each frame back to a file and line in this repo. A real stack trace from production beats inferring the culprit from recent commits. Use the actual error signature to narrow the suspect commit. If no Sentry server is connected, continue with the static diagnostics above (and mention that connecting Sentry would sharpen this step).

Phase 3: Triage

Present findings in order of urgency:

Site Status:

• UP / DOWN / DEGRADED
• Response time and status code
• Health endpoint status
• SSL certificate validity

Likely Culprit:

• Most recent commit with significant changes
• Files changed in that commit
• When it was deployed
• Correlation: did the issue start after this deploy?

Error Patterns Found:

• Unhandled promises, missing error handlers
• Environment variable issues (missing, placeholder values)
• Database connection problems
• Third-party service failures

Resource Issues:

• Memory pressure signals (process.memoryUsage patterns in code)
• Unbounded data growth (arrays that grow without cleanup)
• Connection pool exhaustion (too many DB connections)

Phase 4: Recovery Options

Present in order of speed — for SEV-1/2, always recommend Option 1 first:

Option 1: Rollback (fastest — 2-5 min)

git revert <culprit-hash> --no-edit && git push

This is almost always the right first move. Restore service, then investigate.

When NOT to rollback:

• The rollback would cause data loss (destructive migration already ran)
• The issue isn't in the latest deploy (pre-existing problem that suddenly surfaced)
• The rollback is bigger than the fix (e.g., reverting 50 files when the fix is 1 line)

Option 2: Hot Fix (5-15 min) If the error pattern is clear and the fix is small:

• Apply the fix using Edit tool
• Run tests locally
• Push the fix with a clear commit message: fix: [what was broken] — incident [date]
• Verify with health check

Option 3: Traffic Management (immediate) If the issue is load-related:

• Enable maintenance mode if available
• Scale up infrastructure if possible (Railway: increase instance count)
• Add rate limiting to affected endpoints
• Redirect traffic away from broken feature

Option 4: Investigate Further If the cause isn't clear:

• Check application logs (Railway: railway logs, Vercel: function logs)
• Check database connectivity and query performance
• Check third-party service status pages (Stripe, Resend, Supabase, etc.)
• Check recent environment variable changes
• Check if DNS/SSL certificate expired