Sentry Incident Runbook

Alert fires →
├── Check crash-free rate (Project Settings → Crash Free Sessions)
│   └── Below 95%? → P0
├── Check unique users affected (Issue Details → Users tab)
│   ├── >100/hr on core flow? → P1
│   └── <100/hr or workaround exists? → P2
└── Staging-only or edge case? → P3

# Fetch issue details programmatically
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "
import json, sys
issue = json.load(sys.stdin)
print(f'Title:      {issue[\"title\"]}')
print(f'First Seen: {issue[\"firstSeen\"]}')
print(f'Last Seen:  {issue[\"lastSeen\"]}')
print(f'Events:     {issue[\"count\"]}')
print(f'Users:      {issue[\"userCount\"]}')
print(f'Level:      {issue[\"level\"]}')
print(f'Status:     {issue[\"status\"]}')
print(f'Platform:   {issue.get(\"platform\", \"unknown\")}')
" || echo "ERROR: Failed to fetch issue — check SENTRY_AUTH_TOKEN and ISSUE_ID"

# Fetch latest events for the issue (most recent 5)
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/events/?per_page=5" \
  | python3 -c "
import json, sys
events = json.load(sys.stdin)
for e in events:
    release = e.get('release', {})
    ver = release.get('version', 'N/A') if isinstance(release, dict) else 'N/A'
    print(f'Event {e[\"eventID\"][:12]} | {e.get(\"dateCreated\", \"N/A\")} | Release: {ver}')
" || echo "ERROR: Failed to fetch events"

# Count total events and unique affected users in last 24 hours
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/events/" \
  --data-urlencode "field=count()" \
  --data-urlencode "field=count_unique(user)" \
  --data-urlencode "query=issue.id:$ISSUE_ID" \
  --data-urlencode "statsPeriod=24h" \
  -G | python3 -c "
import json, sys
data = json.load(sys.stdin)
if 'data' in data and data['data']:
    row = data['data'][0]
    print(f'Events (24h):       {row.get(\"count()\", \"N/A\")}')
    print(f'Unique users (24h): {row.get(\"count_unique(user)\", \"N/A\")}')
" || echo "ERROR: Discover query failed"

# Check p95 transaction duration for affected endpoint
curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/events/" \
  --data-urlencode "field=transaction" \
  --data-urlencode "field=p95(transaction.duration)" \
  --data-urlencode "field=count()" \
  --data-urlencode "query=has:transaction event.type:transaction" \
  --data-urlencode "statsPeriod=1h" \
  --data-urlencode "sort=-count()" \
  --data-urlencode "per_page=5" \
  -G | python3 -c "
import json, sys
data = json.load(sys.stdin)
if 'data' in data:
    for row in data['data']:
        txn = row.get('transaction', 'unknown')
        p95 = row.get('p95(transaction.duration)', 0)
        cnt = row.get('count()', 0)
        print(f'{txn}: p95={p95:.0f}ms, count={cnt}')
" || echo "ERROR: Transaction query failed"

# Mark issue as resolved (closes the issue)
curl -s -X PUT \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved"}' \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Status: {d.get(\"status\",\"unknown\")}')" \
  || echo "ERROR: Failed to resolve issue"

# Resolve in next release (auto-reopens on regression)
curl -s -X PUT \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "resolved", "statusDetails": {"inNextRelease": true}}' \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Status: {d.get(\"status\",\"unknown\")} (regression detection enabled)')" \
  || echo "ERROR: Failed to resolve issue"

# Ignore with threshold (snooze until count exceeds limit)
# Use 100 as the re-alert threshold for noisy low-severity issues
curl -s -X PUT \
  -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status": "ignored", "statusDetails": {"ignoreCount": 100}}' \
  "https://sentry.io/api/0/organizations/$SENTRY_ORG/issues/$ISSUE_ID/" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(f'Status: {d.get(\"status\",\"unknown\")} (snoozed until 100 events)')" \
  || echo "ERROR: Failed to ignore issue"

INCIDENT — [Service Name]
Status: Investigating
Impact: [Description of user-facing symptoms]
Started: [Timestamp from Sentry "First Seen"]
Sentry Issue: [Link to issue]
Incident Lead: @[on-call engineer]
Next update: 30 minutes

RESOLVED — [Service Name]
Duration: [Total incident time from first alert to resolution]
Root Cause: [One-line description from investigation]
Fix Applied: [What changed — rollback, hotfix, config change]
Postmortem: [Link — due within 48 hours]

## Incident Postmortem: [Title from Sentry Issue]

### Timeline
- [HH:MM] Alert fired — Sentry issue [ISSUE_ID] created
- [HH:MM] On-call engineer acknowledged
- [HH:MM] Root cause identified via [breadcrumbs / stack trace / suspect commits]
- [HH:MM] Fix deployed — [rollback / hotfix description]
- [HH:MM] Error rate returned to baseline, issue resolved in Sentry

### Impact (from Sentry Discover)
- **Duration:** [X hours Y minutes]
- **Total events:** [count() from Discover query]
- **Unique users affected:** [count_unique(user) from Discover query]
- **p95 latency during incident:** [p95(transaction.duration) from Discover]

### Root Cause (5 Whys)
1. Why did the error occur? [Direct cause from stack trace]
2. Why was that code path triggered? [From breadcrumbs]
3. Why was it not caught in testing? [Gap analysis]
4. Why did the alert take [X] minutes? [Alert rule review]
5. Why is this class of error possible? [Systemic cause]

### Action Items
- [ ] [Preventive measure] — Owner: @[name] — Due: [date]
- [ ] Update Sentry alert rules to catch [pattern] earlier
- [ ] Add regression test covering [scenario from breadcrumbs]
- [ ] Review and tighten ownership rules for [component]