Oncall Runbook

On-Call Runbook Skill

Produce a complete on-call runbook for a service — giving the on-call engineer everything they need to respond confidently to alerts at 3am, without having to ask anyone for help.

A good on-call runbook reduces mean time to resolution (MTTR) by eliminating the "what do I do first?" problem. It is written for the on-call engineer who has just been paged and needs to act, not for someone calmly reading documentation.

Where this sits — the spine's terminus

Last in the incident-response spine: /slo-error-budget (frame) → /debugging-log-analyser → /incident-postmortem → oncall-runbook. It receives the contributing factors and action items from /incident-postmortem and turns the detection/mitigation learnings into an entry that makes the next responder minutes, not hours — closing the loop so the same incident doesn't recur at full cost. Runbook entry, detection/mitigation time, and the loop are defined once in docs/craft/incident-response.md.

The loop

A runbook fails when it's written for a calm reader instead of a paged one at 3am. Phase 1 sets the audience; every later choice serves it.

1. Write for the paged engineer, not the documentarian. The reader has just been woken and needs to act — so lead with the fastest safe mitigation, put copy-pasteable commands first, and defer background. Prose that explains before it acts fails at 3am. Done when: each alert's entry lets a non-expert take the first safe action within a minute of opening it, without reading theory.
2. Turn postmortem learnings into per-alert procedures. For each known failure (the incident-postmortem's are the highest-value), write detect → mitigate → escalate: the exact checks, the copy-pasteable commands, the rollback, and when to page whom. Done when: every alert maps to a procedure with concrete commands and a clear mitigation, not just "investigate."
3. Make escalation and handoff unambiguous. Who to page, when, and how to hand off mid-incident — because the second failure mode after "what do I do?" is "who do I wake, and when is it okay to?" Done when: the escalation matrix names people/rotations and the trigger for each, and the handoff template captures state so the next responder isn't starting cold.
4. Close the loop back to prevention. Flag where a runbook step reveals a gap that should become monitoring or an /slo-error-budget action — the runbook is where the loop's learnings surface the next prevention. Done when: gaps found while writing the runbook are logged as detection/prevention improvements, not silently absorbed.

Required Inputs

Ask for these if not already provided:

• Service name and what it does
• Team and tech lead name
• Alert list — names of alerts that currently page on-call
• Monitoring setup — Datadog / Grafana / CloudWatch / PagerDuty / etc.
• Common failure modes — what breaks most often, and what fixes it
• Escalation contacts — who to call when on-call can't resolve it
• Deployment setup — can on-call roll back? How?
• Service dependencies — what does this service depend on, and what depends on it?

Output Format

On-Call Runbook: [Service Name]

Team: [Team name] | Tech lead: [Name] PagerDuty service: [Link] | Escalation policy: [Policy name] Last updated: [Date] | Next review: [Date + 90 days]

First time on-call for this service? Read the [developer onboarding doc] first — it covers the architecture and how things work. This runbook assumes you understand the service.

Quick Reference

Dashboard: [Link — the first thing to open when paged] Logs: [Link — where to find logs] Runbook index: Jump to the alert that paged you → [Alert list below] Can't resolve in 30 min? Escalate to: [Name] via [Slack / PagerDuty]

Rollback command (memorise this):

[rollback command — e.g. kubectl rollout undo deployment/[service-name]]

Escalation Matrix

Contacts:

Service Architecture (Quick View)

[Upstream callers]
        │
        ▼
[This Service]
        │
        ├──→ [Primary Database]
        ├──→ [Cache — e.g. Redis]
        └──→ [Downstream Service / Queue]

If this service is down, these are affected: [List downstream consumers] If these are down, this service is affected: [List upstream dependencies]

Alert Runbooks

ALERT: [Alert Name 1 — e.g. HighErrorRate]

What it means: [Plain English — e.g. "More than 5% of API requests are returning 5xx errors in the last 5 minutes"] Severity: P1 / P2 / P3 SLO impact: Yes / No — [If yes: this alert means the error budget is burning at [X]× rate]

Step 1 — Acknowledge and assess

# Check current error rate
[query or dashboard link]

# Check which endpoints are erroring
[query or command]

Step 2 — Check recent changes

# Any deploys in the last hour?
[command or link to deployment log]

# Recent config changes?
[where to check]

Step 3 — Check dependencies

# Is the database healthy?
[health check command or link]

# Is [downstream service] healthy?
[health check command or link]

Step 4 — Diagnose

Step 5 — Fix or mitigate

# If caused by bad deploy — roll back:
[rollback command]

# If caused by [specific issue]:
[fix command]

# If caused by upstream dependency:
[mitigation — e.g. enable circuit breaker, reduce traffic, etc.]

After resolving:

• Confirm error rate has returned to baseline
• Check no downstream services were affected
• If P1: open a post-incident review — see [incident-postmortem skill]
• Update #incidents with resolution summary