Oncall Setup

Phase 2 — Interview

Goal: fill the policy blanks that cannot be mined. Ask only these, one block at a time, offering mined suggestions where you have them:

1. Paging criteria. For each metric worth paging on: threshold, sustain window, and exemptions (deploy windows, known-noisy periods). Suggest values from alert history ("this metric's alerts self-resolved under 4% in 11 of 12 cases — suggest paging at sustained >4%/10min") but the human sets the number (rule 15).

2. Severity norms. What's a page vs. a business-hours ping vs. a morning log line.

3. Escalation owners. Resolve every routing-tree conflict flagged in Phase 1; get the real group handles (route to groups, not individuals). If Phase 1 flagged a bus-factor finding, raise it here as a resilience question and let the team decide whether the tree should distribute load differently than history did.

4. Deploy windows. How to tell a deploy is in progress (the deploys capability, a channel, a calendar).

5. Escalation timeout and fallback alerting. Two decisions, both the human's:

   • Timeout: when Claude posts a page-severity finding and @-mentions the routed owner, how long does it wait for acknowledgment before escalating — and to whom? An ack is an explicit affirmative from a human ("ack", "on it", or the team's designated reaction, from a person) — bot posts, alert traffic, and passive emoji do not count. Suggest a default ({{15 min}} → the escalation handle from block 3), but the human sets both the clock and the ladder. Also ask for the terminal step: if the escalation itself goes unacked, what happens — repeat-page via the pager's escalation policy, a wider channel post, or an explicitly accepted "unattended until morning" posture? The ladder must end somewhere deliberate. Without answers, Claude never re-pings on its own.
   • Fallback: when a page-severity finding can't page — no pager bound in STACK.md, or the page call fails — what happens instead? Offer the options and let them choose: @-mention the escalation group in the on-call channel; post to a designated always-watched channel; or hold for the morning log (only sane for teams with no off-hours expectations — say so). Be honest about the first two: Slack @-mentions don't penetrate Do-Not-Disturb, so an @-mention fallback is business-hours-grade coverage — tell the team this before they choose it. Record the choice in ONCALL.md; never invent a fallback mid-incident.

6. Alert-rule proposals: format and install mode. Two decisions:

   • Format: which alerting tool should proposals target, and in what paste-ready native form (monitor JSON, Terraform, PromQL, UI steps)? Prose proposals are not acceptable output — a proposal is something a human can install in under a minute.
   • Install mode: default — Claude drafts, a human installs (the paste is the permission; keeps the kit fully read-only). Or the alert-editor extension, opt-in only: a separate write credential to the alerting tool, additive-only — Claude may CREATE a new rule after explicit per-rule approval in the channel, may never modify, delete, or silence an existing rule, and logs every write to lessons.md. If they opt in, record it in ONCALL.md and STACK.md's access posture. Present the trade honestly: the extension saves a paste; the default keeps "no write credentials to monitored systems" true without asterisks.

7. Confirm the read-only guarantee. Not a question — a statement to make once, so the team knows the contract: this agent never changes the state of any monitored system; its only outputs are messages, log entries, proposed PRs, and pages. There is no allowlist to configure. Teams that want automated mitigation are outside this kit's scope and should design that separately, on an accountable human identity.

8. Lifecycle windows and standing reports. Three decisions, all human-set numbers (rule 15):

   • Staleness/zombie windows: how long an open incident stays quiet before a "looks stale" nudge ({{24h}} suggested) and before the handoff's zombie list ({{72h}} suggested) — these gate what Claude says, never what it changes.
   • Morning sitrep: on or off, what time, and confirm the "post nothing when empty" behavior.
   • Weather report: opt in or skip — show the cost anchor from templates/routines.md and the cadence-guard design before they choose. Skipping is the default and completely fine (status stays on-demand). If they opt in, three things go into ONCALL.md: the cadence targets, the report-page binding, and the mood tier-boundary table (the two base signals and the human-set boundaries mapping each to sunny/partly_cloudy/overcast/stormy — the weather skill refuses to run without it).

Write the answers into ONCALL.md from templates/ONCALL.md, replacing every {{...}}. Template fields no block covered (e.g. handoff cadence, status-on-demand signals): fill with a sensible default, mark each (proposed), and list them explicitly at the gate for confirmation — never leave blanks, never present a default as the user's decision.

Gate: post the completed ONCALL.md diff. The human signs off the policy. Do not proceed.

Phase 3 — Validate

Goal: prove the drafted playbooks against incidents they weren't built from.

1. Hold out 5–10 resolved incidents not used in Phase 1 (or the most recent ones if history is thin — say so). Span the failure classes and include page-severity incidents where they exist; if none exist, the results file states the paging dimension is untested.
2. For each: take only the triggering alert/first message, run the triage skill as if live (read-only), and produce the diagnosis you would have posted. For long-running holdouts, also produce the >30-min update — graded against triage step 6a's story-so-far spec, not just the diagnosis.
3. Grade in a fresh context (a separate session/subagent that didn't produce the diagnoses, prompted skeptically; human confirms), per eval/replay.md: ✅ correct / ⚠️ partially correct / ❌ wrong / 🚫 harmful (would have misdirected mitigation or paged wrongly).
4. Write eval/replay-results.md: the table, per-incident links, and for every ❌/🚫 the playbook change that would have prevented it, as a proposed diff.

Gate: pass = ≥70% ✅+⚠️ and zero 🚫. Present the percentage as a smoke test, not statistics — with 5–10 holdouts one grade swings ~14 points. The real content of this gate is the per-incident review of every ❌/🚫 and its proposed diff; the real quantitative gate is the shadow period, where evidence actually accumulates. On pass, ask to proceed. On fail, apply the proposed playbook diffs (with human review) and re-run with fresh holdouts; a thin-history team that exhausts its holdouts goes to shadow with the alert-watch routine in review-only mode rather than re-testing on incidents the playbooks have now seen. Never lower the bar.

Phase 4 — Install

Goal: turn it on, narrowest first.

0. Verify channel connectivity before anything else. This phase only works from the Slack channel. The checklist, done by the human: /invite @Claude to the on-call channel and each alert channel to be watched; an Owner adds this repo to the channel's access bundle. Then the proof: from the channel, ask @Claude what can you access from this channel? and have it read ONCALL.md back. If it can't read the repo, stop — pasting routines against a repo the channel can't reach fails silently. Clear the "channel connectivity: unverified" gap in STACK.md once this passes.

1. Generate the routine messages from templates/routines.md, with real channel names, cadences, and STACK.md bindings filled in. Order: handoff (read-only) → morning sitrep (read-only, if chosen) → alert investigation (posts diagnoses) → weather (opt-in, event-gated, if chosen in Interview block 8). There is no detection routine to install — detection stays in the team's deterministic alerting; if a service is launching without alerts, propose starter rules per templates/routines.md instead.

2. Recommend the shadow period for the alert-watch routine (the handoff is a read-only weekly report and goes live immediately). Shadow exits on evidence, not the calendar: diagnoses post to a review channel/thread and are graded daily, and promotion to live follows the shadow-exit bar in eval/replay.md — the canonical source, which also covers the quiet-channel case (too few alerts means extend, not promote). 2a. Create the routine registry. After the pastes, create (or update) a channel canvas — or a pinned message where canvases aren't available — titled "Standing work in this channel": every routine's name, schedule, one-line purpose, live-or-shadow status, and last-changed date, plus one closing line ("to change when/where, edit the routine here; to change how/policy, PR {{repo}}"). Humans install routines; you keep this registry current whenever standing work changes, so what's running is legible at a glance to anyone who joins the channel.

3. The human pastes each routine into the channel (routines belong to the channel and its members — you don't install standing work for a team without them seeing exactly what it says).

Gate (final): confirm each routine the human installed by listing the channel's standing work back. Remind them: to change when/where, edit the routine in-channel; to change how/policy, PR the repo. Setup complete.