Measure Before You Fix

Measure Before You Fix

Convention: see conventions/brain-first.md — before re-deriving a diagnosis, search the brain for prior incidents of the same alert. A recurring alert usually has a recorded verdict already.

Route here on any alert whose claim is temporal — "X is stale", "step timed out", "pipeline wedged", "job is slow", "N hours behind". These alerts invite an immediate structural fix (raise the timeout, split the step, reorder the pipeline). Do the measurement first. It is almost always cheaper than the fix, and it frequently invalidates it. (Routing is a harness convention, not a mechanical guarantee — the contract below is the discipline that makes it stick.)

On gbrain surfaces this covers: gbrain doctor staleness checks (e.g. sync freshness, cycle freshness), autopilot cycle alerts, the sync stall watchdog (reason: 'stall_timeout'), and any cron monitor built on top of them.

The rule

One stopwatch measurement of the suspect step, before any code change.

If you cannot state the measured duration of the thing you claim is slow, you do not yet know the root cause, and any fix you write is a guess wearing a diff.

Contract

This skill guarantees:

• No structural fix (timeout raise, step split, pipeline reorder, wrapper rewrite) is proposed before a measured duration of the blamed step exists.
• The measurement targets the specific entity the alert named, not the aggregate (--all hides which member is slow).
• The alert's threshold is compared against the authoritative one (gbrain doctor's warn/fail lines) before the system is declared unhealthy.
• The verdict explicitly distinguishes "needs more time" from "is wedged" — they have opposite fixes.
• Read-only: this skill changes no timeouts, thresholds, or code. It produces a measurement verdict that sizes the fix; the fix itself is a separate, now-informed change.

Procedure

1. Read the alert's own numbers. They often contradict the theory already. (Locks: none means it isn't lock contention. Note it and drop that branch.)

2. Time the suspect step directly. Isolate the smallest unit that the alert blames, and run it with a clock:

   time gbrain sync --source source-a --no-embed

   Run it on the specific named entity, not the aggregate. A whole-brain run hides which member is slow; --source source-a answers the question. Cross-check state with gbrain sources status (per-source sync lag).

3. Compare measured vs. budgeted. Grep every timeout in the wrapper, not just the default in the helper signature:

   grep -n "timeoutMs\|timeout:" <the wrapper or cron script>

   A generous per-call override makes the helper's default irrelevant. Check the call site before blaming the default.

4. Check the alert threshold against the authoritative one. Before concluding the system is broken, confirm the alerter and the audit agree on what "bad" means. gbrain doctor's sync-freshness check defaults to 24h warn / 72h fail (env-overridable via GBRAIN_SYNC_FRESHNESS_WARN_HOURS / GBRAIN_SYNC_FRESHNESS_FAIL_HOURS); a cron monitor paging at 12h is speaking below the authoritative warn line. A monitor that acts early is correct; a monitor that speaks at its act-line is a false-positive generator.

5. Only now design the fix — against the number you measured.

The threshold-mismatch failure mode

A cron monitor legitimately acts earlier than the doctor fails, to keep drift out of FAIL territory. That is good design. The bug is reusing the act-threshold as the alert-threshold: everything between "act" and "warn" becomes a recurring page about a healthy system.

Separate the two constants. Act at the aggressive line, speak at the authoritative one:

const ACT_HOURS = Number(env.MONITOR_ACT_HOURS || 12);      // act early — fine
const ALERT_HOURS = Math.max(ACT_HOURS, DOCTOR_WARN_HOURS); // speak at the audit's line

Symptom to recognize instantly: a repeating alert whose numbers sit below the doctor's own warn line, while the underlying resource looks fine when queried directly.

Red flags that you are theorizing, not diagnosing

• You have a root cause but no measured duration.
• Your fix is a rewrite and you have not run the step once.
• You revise the theory twice without taking a new measurement between revisions.
• The alert says "no recovery in flight" — verify whether recovery is in fact running before believing it (ps for the worker; check the launch flag; gbrain jobs list for queued work).
• "Already up to date" in the step output. That step is not your bottleneck.

When it fails

Follow the agent operator protocol for any gbrain error code, exit code, [AGENT] block or notice block. Specific to this skill:

• gbrain doctor warns or fails: record the exact check and threshold before declaring the system unhealthy; the doctor's 24h/72h sync-freshness thresholds may differ from a cron monitor's.
• gbrain sources status shows held items or lock_busy: that is the measured cause; fix it and re-measure instead of raising timeouts.