Measurement Discipline

Measure One Thing, Then Write It Down

When invoked

Apply the rules below to the measurement or optimization task at hand: baseline before touching anything, change one variable, keep a control where the harness could drift, require a mechanism for any claimed win, and record a loss as REFUTED rather than letting it go unwritten. If the project keeps a measurement log, search it by symptom before proposing or diagnosing anything, then append the resulting entry, success or refutation, as part of finishing the task.

Use this as team practice, or paste it into an AI agent's instructions; it is written as instructions either way. It applies to anything measurable: query latency, cache hit rates, build times, bundle size, model parameters, cost per request. Its purpose is to make every optimization question cost one search instead of one rediscovery, and to make sure the answer you find was honestly obtained.

The rules

1. Baseline before you touch anything, and get the noise floor from repeated runs. One before-number and one after-number cannot tell a real effect from a slow afternoon. Discard the first run as warmup, take at least five or six more, record every one, and state the median and the spread. The spread is your noise floor; nothing smaller is a result.

2. Change one variable. When two things change together and the number moves, you have learned nothing about either, and you will be back here next quarter to learn nothing again. If you must bundle changes to ship, still measure them one at a time first.

3. Keep a control when the harness could drift. Machines get busy, caches warm, background jobs start; a control on a path your change cannot possibly touch tells you how much of the delta was the world rather than you. If the control moves as much as the candidate, your noise floor is wider than you thought, and you say so.

4. A win must beat the noise floor AND come with a mechanism. A delta without a mechanism is a coincidence you will re-litigate, because nobody, including you in six months, can tell it apart from luck. Name what changed: call count went from 1 + N to 2, the plan switched to an index scan, the allocation left the hot path. No mechanism means the experiment is not finished.

5. Product performance goals need product-level evidence. When the stated goal is user-facing speed, such as page load time or a route's production latency, a helper microbenchmark is support evidence, not acceptance by itself. Tie each experiment back to the product surface and stage it is supposed to improve. If the local helper delta is negligible while the page or route remains slow, keep investigating the next bottleneck or record why that ticket's scope cannot move the product metric. When production telemetry exists, such as Datadog RUM/APM page or route spans, record the exact query/window and the baseline or comparison values available there. If access or export is unavailable, record that as an evidence gap instead of substituting local-only numbers for production behavior.

6. A loss is recorded as REFUTED, with its conditions and a re-check trigger. Refuted work that is not written down gets paid for a second and third time by the next person with the same good idea. Record what you tried, the numbers, why it failed, and the conditions under which the answer would change. Results are tied to conditions: a 10x change in data volume, traffic, or hardware reopens them; silently trusting an old result does not.

7. The log is append-only and dated, and every number carries the command that produced it. An unreproducible number is an opinion with decimal places. Never edit a past entry to agree with a newer finding; append the newer finding and say what it supersedes. Pin anything that moves under you: input set, revision, seed, data snapshot, sample size.

8. Label projections as projections. An extrapolation that reads like a measurement will be quoted back at you as one, usually in a decision meeting. Write "projection at this measured rate" and show the rate it came from.

9. Search the log by SYMPTOM before you propose anything, and before you explain anything. Rediscovery is the expensive failure mode, and prior findings are almost always filed under the symptom that prompted them, in prose, not under the name you would give your fix. The trigger is not "am I about to change something", it is "am I about to reach a conclusion about an area that may already have one", so diagnosis counts too: following evidence is the activity most likely to re-derive a recorded finding, and one search for the symptom ends what would otherwise be twenty steps reconstructing something already written down.