Tracking Quality Trends

Tracking Quality Trends

Use this skill when quality gets reported as a number, and nobody can say whether it is better or worse than last time.

A snapshot is almost useless on its own. "Coverage is 71%" prompts an argument about whether 71 is good. "Coverage on changed lines has moved 62 → 71 over three releases, limit 75, goal 85" prompts a decision. Direction is the finding; the absolute value is context.

The second thing this catches is regression in the quality system - a threshold lowered, a job made non-blocking, a waiver renewed for the fourth time. Those never show up in a snapshot, because a snapshot reports what is measured, not what stopped being measured.

When to Use

• quality reporting is a series of unconnected numbers
• a metric is quoted with no baseline and no target
• a team needs to show a quarter of improvement, or explain a quarter without it
• a contract has been re-derived and the question is what moved
• gates are being quietly weakened and nobody has noticed
• a release decision needs direction, not just current state

Operating Principles

• Limit / current / goal, always three numbers. Limit is the threshold that triggers action; current is measured; goal is the target. A metric with only a current value cannot be acted on.
• Direction and magnitude, not just the delta. "−3pp" needs "inside the noise band" or "third consecutive fall" beside it to mean anything.
• Archive the run, do not recompute history. Store each reading with its date, commit, and how it was measured. Recomputed history changes when the method changes, and then the trend is fiction.
• A method change breaks the series. Say so, and start a new one rather than pretending the numbers are comparable.
• Track the quality system, not only the code. Thresholds, gate blocking-ness, waiver counts and ages, suppression counts. A repo whose coverage rose while its threshold fell has got worse.
• Fewer metrics, honestly measured. Six metrics a team acts on beat twenty nobody reads. Every metric needs a named owner and an action if it crosses its limit.
• Every metric carries its caveat. Coverage does not prove correctness; a rising pass rate can mean weaker tests. Report the caveat inline, not in a footnote.
• Never trend a metric that can be gamed without saying so. Coverage, test count, and defect count all move under pressure without quality moving.

Workflow

Phase 1: Fix the metric set

Start from what the contract already implies, and cap it at six to eight. From ./resources/trend-metrics.md:

Each metric needs an owner and a stated action at the limit. A metric with neither is a dashboard decoration.

Phase 2: Archive each run

.qa/
  quality-contract.md          # current
  trends.md                    # the report
  history/
    2026-05-02/{contract.md,metrics.json}
    2026-06-13/{contract.md,metrics.json}
    2026-08-21/{contract.md,metrics.json}

Each metrics.json records the value, the date, the commit, and how it was measured - the tool, the version, and the scope. That last field is what lets a future reader tell a real improvement from a method change.

Phase 3: Diff against the previous run

Per metric: previous, current, delta, direction, and position against limit and goal. Then, separately, the structural diff nobody else reports:

• practices that moved into PRESENT - real wins, name them
• practices that regressed out of PRESENT - the most important line in the report
• practices newly N/A or deferred - usually a contract edit, so check it was deliberate
• thresholds that changed, in either direction
• gates that became non-blocking
• waivers added, expired, renewed
• suppressions added and removed

A threshold quietly lowered from 75 to 60 will otherwise appear as a coverage improvement. Reading the config diff alongside the metric diff is what catches it, and it is the single highest-value habit in this skill.

Phase 4: Establish the noise band

Before reporting a movement as a finding, know what movement is normal. Flake rate and suite duration move on their own; coverage moves with the size of the release; eval scores move with model variance.

Rule: a movement inside the noise band is not a finding, and a movement in the same direction three periods running is a finding regardless of size. Slow drift is what a threshold-based alert never catches.