Assessing Comprehension Debt

Assessing Comprehension Debt

Use this skill when code is arriving faster than anyone is reading it, and every other quality signal still looks fine.

Comprehension debt is the gap between the code that exists and the code the team actually understands. Tests pass, coverage holds, lint is clean - and nobody can explain why the pricing module works. When an agent writes the implementation and the tests, both signals go green without a single human forming a mental model. That is the debt this skill makes visible.

It is advisory, permanently, on purpose. Understanding lives in people's heads and cannot be proven by any signal. What can be measured is the risk of comprehension debt and the absence of evidence of understanding. A gate that claims to measure understanding is lying, and once someone notices, every finding it ever produced loses credibility.

When to Use

• an AI-assisted or agent-generated codebase is growing faster than the team reads it
• reviews are fast, approvals are frequent, and nobody asks questions
• a module ships weekly and no one volunteers to explain it
• a bus-factor conversation needs evidence instead of anecdote
• a quarterly quality review should cover more than test coverage
• onboarding is slow in a codebase that looks well-tested

Operating Principles

• Never blocks. Findings cap below the project's blocking threshold. State that ceiling in the report, every time.
• Measure absence of evidence, not understanding. Say it in those words. The band is a proxy; treat it as a prompt for a conversation, not a score.
• The risk band is directional, not precise. It says "this change is the shape of one nobody understands", not "nobody understands this change".
• AI provenance is a signal, never a penalty. The useful figure is AI-authored and unattested. AI-authored and well-understood is a good outcome, and if trailers are not in use the honest answer is "unknown".
• A human explanation lowers the band. An Intent: trailer or an ADR is evidence that somebody thought about it. That is exactly the behaviour to reward.
• Teach-back beats approval. "I approve" is a click. "I can explain what happens when the provider retries this webhook" is comprehension.
• Never blame individuals. A high band on a module is a system outcome: review load, delivery pressure, tooling. Naming people converts a useful signal into something nobody will run twice.
• Trend over snapshot. One reading is noise. Direction over a quarter is the finding.

Workflow

Phase 1: Establish scope and whether provenance exists

Pick the range: a release, a quarter, a module's history. Then check what signals are available:

git log <range> --format='%h %an %s'
git log <range> --format='%(trailers:key=Intent,valueonly)'
git log <range> --format='%(trailers:key=Assisted-by,valueonly)'
git log <range> --format='%(trailers:key=Comprehension-Attested-by,valueonly)'
git log <range> --numstat

If Assisted-by: trailers are not in use, provenance is unknown - report it as unknown and move on. Do not infer AI authorship from commit size, style, or timing; those inferences are wrong often enough to poison the whole report.

Phase 2: Compute the risk band per change or module

Three inputs, from ./resources/risk-band-rubric.md:

Bands: high (large, branch-heavy, no explanation), medium (one of the three), low (small, or well explained, or both).

A high band means: if nobody understands this, we would not be able to tell. That is all it means, and saying so plainly is what keeps the metric usable.

Phase 3: Check teach-back attestation on high-risk surface

For changes on high-risk surface (recording-change-intent has the rules), look for a record that a named human can explain it:

• a Comprehension-Attested-by: <name> trailer, author-side or reviewer-side
• an entry in the attestation register (attesting-manual-verification)

Status is FULL (attested), NONE (high-risk surface, no record), or N/A (not high-risk surface).

Where a record is missing and it matters, run the teach-back in ./resources/teach-back-protocol.md: four questions, ten minutes, and the answers tell you more than the band ever will. The protocol's value is not the record - it is that the conversation happens.

Phase 4: Report the picture

• Risk bands by module, with the inputs that produced each one
• Attestation status on high-risk surface, and the specific changes with none
• Provenance, if known: AI-authored share, and the share that is AI-authored and unattested
• Trend against the previous assessment, per module
• Concentration - comprehension debt clusters. A module where one person authored everything and nobody reviewed it is the finding, regardless of any band
• The ceiling, stated explicitly: these findings are advisory and do not fail the build

Then the only recommendations worth making: which specific modules deserve a teach-back session, which deserve a walkthrough written down, and which deserve a second pair of eyes on the next change.