CLI Audit

cli-audit

Audit a CLI that already exists against the rules in cli-build. The output is a findings list with evidence, not a score.

cli-build builds. This reads what was built and says where it drifted from what it claims.

Why there is no score

The founding case of cli-build's human-output.md is a CLI that passed every mechanical gate and was unreadable: 275 rows for someone who asked what was playing that evening, a percentage that read backwards, one title repeated fourteen times. No gate failed.

A number over that CLI would have read as healthy. That is the exact failure this skill exists to avoid, so it produces no number. A run yields three streams and they never merge:

A clean deterministic run is not a clean CLI. Say that in the report every time, because the reader will otherwise take the passes as the answer.

Weighting these into one number needs thresholds calibrated against a distribution, and no such distribution has been measured. Inventing weights here would repeat the mistake human-output.md names about scale cutoffs: instinct-picked buckets that never discriminate. When enough CLIs have been audited to know the real spread, that is when a score becomes possible, not before.

0. Resolve the conditional gates first

Most checks in cli-build are conditional. Running all of them against every CLI reports absences as defects when they were deliberate decisions, which trains the reader to skim the findings. Resolve these before running anything, and write each answer into the report:

1. Does any command carry consequence? Money, irreversible writes, third-party side effects, personal data. If no: the entire trust-ladder group (audit log, --dry-run, killswitch, intent tokens, consent gates) is NA, and its absence is compliant. cli-build Phase 4 says so explicitly, and says the decision should be recorded rather than skipped. Look for that record before calling it an omission.
2. Is there a human-facing surface? If the user scoped the CLI machine-only, every judgment check from human-output.md is void.
3. What is the distribution target? Source shebang, npm, or native binary. The shebang and build checks only mean something against a declared target.
4. Is the API asynchronous? Gates --wait and the job ledger.
5. Is it published? Registry drift, zero-tests, and installer-shebang checks only bite once someone can install it.
6. Contract origin: discovered, defined, or mixed? Decides what the JSON contract is even measured against.

Everything sourced from compounding-surface.md (--wait, vocabulary in CI, --deliver, feedback) is labeled in cli-build itself as coming from published work rather than the measured corpus. Report a gap there as a gap against a recommendation, and say which kind it is.

Done when: all six are answered in writing, and the check groups they switch off are named.

1. Run the deterministic layer

scripts/audit.sh --repo <package-root> [--bin <installed-command>] [--json]

Without --bin only the static checks run and every runtime check reports SKIP. That is a materially weaker audit: cli-build Phase 6 is explicit that running a source file verifies a file while running the installed name verifies what ships — the bin entry, the shebang, the resolved dependencies, the banner. Link it first:

cd <package-root> && bun link   # or npm link
scripts/audit.sh --repo . --bin <name>

Read references/checks.md for what each id means and where in cli-build it comes from. Every id traces to a line in the source skill; none were invented here.

A FAIL is a fact, not yet a finding. It becomes a finding when you have confirmed it applies given step 0. A FAIL on home-override in a CLI with no tests is noise.

Verify every claim of absence before it reaches the report. The first run of this skill produced four false absences out of five instrument bugs: an audit log with 41 call sites reported as "no audit log in this CLI", twice, under two different name guesses. A FAIL saying a feature is missing is the one output a reader cannot check cheaply, so it is the one that must be checked before it ships. Open the file and look.

The rule that came out of it: a check that resolves a feature by guessing its function names will eventually announce that the feature does not exist. Resolve the module, read the names it exports, then look for those names elsewhere. An export list cannot be wrong about itself.

Run the script against a second, unrelated CLI whenever you change it. Three of those five bugs were invisible against the CLI the check was written for and appeared immediately against another.

Done when: every non-PASS line is either promoted to a finding or dismissed with the conditional gate that excuses it, and every claim of absence has been confirmed by reading the code.

1b. Measure coverage, before and after

scripts/coverage.sh --bin <name> --max-depth 4

This walks the CLI's own --help tree and reports how many leaf commands exist. Run it before the judgment layer so you know the size of the job, and again at the end with the list of what you actually ran:

scripts/coverage.sh --bin <name> --max-depth 4 --exercised exercised.txt
# Coverage of sunat-cli: 27 of 80 leaf commands exercised (33%)

exercised.txt holds one command per line as invoked, without the binary name. Entries matching no command in the surface are counted separately and reported, because a stale note silently inflating coverage is the failure this artifact exists to prevent.

The ratio goes in the report. An audit of four commands and an audit of sixty produce prose that reads identically; this session's first pass covered 4 of 80 and the gap surfaced only because the reader asked. A number the report cannot omit turns partial coverage from a disclaimer into a fact.

Low coverage is often correct — most of what goes unexercised in a consequential CLI is the write path, and not running it is the right call. Say which it is: deliberate, or not reached.

Done when: the surface size is known before judging, and the final ratio plus the not-exercised list are in the report.