Audit I18n

Audit i18n Coverage

Audit the localization surface and report what is missing, in a form the user can act on and re-run later.

You are the lead. You do Phase 0 yourself with direct Bash, then hand the analysis to the audit-i18n workflow, then independently re-assert what it claims. You do not attribute keys to features, triage candidates, or write the report yourself — the workflow fans that out.

The deterministic checker is the source of truth. scripts/i18n-coverage.py is the only thing that produces findings. The workflow explains findings that already exist; it never discovers them.

The skill is audit-only. It ends at findings + a recommended fix order. It never writes a translation, never edits a strings.xml, never adds a Gradle task or CI gate, never commits, never opens a PR.

The two layers, and why they are separate

Keep the seam clean. The instant an agent re-derives findings by grepping, the audit stops being comparable to the last one and the determinism you paid for is gone. Every number in the final report traces to the JSON; the workflow's contribution is feature names, dates, verdicts and priority.

Hard rule — do not re-derive findings by hand

Never grep for missing strings, diff strings.xml files with shell tools, or count keys yourself. Every number in your report must come out of the script.

The reason is not politeness, it is that a hand-rolled pass is not comparable to the next one. The script is byte-deterministic — identical input produces identical stdout on every run and every machine (no timestamps, no absolute paths, no set-iteration order). That is what makes a report diffable against the report from six weeks ago, and what lets the user prove a fix actually shrank the debt. An ad-hoc grep produces a number nobody can reproduce, and it systematically misses the things the script handles carefully: both directions of parity drift (a module with extras that cancel against its missing keys looks fine to a count comparison), values-night being a theme qualifier and not a locale, and the Android res/ surface being a separate thing that must not be folded into the parity math.

You may read source files to explain a finding the script already reported — naming the feature a key cluster belongs to, or triaging a heuristic candidate. You may not use reading to discover findings.

Phase 0 — Sanity-check the tool

Run this first, always:

python3 scripts/i18n-coverage.py --summary; echo "exit=$?"

Confirm three things before trusting anything downstream:

1. The header line reads allowlist: config/l10n/i18n-allowlist.txt. If it reads allowlist: (none) you are running unsuppressed — see the blind spot below, this failure is silent.

2. The module table lists 10 modules and loc is 9 on every row. Fewer means discovery broke or a locale was dropped wholesale.

3. The exit code is below 16. 16 and above is a hard failure — the script could not find or parse the surface. It shares no bits with the finding mask (1|2|4|8) so that "the check broke" can never be misread as "the code is clean". If you see it, read the failure line, fix the invocation, and re-run. Do not report anything from a hard-failed run.

   An advisory (bit 8) is not a failure and must not be treated as one. It means the tool measured something it wants a human to see — a locale directory missing wholesale, a new <plurals>, a module added since EXPECTED_MODULES was last updated. Those are findings. Carry them into the report; do not abort on them.

Phase 0 is yours and is not delegated: if the tool is broken, fanning out ten agents over its output just multiplies the wrong answer.

Phase 1 — Run the workflow

Resolve these yourself first, with Bash. The date matters: workflow scripts have no clock (Date.now() throws inside one by design, so runs stay resumable), so the report date must be computed here and passed in.

REPO="$(git rev-parse --show-toplevel)"
DATE="$(date -u +%F)"
SKILL_DIR="$REPO/.claude/skills/audit-i18n"

# Never overwrite an existing dated report. Two audits on the same UTC day collide on one
# filename, and the earlier report is the baseline the later one is meant to be diffed
# against — it is uncommitted at that moment, and artifacts/ is gitignored, so an overwrite
# is unrecoverable. Pick the first free suffix instead.
REPORT_PATH="$SKILL_DIR/REPORT-$DATE.md"
n=2
while [ -e "$REPORT_PATH" ]; do
  REPORT_PATH="$SKILL_DIR/REPORT-$DATE.$n.md"
  n=$((n + 1))
done
echo "$REPO" "$DATE" "$REPORT_PATH"

Pass REPORT_PATH through verbatim — the workflow derives its artifact filenames from it, so the raw JSON behind an earlier report is preserved too. Then:

Workflow({ name: "audit-i18n", args: {
  repoRoot:     "<REPO>",
  skillDir:     "<REPO>/.claude/skills/audit-i18n",
  artifactsDir: "<REPO>/.claude/skills/audit-i18n/artifacts",
  reportPath:   "<REPORT_PATH from the snippet above — do not re-derive it>",
  reportDate:   "<DATE>",
  scope:        "full",     // or "exact-only" to skip heuristic triage
  attributeStale: true,
} })

Pass scope: "exact-only" when the user wants a fast parity-and-stubs answer, or when the heuristic candidates were triaged recently and have not moved. It drops the triage agents and leaves [H] reported as raw counts.

What it does: measures (1 agent, writes the JSON), then fans out one attribution agent per module with drift plus a stub adjudicator plus up to three heuristic triagers — all concurrently — then synthesizes one report, then reconciles that report against the raw JSON with two independent agents that did not write it. Typically 10–13 agents.

It returns { reportPath, jsonPath, totals, featureClusters, modulesAttributed, modulesNotAttributed, unreconciledModules, stubVerdictCounts, heuristicSampled, advisories, reconcileVerdicts, confirmedDiscrepanciesFixed, reconcileFailedLenses, unconfirmedDiscrepancies, clean, reportMarkdown, summary }.

No agent writes the report file — the harness refuses report-file writes from subagents. The workflow returns the reconciled report as reportMarkdown; you write it verbatim to reportPath (re-check the path is still free first) before starting Phase 2. If reportMarkdown is missing, the synthesize phase returned nothing: say so and do not hand-write a substitute.

If it returns { halted: true }, stop. That means the checker hard-failed (exit ≥ 16) — it did not run at all. Report the reason to the user and fix the tool before auditing anything. A report generated over a broken measurement reads as a clean bill of health, which is the worst possible output. Advisories alone never halt the run; they are carried into the report.

Phase 2 — Re-assert independently

Do not relay the workflow's self-report unchecked. It already reconciles itself, but it graded its own homework. Run these yourself:

Use the jsonPath the workflow returned — it is keyed to this run's report, not a fixed name.

# 1. The report's headline total must equal the JSON's — the single most load-bearing number.
python3 -c "import json,sys;d=json.load(open(sys.argv[1]));print('missing',d['parity']['missing_total'],'stale',d['parity']['stale_total'],'struct',d['parity']['structural_total'],'stubs',d['stubs']['counts']['errors'],'heuristic',d['hardcoded']['total'])" <jsonPath>
grep -nE "^\| \*\*total\*\*|debt|missing" <reportPath> | head

# 2. The audit is reproducible: re-running the checker reproduces the same JSON.
python3 scripts/i18n-coverage.py --format json | shasum -a 256
shasum -a 256 <jsonPath>

# 3. Spot-check one feature attribution against git yourself. --reverse|head -1 gives the
#    commit that INTRODUCED the key; `-1` alone gives the last commit that touched it, which
#    is a different (and usually wrong) answer. The name="…" anchor stops a prefix-nested
#    sibling key from matching.
git log --format='%h %ad %s' --date=short --reverse -S'name="<a key named in the report>"' \
  -- <module>/src/commonMain/composeResources/values/strings.xml | head -1

Then check the returned fields. clean: true is the only state you may summarize without caveats. When it is false, one of these is non-empty, and each means something specific:

Say so to the user in your summary. Those are the parts of the surface the audit did not fully cover, and they must not be presented as if they were.

If confirmedDiscrepanciesFixed is non-zero, mention it: the report needed correcting against its own source data, which is worth knowing about the run.

Invocations

All paths are repo-relative; run from the repo root.

# Full report, human-readable. The default.
python3 scripts/i18n-coverage.py

# Totals only — good for a before/after comparison or a quick status line.
python3 scripts/i18n-coverage.py --summary

# One detector at a time. Useful because the exit code then isolates that detector.
python3 scripts/i18n-coverage.py --detector parity
python3 scripts/i18n-coverage.py --detector stubs
python3 scripts/i18n-coverage.py --detector hardcoded
python3 scripts/i18n-coverage.py --detector all        # same as omitting it

# Machine-readable. Use this when you need to group, sort, or cross-tabulate findings.
python3 scripts/i18n-coverage.py --json                # shorthand for --format json
python3 scripts/i18n-coverage.py --format json
python3 scripts/i18n-coverage.py --format text         # the default

# Show the raw, unsuppressed finding set — what the allowlist is currently hiding.
python3 scripts/i18n-coverage.py --no-allowlist

# Point at a different allowlist (must resolve INSIDE the repo).
python3 scripts/i18n-coverage.py --allowlist config/l10n/i18n-allowlist.txt

# Two opt-in low-precision stub tiers, both INFO, both off by default.
python3 scripts/i18n-coverage.py --include-latin       # Latin-locale values identical to base
python3 scripts/i18n-coverage.py --near-identical      # case/punctuation-only differences

# Override repo-root detection (normally found by walking up for settings.gradle.kts).
python3 scripts/i18n-coverage.py --repo-root .

A useful pattern for grouping missing keys by feature — the highest-value analysis step — is to pull the distinct key lists out of the JSON rather than the text report:

python3 scripts/i18n-coverage.py --json > /tmp/i18n.json
python3 -c "
import json
d = json.load(open('/tmp/i18n.json'))
for m in d['parity']['per_module']:
    if m['missing_total']:
        print('===', m['module'], 'base', m['base_keys'], 'rows', m['missing_total'])
        for k in m['distinct_missing_keys']: print('  ', k)
"

Exit codes

A bitmask, so a caller can tell exact findings from heuristic ones:

Codes combine: 7 = parity + stubs + hardcoded, which is the current state of develop.

Test hard failure with exit & ~15 (equivalently exit >= 16), never with a narrower mask. The hard-failure code shares no bits with the finding mask, and that is the whole point — the conventional 70 (0b1000110) sets both the stub bit and the hardcoded bit, so exit & 3 would report a crashed run as "2 — exact stub findings". Anything at or above 16 means the check did not run; anything below is a finding set.

A caller wanting proven defects only would check exit & 3 after ruling out hard failure. Do not treat that as an invitation to add a gate — that is explicitly out of scope for this skill.

Reading the three detectors

The single most important thing you do is keep these three straight in the report. Two are exact set operations. One is a regex heuristic over a language with no type-level marking of user-facing strings. Blurring them destroys the report's credibility.

[P] parity — EXACT. Report as fact.

Set difference between each module's base values/strings.xml and each of its values-<loc>/ files, computed in both directions:

• missing — in base, not in the locale. The user sees English in a translated app.
• stale — in the locale, not in base. Dead weight; usually a key the base renamed or removed, which the locale files were never updated to follow.
• structural — an entirely absent locale file (scored as 100% missing, not skipped), duplicate keys within one file, unknown locale qualifiers.

Also reported: uniform, meaning every locale in the module shares one identical missing set and one identical stale set. When uniform is yes, the per-module distinct key list is lossless and you can quote it directly. When it is no, per-locale sets diverge and you must go to the JSON parity.findings rows before making per-locale claims.

Note that missing_total is a row count summed over locales, not a key count. 504 missing rows in a 9-locale module is 56 distinct keys. State both in the report; the user sizes work in keys and sizes debt in rows.

Parity is never allowlist-suppressible. A missing or stale key is not a judgment call.

[S] stubs — EXACT. Report as fact.

A key that is present in a non-Latin-script locale but whose value is byte-identical to the English base and pure ASCII was never actually translated. The file passes parity and still ships English.

Two classes are exempted mechanically, with no allowlist entry needed: shared literals (a single token no locale anywhere translated — Bearer, OAuth, JSON, MCP, SSE, Markdown, mermaid, shadcn/ui) and letterless values (arrows, em dashes, pure format strings).

What survives is tiered by cross-locale corroboration:

• ERROR — at least 4 other locales translated this key. The holdout is a genuine miss.
• REVIEW — only 0–3 others did, so English may be the local convention. Report these separately and say so. 0 means multi-word English that no locale translated: usually a section every translator skipped, occasionally an example value that belongs in the allowlist. A single untranslated WORD that no locale translated (Off, Never) is still exempted as a shared literal — indistinguishable from OAuth mechanically — so only the contiguous-run corroborator can surface it.
• INFO (--include-latin, --near-identical) — off by default and near-100% / 1-in-3 precision respectively. Only surface these if the user asks for exhaustive output, and label them INFO.

Also reported: contiguous runs of ≥3 adjacent untranslated keys in one file. These are the cheapest fixes in the whole report — one localized block of a file, usually a whole form that a translator skipped in one go. Call them out by file:first_line-last_line.

[H] hardcoded — HEURISTIC CANDIDATES. Never report as proven.

Sink-anchored regex matching for English literals in .kt that never reached a strings.xml — Text("…"), contentDescription = "…", ?: "…" error fallbacks, when branches returning display strings, and so on, bucketed into rule families.

Language rules for this section, non-negotiable:

• Call them candidates. Never "missing strings", never "confirmed gaps", never fold their count into the translation-debt total.
• Print the script's own precision estimate and its enumerated false-positive class verbatim rather than paraphrasing it. The script knows which of its rows are wrong; you don't.
• Say explicitly that each one needs a human to decide whether it is user-facing at all.

Triage priority, best-value first:

1. content_desc and text_call — unambiguously on screen, and content_desc is also an accessibility defect. Smallest families, highest hit rate.
2. ui_param — labels/descriptions passed to UI builders. Large, but usually dominated by a few registry-shaped files where one file is one fix.
3. when_branch and continuation — display-string mappings and multi-line concatenations.
4. state_error and elvis_error — error fallbacks. Mixed: real user-facing error copy sits next to developer diagnostics that merely leak.
5. enum_label and sink_call — mostly wire tokens and non-UI helpers. Verify before touching.

Sort candidates by module and by file, not by rule, when recommending work: a single file with 100 rows is one PR, and 100 rows spread over 40 files is not.