Update Lexicon

Updating a lexicon on woogles

This covers a new edition of a lexicon whose alphabet woogles already supports. A brand-new alphabet or letter distribution (for example Slovene in PR #1952 and #1955) is a different, multi-repo job. See reference/new-alphabet.md and stop to plan it with the user before touching anything.

Work carefully. The repo usually has unrelated untracked files, and the old edition's files are still needed by existing games. Several steps affect production and must never be done without explicit approval.

Throughout, OLD is the edition being replaced (e.g. NSF25), NEW is the new one (e.g. NSF26), and WASM=liwords-ui/public/wasm/2024. data/lexica/gaddag is a symlink to WASM, so local dev and CI read the same files.

0. Pin down the inputs

Work these out before building, and confirm them with the user in one message:

Earlier editions were not always filtered. NSF25.kwg contains 1-letter words and words up to 33 letters, so a big "fewer words" number compared with OLD's raw count is expected and not a problem. The diff below only compares words inside the length window.

1. Build and verify the word graphs

cd ~/code/kwgc && make   # only if ./kwgc is missing or stale
python3 .claude/skills/update-lexicon/scripts/build_lexicon.py \
  --input <wordlist> --name NEW --alphabet <alphabet> \
  --prev $WASM/OLD.kwg --max-len <15|21> \
  --out <scratchpad>/NEW

The script normalizes the list: NFC, uppercase, first column only, duplicates removed. It stops if any character is outside the alphabet, drops words outside the length window, and builds NEW.kwg and NEW.kad with kwgc. It then reads them back and checks them exactly:

• DAWG half of the kwg = the kept word set
• .kad = the set of per-word sorted-tile anagram keys
• GADDAG half of the kwg = every reverse(prefix)?suffix entry for every word

It also writes report.json, dropped.txt, added.txt and removed.txt (diff against OLD inside the length window). kwgc exits 0 even when it fails, so rely on the script's checks, never on kwgc's exit code.

Show the user a summary and wait for an OK before going on. Include: raw / kept / dropped (short, long, invalid) counts, the 1-letter words dropped, the blank-only word count (words using zero-count tiles such as Q/X/Z/Ä/Ö/Ü in Norwegian are valid and stay in), added and removed counts with examples, and the file sizes next to OLD's. Stop and ask if:

• any check failed
• --drop-invalid was needed
• removed > ~2% of OLD, or added is far outside normal edition-to-edition churn
• the kwg is much bigger or smaller than OLD's

2. Branch

Start from an up-to-date master on a new branch: git fetch && git checkout -b NEW-lexicon origin/master && git branch --unset-upstream. Without the unset, the branch tracks master. The working tree usually has unrelated untracked files (plans, binaries, test JSON). Never git add -A or git add .. Stage paths explicitly.

3. Put the files in place

Copy into $WASM: NEW.kwg, NEW.kad, NEW.klv2 (from OLD.klv2 or the supplied file), and super-NEW.klv2 if OLD has a super table. Check with md5 that the copies match their sources.

Never delete, rename or overwrite OLD's files. Existing games, annotated games, the analyzer, and macondo's download-on-demand (it fetches from $WASM on GitHub master) all load lexica by name.

4. Wire NEW into liwords

Grep for every mention first: git grep -n 'OLD' -- ':!*.kwg' ':!*.kad' ':!*.klv2' ':!*_pb*'. The usual edits (compare with commit ddb1199d for NSF25 and PR #1381 for CSW24):

• pkg/entity/lexica.go: in AllowedNewGameLexica, replace OLD with NEW. This list is the only thing stopping new games on OLD.
• liwords-ui/src/shared/lexica.ts: rename the AllLexica entry's key, code, and matchName if it's versioned. Update shortDescription, and the longDescription copyright year if it has one. Do not touch lexiconCodeToInternalRatingName or InternalRatingNameToProfileRatingName: ratings are keyed by family (NSF* → NSF21), so players keep their rating across editions. pkg/entity/ratings.go transformLexiconName is the same, so leave it alone.
• liwords-ui/src/shared/lexicon_display.tsx:
  • lexiconOrder: replace OLD with NEW.
  • puzzleLexica: replace OLD if it's there.
  • historicalLexica: add OLD, so annotated games and GCG imports can still use it.
• liwords-ui/src/wasm/loader.ts: add NEW.kad, NEW.klv2, NEW.kwg (and super-NEW.klv2) in sorted position. Keep OLD's entries. The browser loader needs a klv2 next to every kwg.
• liwords-ui/src/lobby/seek_form.tsx: the stored-form migration switch should send OLD and every older edition to NEW. Merge them into one case group; don't just add a new case, or older editions stay pointed at OLD.
• liwords-ui/src/App.tsx puzzleLexicon migration: same, if the family is a puzzle lexicon.
• English only: the definition fallbacks in liwords-ui/src/utils/hooks/definitions.tsx, and any defaults like pkg/league/season_start.go, liwords-ui/src/leagues/admin.tsx, pkg/broadcasts/obs.go, db/migrations/*broadcasts* defaults, and sought_game_interactions.test.ts.
• Anything else the grep turns up. For each hit, decide whether it means "the current edition" (update it) or "that exact edition" (leave it). Test fixtures, GCGs and CGPs almost always mean the exact edition.

Do not regenerate protos, bump go.mod, or touch letter distributions for a version update.

5. Tests that pin the new files

Add a test in pkg/cwgame/lexicon_test.go next to TestCSW15IsNotALaterEdition / TestNSF26IsNotAnEarlierEdition, using the lexiconHasWord helper there. It asserts that a few words from added.txt are present and a few from removed.txt are absent. Pick 2–3 of each, 2–15 tiles long, using only non-zero-count tiles. Also assert a dropped 1-letter and a >max-len word are absent, and check the reverse against OLD. This catches a wrong or mislabeled file being copied in.

In pkg/entity/lexica_test.go, add OLD to the historical list in TestHistoricalLexicaCannotStartGames.