Check Metadata Typos

Check Metadata Typos

Check metadata files for spelling typos using comprehensive spell checking.

Scope Options

Ask the user which scope they want to check:

1. Current step only - Ask the user to specify the step path (e.g., etl/steps/data/garden/energy/2025-06-27/electricity_mix)
2. All ETL metadata - Check all active .meta.yml files in etl/steps/data/{garden,meadow,grapher}/ (automatically excludes ~3,570 archived steps)
3. Snapshot metadata - Check all snapshot .dvc files in snapshots/ (~7,915 files)
4. All metadata - Check both ETL steps and snapshot metadata files

Note: Archived steps and snapshots (defined in dag/archive/*.yml) are automatically excluded from checking as they are no longer actively maintained.

Implementation Strategy

0. Check codespell installation

IMPORTANT: Check if codespell is installed before attempting to use it. Since codespell is now a dev dependency in the project, it should already be installed, but verify first to avoid reinstalling unnecessarily.

# Check if codespell is installed
if ! .venv/bin/codespell --version &> /dev/null; then
    echo "codespell not found, installing..."
    uv add --dev codespell
else
    echo "codespell is already installed"
fi

If codespell is not installed and uv add --dev codespell fails, explain to the user how to install it manually.

1. Exclude archived steps and snapshots

IMPORTANT: Do not check archived steps and snapshots as they are no longer in use.

Archived steps and snapshots are defined in dag/archive/*.yml files:

• ~3,570 deprecated steps (garden, meadow, grapher)
• ~736 deprecated snapshots

To exclude them, extract their paths and create a list of active files:

# Extract archived step paths to a file
for step_type in garden meadow grapher; do
  grep -h "data://${step_type}/" dag/archive/*.yml 2>/dev/null | \
    grep -o "data://${step_type}/[^:]*" | \
    sed 's|data://|etl/steps/data/|' | \
    sed 's|$|.meta.yml|'
done > /tmp/archived_files.txt

# Extract archived snapshots
grep -rh "snapshot://" dag/archive/*.yml 2>/dev/null | \
  grep -o "snapshot://[^:]*" | \
  sed 's|snapshot://|snapshots/|' | \
  sed 's|$|.dvc|' | \
  sort -u >> /tmp/archived_files.txt

# Create list of all metadata files
find etl/steps/data/garden -name "*.meta.yml" > /tmp/all_meta_files.txt
find etl/steps/data/meadow -name "*.meta.yml" >> /tmp/all_meta_files.txt
find etl/steps/data/grapher -name "*.meta.yml" >> /tmp/all_meta_files.txt
find snapshots -name "*.dvc" >> /tmp/all_meta_files.txt

# Filter out archived files
grep -vFf /tmp/archived_files.txt /tmp/all_meta_files.txt > /tmp/active_meta_files.txt

echo "Total files to check: $(wc -l < /tmp/active_meta_files.txt)"

2. Run codespell with ignore list and exclusions

Use the existing .codespell-ignore.txt file to filter out domain-specific terms.

Whichever option the user picked, write that scope's file list to /tmp/codespell_targets.txt and check it. Steps 5 and 6 reuse that one file, so the fix and the verification cannot act on a wider scope than the check did. Never point a later step at a different list.

For option 1 (current step only):

Ask the user for the path, then normalize it. This scope has to accept three shapes, because /create-snapshot step 5 calls it on a single new snapshot file:

• an ETL step directory (etl/steps/data/garden/energy/2025-06-27/electricity_mix) → every .meta.yml in it
• a snapshot stem with no extension (snapshots/who/2026-04-22/mortality) → the matching .dvc
• a single file, .meta.yml or .dvc → itself

# For specific step or single file (option 1)
TARGET="<user_provided_path>"

.venv/bin/python - "$TARGET" > /tmp/codespell_targets.txt <<'PY'
import glob, os, sys

target = sys.argv[1].rstrip("/")
if os.path.isdir(target):
    paths = sorted(glob.glob(f"{target}/*.meta.yml")) + sorted(glob.glob(f"{target}/*.dvc"))
elif os.path.isfile(target):
    paths = [target]
else:
    # A snapshot stem: snapshots/<namespace>/<version>/<short_name>, extension unknown.
    paths = sorted(glob.glob(f"{target}.*.dvc")) or sorted(glob.glob(f"{target}.dvc"))
print("\n".join(paths))
PY

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Two ways this scope silently checks nothing, both of which must be treated as an error rather than a clean result:

• A .dvc target matched by a *.meta.yml-only pattern. A snapshot path has no .meta.yml in it, so a .meta.yml-only search returns an empty list and codespell reports no typos on a file it never opened. The normalizer above exists for exactly this case.
• An empty list from a shell glob. Don't build the list with ls <path>/*.meta.yml: under zsh a glob matching nothing aborts the command before the redirect, so no file is written and a wider list left over from an earlier run gets reused.

Always report the number of files in /tmp/codespell_targets.txt before running codespell. If it is 0, say the path matched nothing and ask for a corrected one — never fall through to a wider scope, and never report "no typos found".

For option 2 (all ETL metadata - garden, meadow, grapher):

# For all ETL step metadata (option 2)
find etl/steps/data/garden -name "*.meta.yml" > /tmp/all_step_files.txt
find etl/steps/data/meadow -name "*.meta.yml" >> /tmp/all_step_files.txt
find etl/steps/data/grapher -name "*.meta.yml" >> /tmp/all_step_files.txt
grep -vFf /tmp/archived_files.txt /tmp/all_step_files.txt > /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Note: Excluding archived steps reduces the scope by ~3,570 files and focuses on actively maintained metadata.

For option 3 (snapshot metadata):

# For all snapshot metadata (option 3)
find snapshots -name "*.dvc" > /tmp/all_snapshot_files.txt
grep -vFf /tmp/archived_files.txt /tmp/all_snapshot_files.txt > /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt

Note: the prose worth checking in a snapshot .dvc lives under meta.origin — description, description_snapshot, title, title_snapshot, citation_full, attribution. Older files instead use the deprecated meta.source.description / meta.source.published_by; both shapes are still in the repo (~2,000 files on origin, ~6,000 on source). codespell reads the whole file either way, so no filtering is needed — but report hits by their real field path, and don't "fix" a typo by migrating a file from source to origin (that's a separate change, out of scope here). ~736 archived snapshots are excluded.

For option 4 (all metadata):

# For all metadata - ETL and snapshots (option 4)
# Use the active_meta_files.txt created in step 1
cp /tmp/active_meta_files.txt /tmp/codespell_targets.txt

cat /tmp/codespell_targets.txt | xargs .venv/bin/codespell \
  --ignore-words=.codespell-ignore.txt