Fix Dependabot

Fix Dependabot Alerts

Resolve open Dependabot security alerts by upgrading vulnerable dependencies across the ETL monorepo — including pip packages (pyproject.toml + uv.lock), lib/ subdirectories, and npm packages in vscode_extensions/.

Goal: leave zero open Dependabot PRs. Security alerts are the priority, but Dependabot also opens routine version-update PRs that have no associated alert (e.g. esbuild minor bumps, or js-yaml bumps in extensions GitHub didn't flag). Don't stop at the alerts — also clear these (Step 4c) so the PR list is empty afterward. The default is to fold every open Dependabot PR's bump into the single batch PR and close the originals; if the user only wants the security alerts, scope to those instead.

Step 1: Fetch and summarize alerts

gh api repos/owid/etl/dependabot/alerts \
  --jq '.[] | select(.state == "open") | {
    number, state,
    package: .dependency.package.name,
    ecosystem: .dependency.package.ecosystem,
    severity: .security_advisory.severity,
    summary: .security_advisory.summary,
    patched: .security_advisory.vulnerabilities[0].first_patched_version.identifier,
    manifest: .dependency.manifest_path
  }'

Present a summary table grouped by severity (critical > high > medium > low) with counts, then the detailed list showing package, ecosystem, severity, and summary.

Ask the user which severities to fix (e.g. "critical and high only" or "all"). If the user already specified a filter in their request, proceed with that.

Step 1b: Audit existing Dependabot PRs

Always list open Dependabot PRs, not just open alerts. Dependabot PRs can remain open after the default branch already contains the requested dependency version (especially old/conflicted PRs where automatic rebases have been disabled).

gh pr list --repo owid/etl --author app/dependabot --state open \
  --json number,title,headRefName,mergeStateStatus,files --limit 100 \
  --jq '.[] | {number,title,mergeStateStatus,files:[.files[].path]}'

For each open PR:

1. Identify the manifest/lockfile and target package/version from the title and changed files.
2. Compare against the current default branch (origin/master or master), not just your working branch. Check whether the dependency is already at the requested version or newer.
   • For uv.lock files, inspect the relevant package entry in the affected lockfile.
   • For npm lockfiles, parse package-lock.json and check all installed versions of the package in that extension.
3. If the PR is obsolete, close it with a clear comment, for example:

   gh pr close <number> --repo owid/etl --comment \
     "Closing as obsolete: the current default branch already has this dependency at the requested version or newer, so this stale Dependabot PR is no longer relevant."

4. If you create a replacement PR that batches or supersedes Dependabot PRs, close the superseded PRs and reference the replacement PR in the close comment.
5. Re-run the PR list and confirm there are no open irrelevant Dependabot PRs before reporting completion.

Map each open PR to an alert (or not). Cross-reference this PR list against the alert list from Step 1. A PR whose package/manifest matches an open alert is a security PR — handled by Steps 2–4. A PR with no matching open alert is a routine version-update PR — handled by Step 4c. Both kinds should be gone by the end.

Step 2: Categorize each alert

For each alert to fix, determine:

1. Ecosystem: pip or npm
2. Direct vs transitive: Is the package listed directly in pyproject.toml (or a lib/*/pyproject.toml), or is it only in the lockfile as a transitive dependency?
3. Manifest location: Which pyproject.toml or package.json owns this dependency?

For pip packages, check all pyproject.toml files:

grep -rn '<package-name>' pyproject.toml lib/*/pyproject.toml

For npm packages, check the manifest path from the alert (usually under vscode_extensions/).

Step 3: Fix pip dependencies

Direct dependencies

If the package appears in a pyproject.toml dependencies or optional dependency group:

1. Update the version constraint to require at least the patched version:

   "package>=old_version"  →  "package>=patched_version"

2. Run uv lock --upgrade-package <package> to update the lockfile

Transitive dependencies

If the package is only in uv.lock (not a direct dependency):

1. Identify which direct dependency pulls it in:

   uv tree --invert --package <vulnerable-package> --depth 1

2. Check if any of those parent packages cap the vulnerable package below the patched version. For each parent, check its constraints on PyPI:

   python3 -c "
   import json, urllib.request
   r = urllib.request.urlopen('https://pypi.org/pypi/<parent>/json')
   d = json.loads(r.read())
   deps = d['info'].get('requires_dist') or []
   matches = [dep for dep in deps if '<vulnerable-pkg>' in dep.lower()]
   print(matches)
   "

3. If no cap blocks it: simply run uv lock --upgrade-package <vulnerable-package>
4. If a cap blocks it: check if a newer version of the parent package relaxes the cap. If it does, upgrade the parent. If the latest parent still caps it:
   • Check if the parent package is actually used in the codebase (grep -rn 'import <parent>\|from <parent>' --include='*.py')
   • If unused: remove it from pyproject.toml (this unblocks the transitive dep)
   • If used: inform the user that the fix requires either waiting for the parent to update, or adding a [tool.uv] override (explain the tradeoff — overrides can cause runtime incompatibilities)

After removing a dependency

When removing a package, check if it's imported anywhere in the codebase. If it is, replace its usage with an alternative approach. For example, when we removed moviepy, we replaced its ImageSequenceClip usage with a direct ffmpeg subprocess call.

Always search broadly:

rg 'import <package>|from <package>' --type py

Step 4: Fix npm dependencies

For each vulnerable npm package:

1. Navigate to the directory containing the affected package.json
2. Update the package:

   cd <directory>
   npm install  # ensure node_modules exist
   npm install <package>@^<patched-version>

3. Verify the update:

   npm ls <package>

npm transitive vs. direct deps

Before adding a package to dependencies, check whether the extension actually imports it:

rg -l "from ['\"]<package>|require\(['\"]<package>" vscode_extensions/<ext>/src

• Imported in src/ → it's a real runtime dependency; bump it in dependencies (npm install <package>@^<version>).
• Not imported (it's a dev-tooling transitive dep, e.g. pulled in by eslint/mocha) → do not add it to dependencies (that creates a phantom runtime dep). Instead force the patched version via the existing overrides block in package.json, then npm install. This matches how Dependabot itself fixes those (lockfile-only).