Map Explorer To Mdim

Map an explorer's views to MDIM views (redirect proposal)

When an explorer is being retired in favour of one or more MDIMs, every explorer view needs a redirect to the equivalent MDIM view. This skill produces the input for that: a CSV of explorer views, a CSV per target MDIM, and a joint proposal mapping each explorer view to a target MDIM view (the suggestion is for human review).

The mapping itself is explorer-specific (how the explorer's dimensions translate to MDIM dimension slugs, and — when there are multiple MDIMs — which MDIM each view routes to). The skill automates everything mechanical (pulling views, the join, the shared-target accounting, validation) and leaves only the per-explorer rules for you to write, seeded with auto-suggested matches.

Inputs

• Explorer slug — matches explorers.slug in the grapher DB (e.g. natural-disasters).
• One or more MDIM catalogPaths — as stored in multi_dim_data_pages.catalogPath, e.g. natural_disasters/latest/deaths#deaths. The MDIMs must be published in the DB you connect to (their fully-expanded views are read from multi_dim_data_pages.config).

DB access (confirm this before running)

Both the explorer and the MDIMs are read from the grapher DB via OWID_ENV, so the scripts only work where that DB actually contains both the explorer and the published MDIMs. There are three ways to point OWID_ENV at such a DB — figure out which one applies before running, and don't assume .env.prod exists:

1. Staging branch (often easiest): if you're on a staging-site-<branch> branch, OWID_ENV already points at that prod-clone DB — run the commands as-is, no prefix.
2. Production, read-only, via an env file: prefix commands with ENV_FILE=<prod creds> DATA_API_ENV=production. Don't assume the file name — check what exists (ls -la .env*); on some machines it is .env.prod, on others .env.live.
3. Some other credentials file: the user may keep prod (or other) DB creds in a different env file — run with ENV_FILE=<their file> [DATA_API_ENV=production].

Preflight — check, then ask if needed:

ls -la .env* 2>/dev/null          # which credentials files exist?

# Connectivity test (swap the ENV_FILE prefix for whatever applies; drop it on a staging branch):
ENV_FILE=<prod creds> DATA_API_ENV=production .venv/bin/python -c \
  "from etl.config import OWID_ENV; print('DB OK:', OWID_ENV.read_sql('SELECT 1 AS x').iloc[0,0])"

If no prod credentials file exists and you're not on a staging branch with the data, stop and ask the user which credentials / env file to use (e.g. "which env file holds DB credentials that can reach the explorer + MDIMs? Or should I run this from a staging branch?"). Then use that file as the ENV_FILE= prefix for both script invocations below. Don't hardcode credentials.

If the connection works but a query returns nothing, the scripts stop with a clear message (explorer slug not found, or MDIM not published in this DB) — that means the DB you reached doesn't have it, so re-check which DB you're pointed at.

Workflow

1. Extract views + scaffold

.venv/bin/python .claude/skills/map-explorer-to-mdim/scripts/extract_views.py \
  --explorer <slug> \
  --mdim <ns/v/short#short> [--mdim <ns/v/short#short> ...] \
  --out ai/<slug>-mdim-mapping

Writes into the out folder:

• explorer_views.csv — id (1..N) + dimension_1..M (explorer display values).
• multidim_<short>_views.csv — one per MDIM; id is letter-prefixed by --mdim order (A1…, B1…, C1…) so ids are unique across MDIMs; columns are the MDIM dimension slugs.
• _scaffold.md — the explorer dimension legend (which dimension_i is which name), the distinct values per dimension, each MDIM's dims/choices, auto-suggested value matches (where a slugified explorer value equals a real MDIM choice slug), and a ready-to-edit mapping_rules.py template.
• _sources.json — machine-readable record of the explorer slug + dimension names and each MDIM's short/prefix/catalogPath/dim-slugs. Consumed by build_mapping.py to emit mapping.json (below); don't hand-edit it.

2. Write mapping_rules.py

Open _scaffold.md, then write ai/<slug>-mdim-mapping/mapping_rules.py defining:

• EXPLORER_DIMENSIONS — list naming dimension_1..N (copy from the scaffold; keep order).
• MDIMS — MDIM short names in the same order as --mdim (= prefixes A, B, C, …).
• route(dims) -> str — given a view's {dimension name: value}, return the target MDIM short name. For a single MDIM this is just return "<short>". For several, it's a decision on some explorer dimension (e.g. natural-disasters routes on Impact: Deaths→deaths, Economic damages (% GDP)→economic_damages, the rest→affected).
• translate(dims, mdim) -> dict — return {mdim_dim_slug: choice_slug} for the target MDIM view, built from the *_MAP dicts. Only include slugs the MDIM actually has (e.g. economic_damages has no metric — single-choice dims are pruned from MDIM views).
• (optional) DEFAULT_MDIM = "<short>" — the catch-all target for the bare explorer URL (see mapping.json → catchAll). Omit it and the best-fitting MDIM is chosen automatically (the one receiving the most resolved views; tie-break = earliest in MDIMS). Set it only when the automatic pick isn't the MDIM you'd want a param-less explorer link to land on.

The scaffold seeds the *_MAP dicts with slugify(value) guesses. Verify every entry — slugify won't catch label↔slug differences like Decadal average→decadal, Injuries→injured, Volcanoes→volcanic_activity, or aggregate collapses like All disasters/All disasters (by type)→all_stacked.

3. Build the proposal

.venv/bin/python .claude/skills/map-explorer-to-mdim/scripts/build_mapping.py --out ai/<slug>-mdim-mapping

Writes mapping_proposal.csv, one row per explorer view:

It also writes mapping.json — the machine record — and admin_bulk_payload.json, which is what you actually apply. The API exists: POST {admin}/api/multi-dim-redirects/bulk (handleBulkCreateMultiDimRedirects), also reachable from the Bulk-create redirects from JSON button on /admin/multi-dim-redirects. This endpoint is the only way to apply an explorer redirect: the CSV CLI that map-charts-to-mdim hands over (createMultiDimRedirectsFromCsv) accepts an /explorers/ source, but it writes no sourceQueryParams — so the row bakes as one unconditional rule and every view of the explorer 302s to a single MDIM view, per-view routing gone. Its Zod schema deliberately mirrors this file's catchAll + redirects shape, ignores keys it doesn't know (sourceViewId, viewId, mdim, stats, targets), and reports target: null entries as skipped.

Post admin_bulk_payload.json, never mapping.json. The payload has empty-valued source dimensions stripped out. That is mandatory, not cosmetic: a condition is matched against the incoming URL's params, and an absent param is not an empty string — so a condition of {"Period": ""} can never match, and every view carrying one silently falls through to the catch-all instead of its intended target. mapping.json keeps them because it is the faithful record of the view grid.

Unlike the CSV (positional dimension_N columns, meant for a spreadsheet), the JSON carries every identifier a redirect needs:

{
  "explorer": { "slug": "...", "dimensions": ["<name>", ...] },
  "targets":  [ { "mdim": "...", "catalogPath": "ns/v/short#short", "dimensions": ["<slug>", ...] } ],
  "stats":    { "total": N, "resolved": N, "unresolved": N },
  "catchAll": {                        // bare explorer URL (no query params) fallback
    "source": { "explorerSlug": "..." },
    "target": { "mdim": "...", "catalogPath": "ns/v/short#short",
                "viewId": null, "dimensions": {} }   // no params → the MDIM's default view
  },
  "redirects": [
    {
      "sourceViewId": 1,
      "source": { "explorerSlug": "...", "dimensions": { "<name>": "<value>", ... } },
      "target": {                       // null when unresolved
        "mdim": "...", "catalogPath": "ns/v/short#short",
        "viewId": "A2",                 // internal id, cross-references the CSVs
        "dimensions": { "<slug>": "<choiceSlug>", ... }
      },
      "sharedTargetSourceIds": [1, 29, 57],   // present only when >1 source shares this target
      "unresolvedReason": "..."               // present only when target is null
    }
  ]
}

The source view is identified by the explorer slug + dimension name→display-value (the explorer URL query params); the target view by the MDIM catalogPath + dimension slug→choice-slug (the MDIM URL query params). Unresolved views are kept with target: null so the API can see the full picture; a consumer typically skips them.

An unresolved view is not a view left alone. The endpoint reports target: null entries as skipped, so they get no rule of their own — and the catch-all constrains no params, so it matches them instead. Those URLs land on the target MDIM's default view, carrying the explorer's own params into the grapher URL, rather than on anything equivalent to the view that was asked for. Leaving views unresolved is a deliberate call (they may genuinely have no MDIM counterpart), never a no-op; preflight reports the count so it gets made on purpose.

catchAll is always present: it redirects the bare explorer URL (no query params) — and serves as the sensible fallback for any view a consumer doesn't route individually — to the best-fitting MDIM with no query params, which grapher renders as that MDIM's default view (hence viewId: null, dimensions: {}). The best-fitting MDIM is the one most views resolve to, or whatever DEFAULT_MDIM in mapping_rules.py overrides it to.

The catch-all's destination is the only one nothing pins. Its row stores viewConfigId = NULL, so it resolves at request time to whichever view the MDIM renders for an empty selection — the first view in config order (grapher's filterToAvailableChoices takes the first available view's choice at every dimension). Rebuilding the MDIM can reorder its views, or edit that view's config in place, and the bare explorer URL then lands somewhere nobody reviewed. Extraction records that view in _sources.json (not in the payload, which must stay byte-reproducible) and preflight warns when it moves. A warning, not a blocker: the destination is a moving reference by construction, so blocking would flip an approved catch-all to NOT READY on any unrelated MDIM rebuild. To pin it, give the catch-all an explicit target view instead.

The script prints a validation report: how many explorer views resolved, distinct MDIM views hit per MDIM, how many rows share a target, and FLAGS for any explorer view that didn't resolve to a real MDIM view (fix the rules and re-run until there are no flags).

4. Review

Sanity-check the flagged rows and the judgment calls (approximate type matches, aggregate collapses, MDIM choices with no explorer source). For a topic owner's sign-off, build the side-by-side review page: each explorer view on the left, the MDIM view it will redirect to on the right, with approve/flag controls.

.venv/bin/python .claude/skills/map-explorer-to-mdim/scripts/build_review.py \
    --mapping-dir ai/<slug>-mdim-mapping \
    --explorer-slug <slug> \
    --mdim-slug <short_1>=<grapher_slug_1> \
    --mdim-slug <short_2>=<grapher_slug_2> \
    [--host https://ourworldindata.org] \
    [--output ai/<slug>_view_review.html] \
    [--no-coverage]

--mdim-slug maps each MDIM short name in mapping_rules.MDIMS to its published Grapher slug (the /grapher/<slug> part of the URL, not the catalogPath). _sources.json records the slugs extraction saw, and multi_dim_data_pages.slug has them if you have DB access; otherwise ask. The default host is production; for MDIMs that only exist on a staging branch pass --host https://staging-site-<branch>.

The script prints a coverage summary before writing the HTML: rows, distinct MDIM targets, many-to-one collapses, unresolved rows, MDIM views never targeted. Read it; it is the fastest way to spot a mapping gap the reviewer cannot see by eye.

What the reviewer gets: one pair at a time in iframes, the selection shown as chips above each chart, Approve / Flag / Clear with an optional note, keyboard navigation (← →, a, f, c), filters and live counts. Decisions auto-save to the browser's localStorage, can be mirrored to a JSON file on disk (Chrome/Edge), and Import merges another reviewer's export. The file is self-contained: send it directly, or publish it to vibe.owid.io with /owid-staff:create-vibe-app (from the owid-staff plugin, auto-installed here) when the whole team needs a link.

When the reviewer is done, ask for the exported JSON (or the auto-saved one): every row carries status (approved / flagged / blank) and note. Approved rows are ready to wire up; flagged rows need a second pass with the user. Corrections go through mapping_rules.py and a rebuild (step 3), never through the HTML.

Gotchas:

• localStorage is per browser and per file path, so switching machine loses the decisions unless they were exported or mirrored to disk.
• Single-choice MDIM dimensions are pruned from the URLs, matching the wide block of mapping_proposal.csv; do not add them back by hand.
• Explorer URL parameters are display names (Disaster Type=Floods), which is what the dimension_1..N columns hold. hideControls=true is appended to both sides; "open ↗" shows the view with its controls.
• A mapping hand-made in another schema (for example a colleague's multi-block CSV) must be converted to mapping_proposal.csv plus mapping_rules.py first; a one-off adapter in ai/ is fine.

5. Audit what references the explorers (ALWAYS offer; run when the user says yes)

ENV_FILE=<prod creds> DATA_API_ENV=production .venv/bin/python \
  .claude/skills/map-explorer-to-mdim/scripts/audit_references.py \
  --mapping ai/<a>-mdim-mapping --mapping ai/<b>-mdim-mapping \
  --out ai/<combined>-redirects

Writes references.csv + references.md across all the explorers in one pass. It resolves each referencing URL through the rules the payload would create, so every row says which view the reader lands on — and separates the link had no params from the link names a choice the explorer has since dropped, which lands on the MDIM's default view and needs authoring attention rather than a URL swap.

The timeline differs from the chart skill's, and this is the thing to say out loud: an embedded explorer breaks the moment the redirect is created, not later at unpublish, because the embed renders by fetching the explorer page and parsing it. So the 🔴 rows are migrated before step 7, not after.

The report also carries a ⭐ Featured metrics section — the one surface where this skill's usual "a link survives the 302" reasoning inverts. A featured metric is a topic-page slot held by URL, resolved only when Algolia indexes, matching pathname and exact params against published records. So it does not survive: it empties silently, and cannot be re-added once the explorer is gone. Hence step 5b.

5b. Swap the featured metrics (before the redirect — it cannot be undone after)

Work the ⭐ section of references.md, by hand at /admin/featured-metrics. These are editorial slots, so ask whoever owns the topic before repointing their rail.

Per row — add the MDIM view under the same tag and income group, drag it to the old ranking, delete the old row, then re-apply boost in search if it was on. One explorer view can hold several rows: the key is (URL, tag, income group). Full procedure: docs/guides/data-work/redirect-to-mdims.md.

Why here and not after step 7: creating the redirect darkens the explorer on the spot, and adding a featured metric requires a published slug. Once the redirect exists, no replacement is accepted and no record of the ranking survives. Unlike an embed, which breaks visibly, this fails quietly. The replacement is the bare MDIM view, not the redirect target — the admin strips reader params on paste and never validates the dimension params.