Map Charts To Mdim

Map charts to MDIM views (redirect proposal)

When individual grapher charts are being retired in favor of MDIMs, each chart needs a redirect to the equivalent MDIM view. Unlike explorers (see map-explorer-to-mdim), no human mapping rules are needed: a chart is a single view, and the match is computed from indicator IDs.

This skill proposes and validates; it never writes. Redirects are created by a human running grapher's createMultiDimRedirectsFromCsv CLI, which also unpublishes the source charts in the same transaction. Everything here builds toward that one command and proves in advance that it will succeed.

The target mechanism is the multi_dim_redirects table (source path → multiDimId + viewConfigId). Grapher's admin API can write it too, but not for these charts: it rejects any source that already has an old slug pointing at it, and its bulk endpoint accepts explorer sources only. The CLI (owid-grapher/devTools/createMultiDimRedirectsFromCsv.ts) exists precisely for the chart case.

And only for that case: one bare chart path per row. Two shapes the CSV pipeline cannot express. Neither is worked around by editing the CSV — one has another route, the other needs the CLI changed:

• Source query params. The source column is a path. Anything after ? survives into multi_dim_redirects.source verbatim, and chart sources are baked as a flat slug → path map that strips only the /grapher/ prefix (getGrapherToMultiDimRedirects → getMultiDimRedirectTargets), so the row matches no request and the redirect never fires. Conditioning on the incoming params is what the sourceQueryParams column does, and the CLI never writes it — grapher honors it for /explorers/ sources only (they bake to a query-param decision tree resolved at the edge), and the admin route rejects it outright on a /grapher/ source. A per-params chart redirect therefore needs changes on both sides, CLI and serving. preflight.py rejects a source carrying a query string for this reason.
• Explorer sources. /explorers/... passes the CLI's source regex, and the row it writes is worse than inert: with sourceQueryParams NULL it bakes as an unconditional rule, and explorer redirects are consulted on every request (unlike chart redirects, which fire only on a 404), so the entire explorer — every view, whatever the params — starts 302-ing to the single target view. The chart handling it also skips, silently: the source is never unpublished (getChartSlugsToUnpublish skips non-/grapher/ sources), and the chained old-slug migration early-returns. Explorers go through the admin bulk endpoint, which is map-explorer-to-mdim's job, not this skill's.

Inputs

Exactly one chart selection:

• --tag "<tags.name>" — published charts with that exact tag (the script prints nearby tag names, e.g. Economic Inequality vs Economic Inequality by Gender, so the user can widen the selection knowingly).
• --slugs a,b,c or --slugs @file — explicit chart slugs.
• --dataset-id N — published charts using any variable of that dataset.

Targets: all published MDIMs by default (auto-discovery); restrict with repeatable --mdim <catalogPath> (as in multi_dim_data_pages.catalogPath).

DB access (confirm this before running)

Charts and MDIMs are read from the grapher DB via OWID_ENV — read-only is enough for the proposal phase. Point OWID_ENV at a DB that has both:

1. Staging DB: prefix with STAGING=1 (the current branch's staging-site-<branch> DB; STAGING=<name> for another branch's). Being checked out on the branch is NOT enough by itself — without the prefix, OWID_ENV silently points at your local dev DB, which also passes the connection preflight.
2. Production, read-only, via an env file: prefix with ENV_FILE=<prod creds file> DATA_API_ENV=production. Don't assume the file name — check what exists (ls .env*); on some machines it's .env.prod, on others .env.live.
3. Some other credentials file: ENV_FILE=<their file>.

If no suitable env file exists and there's no staging server to point at, stop and ask the user — never hardcode credentials.

The extractor prints the environment it resolved (grapher DB: ...) — check that line matches what you intended before trusting the output. All links in the artifacts (and the review HTML panes) default to that environment's site, so a staging extraction is reviewed against staging; --host overrides.

ls .env* 2>/dev/null
ENV_FILE=<file> DATA_API_ENV=production .venv/bin/python -c \
  "from etl.config import OWID_ENV; print('DB OK:', OWID_ENV.read_sql('SELECT 1 AS x').iloc[0,0])"

Workflow

1. Extract + match

ENV_FILE=<prod creds> DATA_API_ENV=production .venv/bin/python \
  .claude/skills/map-charts-to-mdim/scripts/extract_and_match.py \
  --tag "Economic Inequality" \
  --out ai/economic-inequality-charts-mdim-mapping

Writes into --out:

Handoff convention: one source per JSON. Like the explorer→MDIM redirect deliverables (ai/explorer-mdim-redirects/, one admin_bulk_payload.json per explorer), each JSON handed over must describe exactly ONE source page. For charts that means one file per chart, in payloads/. The combined mapping.json is the machine record that preflight.py and audit_references.py read — don't hand that one over as a copy-paste payload.

Matching: a chart matches a view when the y-variable-ID sets are equal AND x/size/color agree (absent == absent) — after decoration indicators are stripped from x/size/color on both sides: population as Marimekko width / bubble size, and regions#owid_region as continent coloring (matched by catalogPath, so version bumps don't matter). Charts and views carry these inconsistently without changing what data is plotted — an MDIM view adds x=population for its Marimekko tab, an editor adds continent colors to a line chart — and requiring them to agree literally drops same-y pairs into none with no report at all (equal y sets means they aren't even a near miss). A match made across such a difference says so in the proposal's note column. Content indicators (a scatter's GDP-per-capita x, a "political regime" coloring) are never stripped. Two guardrails keep the rule from overreaching: a scatter's x slot is never stripped — on a ScatterPlot, x=population is the plotted relationship, not decoration (size/color there still are) — and the population pattern is end-anchored to the raw head-count columns (#population, #population_historical, #population_projection), because the same dataset also carries population density columns that are content. Several matching views → tiebreak on chart type; still ambiguous → reported, never guessed. Any partial indicator overlap that is not an exact match → near_miss, reported only — not just subset/superset: a chart plotting {A, B} against a view plotting {A, C} shares A, so calling it none would assert an overlap check that came out the other way. Quality labels:

• exact — one view matches (possibly via the chart_type tiebreak — check the tiebreak column).
• forced / skipped — set by overrides.csv (below).
• ambiguous — several views match; needs a human pick.
• near_miss — overlapping but unequal indicator sets; the diff is spelled out.
• none — no view shares the chart's indicators. An accepted outcome — only matched charts get redirects; don't force the rest.

Twin suspects. ID matching is blind to one real equivalence: a dataset that publishes the same series in two tables (WID/LIS do — inequality#share_top_10__… for the standalone charts vs incomes#share__…quantile_10… for the MDIM, with identical values). The extractor flags these — unmatched charts whose y comes from the same dataset as a slot-compatible view with a similar column name — in the run report and in mdim_suggestions.md as twin suspects, with the exact overrides.csv line to use. A suspect is NOT a match: verify first that the two indicators' values are identical (fetch both api.ourworldindata.org/v1/indicators/<id>.data.json and compare every entity-year), then force it, citing the verification in the override note.

Either side carrying several indicators in one x/size/color slot has no chart-shaped signature, so it is excluded from matching rather than truncated to the first: a truncated signature can spuriously exact-match a counterpart that lacks the rest. Charts land in none with the reason in note; views are dropped from the target pool with a warning. overrides.csv can still force either.

Conflicts vs CLI-required. Matched charts are checked (read-only SQL) against the same conditions the apply CLI validates. Two outcomes, and the distinction matters:

• cli_required — the chart has old slugs redirecting into it, or site redirects pointing at it. Not a blocker: the CLI migrates both. (The admin API rejects exactly these as redirect chains, which is why it isn't the apply path.) The old slugs are listed in the proposal and carried into payloads/*.json as oldSlugs.
• conflict — a genuine blocker that would abort the CLI's transaction: the source is already a redirect source, the chart's own slug is itself an old slug, the target MDIM's /grapher/<slug> is a redirect source, a self-redirect, or one of the old slugs is already a redirect source elsewhere. These land in mapping.json → conflicts[] and are kept out of the CSV — and, because being absent from the CSV is not the same as being finished, they are also listed in HANDOFF.md with their blocker, next to the already-redirected rows.

Charts already redirected to the same target are counted as already_done — no redirect to create, but they are still published (the extractor only selects published charts), so they shadow their redirect and it never fires. The CLI cannot finish these: their source is already a multi_dim_redirects source, which its own validation rejects, so including them would abort the whole run. Preflight reports them as MANUAL; unpublish each one in the grapher admin. The exception is a chart carrying old slugs — hand-unpublishing deletes its chart_slug_redirects rows and those slugs become hard 404s, so take those to the Grapher team instead. That manual unpublish breaks embeds exactly as the CLI's would, so already_done charts sit behind the same reference gate and appear in the step-4 audit alongside the proposed redirects.

Many charts → one view is legitimate (e.g. a line chart and its map twin) — surfaced via shared_target_chart_ids, never collapsed.

2. Review

Side-by-side HTML (chart left, proposed MDIM view right; approve/flag with notes, decisions persist in the browser):

.venv/bin/python .claude/skills/map-charts-to-mdim/scripts/build_review.py \
  --mapping-dir ai/economic-inequality-charts-mdim-mapping

Rows without a target show their candidates / near-miss diff instead of an iframe, so ambiguous cases can be triaged into overrides.csv from the same UI. The proposal CSV also works standalone in a spreadsheet (URLs are clickable).

When the review is done, export the decisions (⬇ JSON or ⬇ CSV) — the apply step consumes that file so flagged charts are excluded.

3. Overrides + re-run (only if needed)

For ambiguous rows (or to force/suppress a match), write <out>/overrides.csv — it is never overwritten by re-runs:

chart_id,action,note
1234,SKIP,keep this chart live
5678,poverty_inequality/latest/gini#gini|metric=gini__welfare=disposable,picked over the WID twin
9012,01981234-abcd-7def-8123-456789abcdef,forced by view_config_id

Then re-run step 1 (same command) — overridden rows become forced/skipped and mapping.json regenerates.

4. Audit what references the charts (ALWAYS offer; run when the user says yes)

A redirect only rescues plain hyperlinks. Anything that embeds the chart — explorers, data insights, static viz, article chart blocks — resolves it by id or slug and keeps rendering the old config, so it breaks when the source chart is unpublished (which the apply step always does).

ENV_FILE=<prod creds> DATA_API_ENV=production .venv/bin/python \
  .claude/skills/map-charts-to-mdim/scripts/audit_references.py \
  --mapping ai/economic-inequality-charts-mdim-mapping

The sweep itself is the shared find-chart-references skill; this script is the redirect-specific consumer that adds the replacement URLs. It writes references.csv + references.md, one row per reference. The report is organized by what the reader does, not by severity tier: embedded charts and text links sit adjacent in one "Google Doc edits" section (one editing pass per doc covers both — 🔴 embeds break on unpublish and gate the CLI, 🟡 links stay functional behind the 301), with reader-facing section names ("Embedded charts", "Text links", "Front-matter chart URLs") rather than raw ArchieML tokens (those live in the CSV's component column). Topic-page All charts entries collapse to a per-page summary and need NO action: the block lists only published charts (GdocPost.loadRelatedCharts filters on isPublished), so entries drop out on their own at the next bake — and no replacement is possible either, because the block is built from charts × chart_tags only and cannot list MDIMs; featuring the MDIM on a topic page is a separate gdoc-authoring change. Featured metrics get their own ⭐ section, and they are the opposite case: the All charts block heals itself, a featured metric does not. It is a topic-page slot held by URL, matched on exact pathname and params only when Algolia indexes, against published records — so unpublishing empties it silently. It does not block the CLI, for the same reason a key chart doesn't, but it is unrecoverable afterwards — hence step 4b rather than post-migration cleanup. Narrative charts get their own table (before the All charts summary), one row per chart: the admin editor link and parent chart; "create from this view" — the target view carrying that narrative chart's stored controls, which is both the rendering to match and the place to create from; "text to re-apply" (see below); the pages that actually embed it, resolved by a second hop the sweep doesn't make (posts_gdocs_links on linkType='narrative-chart', both narrative-chart and key-insights components), each with its doc link and the name to search for; and the ordered steps. Each row carries only the order that applies to it, on the rule above: a published page among the users makes create → repoint → delete mandatory; with none, the row emits the delete → create shortcut that reuses the same name (a draft reference then keeps resolving, since pages reference the name). ℹ️ unpublished/draft pages close the report.

Create from the view, not from a create link. Tell the operator to open the target view and use the chart's own "Create narrative chart" admin control: the MDIM page builds that control's target from whichever view is on screen (site/multiDim/MultiDim.tsx), so the new chart is parented to the right view and inherits the controls set on it. A bare /admin/narrative-charts/create?type=multiDim&chartConfigId=<viewConfigId> link looks equivalent but opens a copy of the MDIM's default view, so never hand that out as the create step.

The new chart opens at the parent view's defaults — the state does not carry over. The control gets the parent right, but not the state on top of it: the dimension selection, the entity selection and tab/time all come up at defaults, and authored text never transfers at all. So the report's "Set by hand after creating" column lists all three groups per chart, in the order they get applied in the editor:

1. view dimensions — the target view's own dimension values (from the proposal), because the new chart opens on the MDIM's default view;
2. controls — taken from the narrative chart's authored layer (narrative_charts.patchConfigId → chart_configs.config) (the delta its author typed on top of the parent) plus its stored queryParamsForParentChart, and listed chart type first (it decides which other controls exist), then the entity selection (the most visible thing to get wrong), then the rest alphabetically. Entities are always shown by name, never as codes: a URL param spells them ZWE~MDG, so those are resolved against entities before display (unknown codes pass through). The patch and the params encode the same state, so a URL param is dropped when the patch already carries the equivalent config key (country ↔ selectedEntityNames, focus ↔ focusedSeriesNames, time ↔ minTime/maxTime) — otherwise the cell asks for one setting twice in two spellings, and the config form is what the editor's fields expose;
3. FAUST text — title, subtitle, note, sourceDesc, hideAnnotationFieldsInTitle from the patch. Miss these and the replacement silently renders the view's wording instead of the text the article was written around.

Every group is diffed against the target view's config, so only genuine differences are asked for. dimensions and $schema are excluded from the patch on purpose: the new parent view supplies them, and re-applying the old ones would repoint the chart at the retired chart's indicators.

Suggest the replacement's name, and check it is free. When a published page holds the original name the replacement needs a different one — and that name is permanent, since create rejects an existing name and there is no rename, so it is not a staging name to tidy up after the delete. The report suggests <original>-mdim (falling back to -mdim-2, -mdim-3, …), validated against every name in narrative_charts so the suggestion cannot be the one thing create refuses; suggestions handed out within a run are reserved as they go. In the delete-first case there is nothing to suggest — the row names the original, which the delete frees for reuse.

Admin routes need the admin origin. A narrative chart's where_path is /admin/narrative-charts/<id>/edit, and the public site does not serve /admin — prefixing it with the site host yields a link that 404s. Both the sweep and this consumer route such paths through a helper (admin_url / absolute_url) that strips the admin root's own /admin suffix before joining, or the result carries /admin/admin/.

Pure SQL, so read-only credentials are enough. It sweeps both the current slug and every old slug that reaches the chart — references written before a rename point at the old one.

Watch for the param-collision warning. Grapher merges the incoming URL's params over the redirect target's, so a link carrying ?metric=… overrides an MDIM dimension of the same name and lands the reader on the wrong view. The audit flags those explicitly.

Mention this step every time (like /update-dataset step 7); running it is the user's call, since a wide sweep costs tokens — but preflight gates on the same embedded references, so it becomes mandatory before applying.

Narrative charts do not block a migration. One parented to a chart owns a materialized full config and renders from it, so unpublishing the parent leaves it intact; only its generated "Explore the data" href uses the parent slug, and the 301 covers that. They are classified link, reported by preflight but never gated on — gating would strand every such chart behind raw SQL for no reader-visible gain. Do check the href's query params for collisions with the target view's dimensions (step 4 flags them).

To actually fix one, replace it — don't try to repoint it. The parent columns are INSERT-only, so there is no repointing API and never will be one by design; the route the Grapher devs endorse is to create a new narrative chart from the equivalent MDIM view and delete the old one. Three API facts set the order, and getting it wrong strands you:

• create rejects a name that already exists, and requires kebab-case;
• delete refuses while a published post references the name;
• update writes only query params — there is no rename.

So the obvious sequence (delete, then recreate under the same name) is blocked in precisely the case that matters — a narrative chart embedded in a published article. Two paths, and the references audit (step 4) tells you which applies:

No published post references it → delete, then create the replacement with the same name. The name is preserved and nothing else changes.

A published post references it (the usual case) → do it in this order:

1. Create the replacement from the MDIM view, under a new kebab-case name.
2. Update the article(s) to reference the new name.
3. Delete the old one — now unreferenced, so the delete succeeds.

Never delete first. The delete will fail, and unpublishing the article to force it through breaks the page for readers.

Do the create from the target view in the site UI, using the chart's own "Create narrative chart" admin control (see the rule above). The three routes do NOT behave alike, so don't describe them interchangeably:

So: use the control, then set what the report's Set by hand after creating column lists. Prefer the UI over the API anyway — AdminAPI has get_narrative_chart and update_narrative_chart but no create or delete, so scripting means hand-rolled HTTP for both ends of the swap.

If you do script it, the endpoints are POST {admin_api}/narrative-charts and DELETE {admin_api}/narrative-charts/<id>:

{
  "type": "multiDim",
  "name": "<kebab-case-name>",
  "parentChartConfigId": "<the MDIM view's chart_configs.id>",
  "config": { /* the OLD narrative chart's rendered full config */ }
}

You already have parentChartConfigId: it is target.viewConfigId in the redirect payload for the chart this narrative chart hangs off (payloads/<chart_slug>.json, or target_view_config_id in mapping_proposal.csv). That is the MDIM view's chart_configs.id — the same value find-chart-references reports as config_id on an mdim view row. So "which MDIM view do I parent the replacement to" is answered by the proposal: it is the view the chart was going to redirect to. Get the config from AdminAPI.get_narrative_chart(<old id>)["configFull"]: the endpoint derives the patch itself by diffing what you pass against the new parent, so pass the rendered full config, not the old patch.

Replacement is manual, and that is a deliberate call, because the number of narrative charts that can ever be in this situation is tiny. The population is not "all narrative charts" — it is narrative charts whose parent chart is itself a redirect candidate, i.e. has an exact MDIM-view match. Applying this skill's own matching rule to every published chart with narrative children gives a site-wide ceiling of a handful of narrative charts, if every matchable chart were migrated — not a per-migration figure.

Re-measure rather than assuming that ceiling: it grows as MDIMs are published, since each new MDIM view can turn an existing chart into a candidate. ai/ in this repo has the query; the shape is — narrative charts on published parents → those parents' indicator slots → compare against every published view's slot signature. Revisit the decision if a single migration would require replacing more than a handful, and say so in the PR rather than grinding through them silently.

A narrative chart parented to an MDIM view blocks that MDIM's next re-publish, and no issue tracks it. Manual replacement does nothing for this. It happens because cleanUpOrphanedChartConfigs deletes multi_dim_x_chart_configs rows with no narrative-chart guard and the FK refuses (ER_ROW_IS_REFERENCED_2). Renaming a dimension or choice slug is enough to trigger it, and it surfaces as a data update failing with an opaque FK error. Replacement makes this more likely, not less: every replacement creates a new MDIM-parented narrative chart. If the count of MDIM-parented narrative charts on production grows, file it as its own issue — the write-up is in ai/narrative-charts-grapher-issue.md (§2), alongside ai/narrative-charts-slack-post.md.