Add Provider Regions

Add provider regions

Add an external data provider's regional grouping (World Bank, WHO, UN, IEA, Pew, Maddison, WID, ILO, …) to OWID's shared regions dataset. The regions are defined once in regions.yml; the grapher step then turns each provider into a categorical country→region map indicator ({defined_by}_region) that powers the world-region-map-definitions page.

There are two repos involved: the etl repo (Steps 1–8 — definitions, indicators, metadata, optional chart migration) and the owid-grapher repo (Step 9 — registering the provider in the frontend so its regions show up in entity selectors, tooltips, and admin presets). The owid-grapher work is a separate PR done after the ETL regions are merged and published to the catalog.

The defining principle of this skill: the country composition of each region must come from the source — either the provider's own dataset or a reference document the provider publishes — and be verified by set-equality. Never hand-type or guess which countries belong to a region.

Inputs

• Provider — name and a short slug for defined_by (e.g. wb, who, maddison). Region names will be suffixed (Provider).
• Dataset path (if one exists) — the provider's garden/grapher dataset, e.g. data/garden/<ns>/<version>/<short_name>. Many providers ship their regional grouping inside their own dataset.
• Reference URL (optional) — where the provider publishes the classification, used only if the dataset doesn't encode it.

All paths below are repo-relative. Always use .venv/bin/ for python/etl/etlr.

The files you'll touch in etl:

• etl/steps/data/garden/regions/2023-01-01/regions.yml — region definitions (the header comment documents every field).
• etl/steps/data/grapher/regions/2023-01-01/regions.py — builds the {defined_by}_region indicators (only edit for a cross-tier back-fill, Step 3).
• etl/steps/data/grapher/regions/2023-01-01/regions.meta.override.yml — origin anchors + per-indicator metadata.
• Not regions.codes.csv — that file lists countries and OWID historical codes only; provider aggregates never go there.

And in owid-grapher (Step 9, separate PR): the auto-generated packages/@ourworldindata/utils/src/regions/regions.data.ts, a few hand-maintained label registries, and the two region-color dictionaries in packages/@ourworldindata/grapher/src/color/CustomSchemes.ts. Reference PRs: owid/owid-grapher#6465 (IEA regions) and #6852 (regions colored by name on categorical maps).

Step 1 — Find the region composition (the key check)

Inspect the provider's dataset for an embedded aggregation, in this priority order:

1. A categorical region / subregion column mapping each country to its region.
2. Region entities present as rows in the data (e.g. "East Asia (Provider)" alongside countries).
3. A table-of-contents / dictionary table with tier columns (some providers ship *_region, *_subregion_broad, *_subregion_detailed style columns — these are authoritative tier definitions).

from owid.catalog import Dataset
ds = Dataset("data/garden/<ns>/<version>/<short_name>")
print(ds.table_names)
tb = ds.read("<table>", safe_types=False)
print([c for c in tb.columns])
# region column -> country mapping:
print(tb.dropna(subset=["region"]).groupby("region")["country"].unique())
# or region entities present in the data:
print(sorted(c for c in tb["country"].unique() if "(" in str(c)))

If the composition is in the data: derive each region's member set directly from it. This is the source of truth.

If it is NOT in the data: ask the user for a reference — a link, PDF, or doc where the provider publishes the classification (e.g. a "regional groupings" page or methodology annex). Fetch it with WebFetch and derive membership from there. Keep the reference URL; it becomes the origin url_main in Step 6.

Step 2 — Resolve members to OWID region codes

Each member must map to a code that exists in regions.yml. Use regions.codes.csv for ISO alpha-2/alpha-3, with a name/alias fallback for non-standard codes (microstates, Kosovo-style cases).

import csv, yaml
regions = yaml.safe_load(open("etl/steps/data/garden/regions/2023-01-01/regions.yml"))
by_code = {r["code"]: r for r in regions}
name_to_code = {}
for r in regions:
    name_to_code.setdefault(r["name"], r["code"])
    for a in r.get("aliases", []):
        name_to_code.setdefault(a, r["code"])
alpha2, alpha3 = {}, {}
for row in csv.DictReader(open("etl/steps/data/garden/regions/2023-01-01/regions.codes.csv")):
    if row["iso_alpha2"]: alpha2[row["iso_alpha2"]] = row["code"]
    if row["iso_alpha3"]: alpha3[row["iso_alpha3"]] = row["code"]

def resolve(member):  # member is a provider country name or code
    return alpha2.get(member) or alpha3.get(member) or name_to_code.get(member)

unresolved = [m for m in provider_members if resolve(m) is None]
assert not unresolved, f"Resolve these before continuing: {unresolved}"

Handle deliberately:

• Historical entities the provider assigns to a region (OWID_USS, OWID_CZS, OWID_YGS, OWID_SDN, …) — include them where the source does.
• OWID aggregate codes (e.g. Channel Islands OWID_CIS) — if the provider lists a sub-territory that OWID models as an aggregate, decide once where it lives and avoid listing it twice across regions (the garden step's duplicate-member check will catch a double-count). Document any such choice with a # NOTE: in the YAML.
• A region's members may reference other aggregate codes (not just countries); the garden step's replace_aggregate_members expands them recursively (see UN M49 in regions.yml for the pattern).

Step 3 — Decide the tier structure

• Single flat partition (most providers): one defined_by: <provider>, one indicator. Done.
• Hierarchical provider (broad regions split into subregions): use one defined_by per level — <provider>_1 (broadest), <provider>_2, … Look at the existing un_m49_1/2/3 and ilo_1/ilo_2 sections in regions.yml as templates.

Why split: the grapher step inverts all aggregates sharing a defined_by into a single country→region column. If two tiers share one defined_by, every country lands in 2+ regions, producing "belongs to multiple regions" warnings and a scrambled, order-dependent indicator. One defined_by per level keeps each indicator a clean partition.

Two rules for multi-tier providers:

• Completeness: each kept tier must partition the provider's covered world. You cannot keep a sub-breakdown of one parent without its siblings — e.g. if you keep one parent's sub-regions, keep every parent's, so the tier still covers everyone. Drop intermediate levels that don't form a complete partition; keep only tiers that are both useful and complete.
• Region shared across tiers: when one region exists at two levels (a broad region with no finer breakdown), it carries a single defined_by, so it appears natively in only one indicator. Tag it at one level, then back-fill it into the other level's indicator in the grapher step (Step 6c).

Step 4 — Edit regions.yml

Append a section, delimited like the others:

##########
# <Provider full name>
##########
- code: PROVIDER_XXX
  name: "<Region> (<Provider>)"
  region_type: "aggregate"
  defined_by: <provider>            # or <provider>_1 / <provider>_2 for tiers
  members:
    - ISO3
    - ISO3
    - PROVIDER_SUB                  # may reference another aggregate code

• Names carry the (Provider) suffix and should match the entity names the provider's own dataset publishes (so charts line up).
• Codes are PROVIDER_XXX, uppercase, unique.
• For composite levels whose members are sub-aggregates, you can either list the sub-aggregate codes (expanded recursively) or list the union of countries directly. When you need the country union, compute it programmatically rather than hand-typing:

def expand(code):
    out = []
    for m in by_code[code]["members"]:
        out.extend(expand(m) if m in by_code and m.startswith("PROVIDER_") else [m])
    return sorted(set(out))

For anything beyond a couple of regions, regenerate the whole provider section with a small script (compute members, emit the YAML block, splice it into the file) rather than many manual edits — it's less error-prone and keeps formatting uniform.

Step 5 — Build and verify the garden step

.venv/bin/etlr data://garden/regions/2023-01-01/regions

(No --force — editing the YAML is enough to trigger a rebuild.) The step runs sanity checks: unique codes/names, unique members within a region, all referenced codes exist, and cycle detection during aggregate expansion.

Then verify against the source and the partition property:

from owid.catalog import Dataset
import json
tb = Dataset("data/garden/regions/2023-01-01/regions").read("regions").reset_index()

# 1) set-equality vs the source mapping (expected = region -> set of OWID codes from Step 1/2)
m = {r["code"]: set(json.loads(r["members"])) for _, r in tb[tb["defined_by"].str.startswith("<provider>")].iterrows()}
for code, exp in expected.items():
    assert m[code] == exp, f"{code}: missing {exp - m[code]}, extra {m[code] - exp}"

# 2) each tier partitions the same set and is pairwise-disjoint
for tier, codes in {"<provider>_1": [...], "<provider>_2": [...]}.items():
    union = set().union(*(m[c] for c in codes))
    for i, a in enumerate(codes):
        for b in codes[i+1:]:
            assert not (m[a] & m[b]), f"{a} & {b} overlap in {tier}"
    print(tier, len(union), "countries")

If a sanity check fails, fix the upstream logic or the membership — don't suppress the assertion.

Step 6 — Grapher indicators + metadata

The grapher step auto-creates a {defined_by}_region column for every defined_by. Each needs a metadata block or the build fails on a missing title (grapher_checks).

6a. Origin anchor — add to the definitions: block in regions.meta.override.yml, taken from the provider dataset's actual origin:

from owid.catalog import Dataset
ds = Dataset("data/garden/<ns>/<version>/<short_name>")
tb = ds.read(ds.table_names[0], safe_types=False)
o = tb[[c for c in tb.columns if c not in ("country", "year")][0]].metadata.origins[0]
print(o.producer, "|", o.title, "|", o.url_main, "|", o.date_accessed, "|", o.attribution, "|", o.attribution_short)

  origins_<provider>: &origins_<provider>
    producer: <producer>
    title: <title>
    url_main: <url_main>              # or the reference URL from Step 1
    date_accessed: "<YYYY-MM-DD>"
    attribution: <attribution>        # only if the source defines it
    attribution_short: <short>        # only if the source defines it

Omit date_published. Region definitions are time-invariant; a publication year would render next to the source line below the chart, where it's meaningless.

6b. Indicator block — one per {defined_by}_region, under tables.regions.variables:

      <provider>_region:                       # or <provider>_1_region / <provider>_2_region
        title: World regions according to <Provider>
        description_short: |-
          Regions as defined by <Provider full name>.
        type: ordinal
        sort:                                    # legend/map order — see ordering rule below
          - <Region> (<Provider>)
          - ...
        origins:
          - *origins_<provider>
        presentation:
          grapher_config:
            hideAnnotationFieldsInTitle:
              time: true                        # hide the placeholder data year in titles
            map:
              tooltipUseCustomLabels: true          # tooltip shows the stripped label too — see below
              colorScale:
                baseColorScheme: OwidCategoricalMap    # name-keyed region colors — see Step 9
                customCategoryLabels:
                  # one entry per region in `sort` — drops the suffix from the
                  # legend, and from the tooltip via the flag above
                  "<Region> (<Provider>)": "<Region>"
                customHiddenCategories:
                  "No data": true
                # No customCategoryColors. Colors live in MapContinentColors (Step 9),
                # keyed by region name; a block here would override them and fork the
                # source of truth.

customCategoryLabels alone only fixes the legend. Hovering a country still shows the raw "<Region> (<Provider>)", because the map tooltip falls back to the unformatted value unless map.tooltipUseCustomLabels: true is set (MapChartState.formatValueForTooltip — it looks up the bin's label only behind that flag). Set both, always, and give every region in sort a label entry: a region you miss keeps its suffix in the legend and the tooltip while its siblings lose theirs, which reads as a data error rather than a missing config line.

Set the palette, don't hardcode the colors. A categorical map with no baseColorScheme falls back to BuGn — a sequential green ramp (MapChartState.ts:53). Region colors are only looked up by name when the map is on OwidCategoricalMap, the scheme that carries colorMap: MapContinentColors. Setting it on the indicator means every chart built on it inherits the palette (the same inheritance that already carries customCategoryLabels). A chart's own patch still wins over the inherited value, so once the chart exists, confirm on staging that it isn't patched to something else (world-regions-according-to-pew is patched to continents — a chart palette on a map, which pulls the strong colors instead of the muted ones).

Ordering rule for sort: the map legend renders as a single row in sort order, so order the regions to read left-to-right across a world map. The house sweep is:

(North/Northern) America → Latin America / Caribbean → Africa (north to south within the slot) → Middle East / North Africa → Europe → CIS / Russia / Central Asia → South Asia → East and South-East Asia → Australia and New Zealand → Oceania

Drop what the provider doesn't have, keep the rest in this relative order. Three wrinkles worth knowing. Europe sits after Africa and the Middle East, because Europe and Africa share the same longitudes (Europe north, Africa south) so west-to-east alone can't separate them — the convention sweeps the southern band first. At sub-region granularity, a "Western Asia" that the provider models as part of Asia stays in the Asia block rather than moving up to the Middle East slot (compare un_m49_2 with ei). And which Africa regions land in the Africa slot depends on how the provider splits the continent: where Africa has its own sub-regions they run north to south inside the slot — Northern Africa then Sub-Saharan Africa (un_m49_2, ilo_2), or Northern Africa then the compass sub-regions (fao_2). Where instead North Africa is folded into a Middle East and North Africa bucket, that bucket is not part of the Africa slot at all — it takes the Middle East slot — so Sub-Saharan Africa is alone in the Africa slot and therefore comes first (wb, unsdg, pew, wid, fao_sdg).

The order is defined twice — keep the two equal. This sort drives the published map's legend; customRegionDisplayOrder[<provider>] in owid-grapher's RegionTooltipData.ts (Step 9) drives the legend of the mini-map in the region hover. When they diverge, the same provider lists its regions in two different orders on the same page. Treat customRegionDisplayOrder as the reference and copy it into sort verbatim; if you're adding a new provider, write the order once and paste it into both.

Reordering is only color-safe once the regions are pinned. For a region with a MapContinentColors entry, sort moves legend positions and nothing else — the color follows the name. For a region without one, OwidCategoricalMap falls back to handing out palette colors by position, so reordering silently recolors the map. Check every region of the tier against MapContinentColors before touching sort: if any are unpinned, pin them first (Step 9, with the design sign-off) and reorder in the same change, or leave the order alone and say why in a # NOTE: beside it. The two edits look independent and are not.

6c. Cross-tier back-fill — if Step 3 found a region shared across tiers, add the masked back-fill (one extra fillna/masked assignment) to grapher/regions/2023-01-01/regions.py after the inversion loop, mirroring the existing process_un_definitions pattern.

6d. Build and verify:

.venv/bin/etlr data://grapher/regions/2023-01-01/regions

from owid.catalog import Dataset
tb = Dataset("data/grapher/regions/2023-01-01/regions").read("regions")
for col in ["<provider>_region", ...]:
    o = tb[col].metadata.origins[0]
    print(col, tb[col].notna().sum(), "countries,", tb[col].nunique(), "categories | attr:", o.attribution_short, "| date_published:", o.date_published)

Confirm: the new columns exist with titles, attribution carried, date_published is None, no "belongs to multiple regions" warning in the build log, and (multi-tier) each indicator covers the full partition.

Step 7 (optional) — Migrate an existing in-dataset region chart

Do this only if the provider's own dataset already has a region variable powering a region-definition map chart that should now read the shared indicator (an "indicator upgrade"). Skip otherwise.

Find the old variable and its chart on the staging DB for the current branch:

from etl.config import OWID_ENV
# variables of the provider dataset (note: query % LIKE with params=)
OWID_ENV.read_sql("""
  SELECT v.id, v.shortName FROM variables v JOIN datasets d ON v.datasetId=d.id
  WHERE d.shortName=%(s)s AND v.shortName='region'
""", params={"s": "<short_name>"})
# charts using it — slug lives on chart_configs, not charts
OWID_ENV.read_sql("""
  SELECT c.id, cc.slug, cc.config->>'$.title' AS title
  FROM charts c JOIN chart_dimensions cd ON cd.chartId=c.id
  JOIN chart_configs cc ON cc.id=c.configId
  WHERE cd.variableId=%(v)s
""", params={"v": OLD_VAR_ID})

Repoint the chart at the new {provider}_region variable:

from etl.config import OWID_ENV
from apps.chart_sync.admin_api import AdminAPI
api = AdminAPI(OWID_ENV)
cfg = api.get_chart_config(CHART_ID)
cfg["dimensions"] = [{"property": "y", "variableId": NEW_VAR_ID}]
cs = cfg.setdefault("map", {}).setdefault("colorScale", {})
# re-key the labels onto the NEW "(Provider)"-suffixed category values (the new
# variable's categories carry the suffix):
cs["customCategoryLabels"] = {f"{name} (<Provider>)": name for name in OLD_REGION_NAMES}
# put the map on the name-keyed palette and drop any hardcoded colors — the old
# chart's customCategoryColors would override MapContinentColors (Step 9):
cs["baseColorScheme"] = "OwidCategoricalMap"
cs.pop("customCategoryColors", None)
api.update_chart(CHART_ID, cfg)

Staging admin writes work behind Tailscale without ADMIN_API_KEY. The change surfaces in chart-diff for review before it reaches production. Verify by re-reading the config: dimensions point at the new variable, labels are re-keyed, customCategoryColors is gone, and the map renders in the muted map colors (chart-diff will show the color change — that's expected, and it's what Step 9 pins).

Step 8 — Commit and open a PR

make check
git add etl/steps/data/garden/regions/2023-01-01/regions.yml \
        etl/steps/data/grapher/regions/2023-01-01/regions.meta.override.yml \
        etl/steps/data/grapher/regions/2023-01-01/regions.py   # if back-fill added
git commit -m "📊🤖 Add <Provider> regions to regions dataset"

If not already on a feature branch, create one and a PR with etl pr "Add <Provider> regions" data, then push. In the PR body, open with the disclosure blockquote (> _Written by Claude <model name> — @<handle> at the wheel._, model name = the model actually generating the content) and keep any reviewer attribution out of committed code/YAML.

Heads-up: once this merges, the post-merge deploy is slow (often hours; see Step 9 → Sequencing), so don't expect to chain straight into Step 9.

Step 9 — Register the provider in owid-grapher (separate repo + PR)

The grapher frontend keeps its own copy of the regions and a few hand-maintained registries. The provider must be added there too, or its (Provider) entities won't be grouped/labelled correctly in entity selectors, map tooltips, and admin presets.

Sequencing — and expect a long wait: the frontend's regions.data.ts is regenerated from the production catalog (https://catalog.ourworldindata.org/external/owid_grapher/latest/regions/regions.csv). So do Step 9 after the ETL PR (Step 8) is merged and the data://external/owid_grapher/latest/regions step has rebuilt on prod. That rebuild is slow — often hours, not minutes — because editing the regions dataset invalidates a huge swath of the DAG (almost everything that aggregates by region or merges population/regions depends on it), so the post-merge deploy has a lot to rebuild before the new regions reach the catalog. Don't run yarn runRegionsUpdater until the regions are actually live there, or it'll regenerate from stale data. Verify first:

curl -s "https://catalog.ourworldindata.org/external/owid_grapher/latest/regions/regions.csv?nocache" | grep -c "PROVIDER_"

A non-zero count means it's ready. If you're waiting, poll this every few minutes rather than running the updater blind.

Preview from your ETL branch's staging catalog while you wait. The updater reads ETL_REGIONS_URL if it's set (devTools/regionsUpdater/update.ts), and every ETL staging server publishes its own catalog on port 8881 with the same schema as prod. So you can regenerate regions.data.ts — and everything downstream of it, including the colors and the test page — before the ETL PR is anywhere near merged:

The host name is not the branch name — derive it, don't hand-write it. etl.config.get_container_name() replaces /, . and _ with -, strips a leading staging-site-, truncates what's left to 28 characters, then drops any trailing hyphen. Substituting the raw branch (or truncating it yourself) silently produces a host that is either unreachable or, worse, another branch's staging server. Get it from the etl repo:

.venv/bin/python -c "from etl.config import get_container_name; print(get_container_name('<etl-branch>'))"

Then, in owid-grapher, use that value (it already carries the staging-site- prefix):

ETL_REGIONS_URL="http://<container-name>.tail6e23.ts.net:8881/external/owid_grapher/latest/regions/regions.csv" \
  yarn runRegionsUpdater

Use this for previews and for the color review only — never merge the grapher PR on a staging-derived regions.data.ts. Once the regions are live on prod, re-run yarn runRegionsUpdater with no env var so the committed file is prod-derived, and push that regeneration as the last commit (see Renaming existing regions for why a stale/hand-edited regions.data.ts is a trap). The colors don't change in that regeneration — only the region data does.

Work in the owid-grapher repo on a new branch. There are two ways to register a provider — regionGroupLabels in RegionGroups.ts documents the split in its own comments: "…where we have region definition about what constitutes these regions in regions.ts" vs "…we don't have region definitions … (we recognize them by their suffix)". They correspond to the RegionDataProvider and AdditionalRegionDataProvider types.

Full-definition provider — a RegionDataProvider

The provider's regions (with member countries) live in regions.data.ts, so it's a RegionDataProvider. This is the natural outcome once you've added the provider to the ETL regions dataset (Steps 4–8), and it's what gives both entity-selector grouping and hover tooltips. Use it whenever the regions are formally defined.

1. Regenerate regions.data.ts:

   yarn runRegionsUpdater

   This fetches the catalog CSV and rewrites packages/@ourworldindata/utils/src/regions/regions.data.ts (auto-generated — never hand-edit). The new PROVIDER_XXX aggregates appear with definedBy: "<provider>". The RegionDataProvider / RegionGroupKey / TooltipKey union types are derived from this file, so the provider key becomes known to the type system automatically.

2. Add the hand-maintained labels (TypeScript will fail to compile until each Record<RegionDataProvider, …> has an entry — that's your checklist):

   • adminSiteClient/EntityPresets.ts → REGION_DATA_PROVIDER_LABELS: <provider>: "<Short> regions" (short, for the admin dropdown).
   • packages/@ourworldindata/grapher/src/core/RegionGroups.ts → regionGroupLabels: <provider>: "<Provider full name> regions" (in the "we have region definitions" group).
   • packages/@ourworldindata/grapher/src/seriesLabel/RegionTooltipData.ts → descriptions: <provider>: "The **<Provider>** defines [N world regions](https://ourworldindata.org/world-region-map-definitions#<anchor>):" — embed the article link in the count phrase, matching the WB/WHO/UN entries. <anchor> is the slug of the provider's article-section heading (e.g. maddison-project-database-maddison); you must add that section to the article — see Step 10. Optionally add a left-to-right map order to customRegionDisplayOrder (omit → alphabetical). See "The region hover" below for what these edits drive.

Suffix-only provider — an AdditionalRegionDataProvider (e.g. FAO, OECD)

The frontend recognizes the provider's entities purely by the (Provider) name suffix; no member definitions in regions.data.ts. Use this only when the regions are not in the ETL regions dataset — it's lighter, but there's no hover tooltip and no member validation.

• packages/@ourworldindata/grapher/src/core/GrapherConstants.ts → add the slug to ADDITIONAL_REGION_DATA_PROVIDERS (this defines the AdditionalRegionDataProvider type).
• adminSiteClient/EntityPresets.ts → ADDITIONAL_REGION_DATA_PROVIDER_LABELS: <provider>: "<Short> regions".
• packages/@ourworldindata/grapher/src/core/RegionGroups.ts → regionGroupLabels: <provider>: "<Provider full name> regions" (in the "recognize by suffix" group).
• No RegionTooltipData entry (its TooltipKey only covers full-definition providers).

Multi-level full-definition providers define one RegionDataProvider per level (e.g. un_m49_1/2/3; ILO uses ilo_1/ilo_2) — each level gets its own label, sort order, and tooltip. Mirror the un_m49_1/2/3 entries in RegionGroups.ts.

Don't drop the bare suffix slug. parseLabel resolves an entity to a provider by its (Provider) name suffix — lowercased, spaces stripped — not by definedBy. So when the suffix doesn't match the per-level keys (e.g. "Northern Africa (ILO)" → ilo, not ilo_1), the bare slug (ilo) must also stay registered in ADDITIONAL_REGION_DATA_PROVIDERS + its ADDITIONAL_REGION_DATA_PROVIDER_LABELS and regionGroupLabels entries — it's the recognition handle. Drop it and providerKey comes back undefined, which silently breaks both entity-selector grouping and the hover even though the per-level definitions exist. Keep it alongside the per-level providers, exactly as unm49 sits beside un_m49_1/2/3. Single-tier providers whose suffix already equals their key (Maddison → maddison, WID → wid) need no extra entry.

The region hover (tooltip) — full-definition providers only, fully data-driven

When a Grapher chart plots a region entity (e.g. "Sub-Saharan Africa (ILO)") as a series, hovering its label shows a tooltip with a description, a mini world map, and a legend (RegionTooltip.tsx → RegionMap.tsx). Worth understanding because it surprises people:

• It is NOT tied to any published chart and renders entirely from regions.data.ts + the registries — you don't need to publish a chart for hovers to work. The descriptions text does link into the world-region-map-definitions article (an anchor), so add the matching section to that article (Step 10) or the link lands on the page top. The tooltip itself renders regardless.
• The mini-map's configuration is computed in code, not taken from your ETL metadata or the chart's customCategoryColors:
  • Membership (which country → which region) comes from regions.data.ts (getCountriesByRegion); no-data countries fall back to gray.
  • Geometry is owid-grapher's bundled world geojson (getGeoFeaturesForMap).
  • Colors come from getRegionsForKey, which looks up MapContinentColors[regionName] first and only falls back to CategoricalMapPalette17[index] for regions that aren't pinned (where index is the region's position in customRegionDisplayOrder[<provider>] in RegionTooltipData.ts — or alphabetical if you omit it). Pin the provider's regions (next section) and the hover follows automatically.
• So for a full-definition provider the hover is two required hand-edits in RegionTooltipData.ts — descriptions[<provider>] (text + article link) and optionally customRegionDisplayOrder[<provider>] (left-to-right map order, which fixes the legend order and the fallback palette assignment) — plus the color step below.
• Which map a region shows is its own defined_by tier, because regionIconInfo returns tooltipKey: region.definedBy (in SeriesLabelState.ts) — not the name suffix. So for a multi-tier provider each tier needs its own descriptions[<provider>_1] / descriptions[<provider>_2] entry, and a region tagged <provider>_1 always hovers to the level-1 map, one tagged <provider>_2 to the level-2 map.

Shared-region caveat (multi-tier providers). A region that exists in two tiers carries a single defined_by, so its hover only ever shows that one tier's map — the indicator-level back-fill (Step 3 / Step 6c) populates the other tier's map indicator but does not give the entity a second defined_by in regions.data.ts. Concrete consequence: in a chart built on the level-2 indicator, the shared region hovers to the level-1 map while its level-2 siblings hover to the level-2 map. It's not wrong (the level-1 map still highlights it), but the "belongs to a set of N regions" framing differs for that one entity. There's no way to make it show both — tagging it the other tier just flips which map it shows. (e.g. ILO's Arab States, tagged ilo_1, always hovers to the 5-region broad map.)

And it leaves a gray hole in the other tier's hover map — fix with a frontend back-fill. Because the shared region is absent from the other tier's provider set in regions.data.ts (getAggregatesByProvider filters by exact definedBy), its member countries are unmapped there and render gray (no-data) whenever you hover another region of that tier. Concretely: hovering any ilo_2 subregion grays out the 12 Arab States countries, and the legend shows 10 regions instead of 11 — even though the ETL ilo_2_region indicator (and its published map) shows Arab States colored, because the ETL back-fill never reaches regions.data.ts. Mirror that back-fill centrally, in getAggregatesByProvider (regionsUtils.ts), via a data-driven map — PROVIDER_REGION_BACKFILLS = { ilo_2: ["Arab States (ILO)"] }, appended to the direct definedBy matches. Do it there, not in getRegionsForKey — getAggregatesByProvider feeds every consumer of the sub-tier (the hover and the admin entity presets, …), so a fix in getRegionsForKey alone leaves the presets still dropping the region. Add the shared region to customRegionDisplayOrder[<tier>] for a stable legend slot. Then that tier's hover is a complete partition matching the published map. (The shared region itself still hovers to its home tier — this only fills the hole the other regions' map would otherwise have.)

Region colors — two dictionaries, two jobs

Colors are not an ETL artifact. Nothing you can write in regions.meta.override.yml sets a region's color. The ETL side sets exactly one thing — baseColorScheme: OwidCategoricalMap (Step 6b), which decides that colors are looked up by region name. The color values themselves are TypeScript in owid-grapher, so the whole color conversation happens in the Step 9 PR, on an owid-grapher branch:

So the answer to "where do I show the colors while I'm working in ETL?" is: you don't — you open the grapher branch and show them from its staging server. You don't have to wait for the ETL PR to merge to do that; regenerate regions.data.ts from your ETL branch's staging catalog (see the ETL_REGIONS_URL note at the top of Step 9), push the grapher branch, and its test page renders the new provider with your proposed colors.

Both dictionaries live in packages/@ourworldindata/grapher/src/color/CustomSchemes.ts, and every provider needs an entry in both:

Colors are looked up by region name, not by legend position (ColorScale.ts picks colorScheme.colorMap[value] before falling back to the positional palette), so the assignment is stable no matter how the sort order changes later. Precedence, highest first: a chart's/indicator's customCategoryColors → the scheme's colorMap → the positional palette. That's why Step 6b bans customCategoryColors: it silently defeats everything below it.

The two are not interchangeable, and neither one covers for the other. MapContinentColors spreads ...ContinentColors, so a region added to only ContinentColors still renders on maps — in the strong chart color, which is the thing the muted map vocabulary exists to avoid. And the admin color picker reads Object.entries(ContinentColors) directly, so a region added to only MapContinentColors never appears there. Always add both blocks, in the same region order, each headed // <Provider> regions.

Pick the colors by analogy with what's already curated. The hues below are the house convention, read off the providers already in the file — a region should get roughly the same color everywhere it appears:

• Mirror the nearest already-curated provider. Before consulting the table, find the provider whose regions are closest in composition and count and copy its assignment wholesale, noting it in the comment — the file already does this (// IEA regions (mirroring the Energy Institute regions), // FAO SDG regions (mirroring the UN SDG regions)). Two providers that carve the world the same way should look the same.
• No two regions of one provider may share a color. Where the table would collide, move to a neighboring hue or an extended map color, and prefer keeping the broadest region on the canonical hue. Precedents: UN M49 pushes South-eastern Asia to DarkOrange/LightOrange to clear Eastern Asia's green; Maddison's East Asia is Copper/LightCherry because its greens are taken by South and South East Asia.
• Multi-tier providers get pinned too. Because colors are name-keyed, a region shared across tiers (e.g. Arab States (ILO), tagged ilo_1 but back-filled into the ilo_2 map) carries one color that is correct in both tiers. Pin every region of every tier; check the tiers for collisions independently, since each tier is its own legend.
• Use the named constants, never raw hex — "<Region> (<Provider>)": OwidDistinctColors.Peach / OwidMapColors.SoftOrange. OwidMapColors is declared above both dictionaries, so it's available to each.
• Keep the comments to labels. These dictionaries are dense lists, and their section headings are one line naming the block — // FAO subregions (level 2), // UN M49 regions (level 3) — phrased like the headings already around them, tier number included. Resist explaining the choices in the file: why this provider mirrors that one, which region is listed in a different block, how many hues a continent's split needed. That reasoning is real and worth writing, but it belongs in the PR description and the commit message, where it is read once by a reviewer, not in a file that is read whenever someone adds a region.
• The one comment worth keeping is the one that stops a future edit going wrong — e.g. that new colors are deliberately outside the CategoricalMapPalette sets. Even that is a clause, not a paragraph.
• New palette colors go after the existing groups, at the end of OwidMapColors, never inserted between the named ones. Slotting a color next to Extended or Main implies it belongs to the positional palette sets those groups feed, which is exactly what it must not do.