Create Explorer

Step 6 — Post-processing the chart (table-driven only)

After paths.create_explorer() returns the explorer c, you can mutate it before c.save():

• c.sort_choices({dim_slug: lambda x: sorted(x)}) — control the order of dimension dropdowns. Useful when slugs sort poorly alphabetically.

• c.group_views(groups=[...]) — bundle multiple existing views into a new multi-indicator view. Each entry in groups:

  • dimension: the dimension whose choices are being collapsed.
  • choices: the choice slugs to combine (omit for all).
  • choice_new_slug: name for the new collapsed choice (e.g. combined, total, breakdown).
  • view_config: chart-level config for the new view (chartTypes, title, subtitle, selectedFacetStrategy, …). Title can be a template like "Population aged {age}" evaluated against params.
  • view_metadata: data-page metadata (description_key, etc.) for the new view.
  • replace=True to drop the originals; default keeps both.
  • overwrite_dimension_choice=True if choice_new_slug collides with an existing choice and you want grouped views to win.
  • Use cases: an explorer with sex={female, male} views — group_views adds sex=combined showing both timeseries on one chart. Same pattern works for age brackets, region groups, conflict types, or any dimension where users may want a single multi-line chart.

• c.edit_views([...]) — apply chart-level config to many views at once, optionally scoped by dimension. Each entry is {"dimensions": <filter>, "config": {...}, "metadata": {...}}; the framework merges entries by specificity (more dimensions in the filter = wins on conflicts). Use this in preference to set_global_config whenever you have per-slice overrides:

  c.edit_views([
      # No filter → applies to every view (defaults).
      {"config": {"type": "LineChart DiscreteBar", "hasMapTab": True}},
      # Scoped override — only this exact (gas, accounting, fuel, count) cell gets a Slope tab.
      {
          "dimensions": {"gas": "co2", "accounting": "territorial", "fuel": "all_fossil", "count": "per_capita"},
          "config": {"type": "LineChart SlopeChart DiscreteBar"},
      },
  ])

  Callable values inside config (e.g. "title": lambda v: ...) are evaluated against each matching view, so dimension-aware text templates still work. set_global_config is just a one-entry shortcut for edit_views; reach for edit_views once you have more than one slice to address.

• Manual loop over c.views — for things edit_views can't reach: per-indicator display blocks (numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, display.color, …). These live on view.indicators.y[i].display, not on view.config. Match views via the view.matches(**kwargs) helper, which accepts a single value or a list (list = OR semantics):

  for view in c.views:
      if view.matches(gas=["methane", "all_ghg"], count="per_capita"):
          decimals = 1
      elif view.matches(gas="warming_impact", fuel=["land_use", "fossil_plus_land_use"]):
          decimals = 3
      else:
          continue
      for indicator in view.indicators.y or []:
          indicator.display = {**(indicator.display or {}), "numDecimalPlaces": decimals}

  Pattern: migration_flows.py's add_display_settings(c). Avoid this loop when the setting can live on the indicator's garden metadata instead.

• choice_renames={dim: {slug: display_name, ...}} (passed directly to create_explorer) — map slug → display name when you need to derive the display label programmatically. Model: chart/minerals/latest/minerals.py.

• Sidecar <short>.dims.yaml — when the column → dimensions map exceeds ~50 entries, lift it out of the Python step into a sidecar YAML loaded at module-import time. Keeps <short>.py focused on logic and turns dim-tagging changes into a 1-line YAML edit. Model: etl/steps/viz/explorer/emissions/latest/co2.{py,dims.yaml}:

  from pathlib import Path
  import yaml
  COLUMN_DIMENSIONS = yaml.safe_load((Path(__file__).parent / "co2.dims.yaml").read_text())

Step 7 — DAG entry

In dag/<ns>.yml:

viz://explorer/<ns>/latest/<short>:
  - data://grapher/<ns1>/<v1>/<dataset1>
  - data://grapher/<ns2>/<v2>/<dataset2>
  # ... one line per unique upstream grapher dataset

Place near related explorer entries (or alongside the upstream grapher steps) for discoverability.

Step 8 — Verify

Hand off to the user:

1. .venv/bin/etlr viz://explorer/<ns>/latest/<short> --grapher — runs the step and upserts the explorer to the staging DB.
2. Open http://staging-site-<branch>/admin/explorers/preview/<slug> and spot-check:
   • default view (no dimensions toggled)
   • every dimension switch
   • map tab (if applicable)
   • country/entity picker
   • default selection
3. For migrations: diff the resulting TSV against the legacy owid-grapher/explorers/<slug>.explorer.tsv. Cosmetic differences (column ordering, whitespace) are acceptable; structural differences (missing views, swapped dimension orderings) are not. The Wizard's apps/wizard/app_pages/explorer_diff/ page does this comparison interactively for staging vs production.
4. make check.

Hand the user the exact etlr command — don't run it yourself.

Common pitfalls

• build_views does not propagate per-view display from indicator metadata — see TODO at etl/viz/chart/core/expand.py:313. Each auto-expanded view gets indicators.y[0] with only catalogPath, no display. If you need different color scales per view, either (a) put them in the indicator's presentation.grapher_config.map.colorScale so they apply at chart render time, or (b) post-process c.views in Python.
• type: LineChart is not a valid grapher_config field when authoring via indicator metadata — use chartTypes: ["LineChart"] (the schema is an array). Per-view config.type in the explorer YAML still accepts strings.
• tb.read(..., load_data=False) is essential when you only need column metadata to set dimensions; loading data unnecessarily slows the step.
• YAML schema requires views: key in the explorer config even when empty. Pass views: [].
• Indicator must be re-published to MySQL with the new grapher_config before the explorer view can inherit it. Re-run etlr --grapher data://grapher/... after editing garden metadata; the explorer step alone won't refresh the indicator's stored config.
• YAML anchors / merge keys (&common_view, <<: *common_view) — don't use them. They can't filter by dimension and add per-view noise. Use definitions.common_views instead.
• Hyphens in short_name — file is food_footprints.py but short_name="food-footprints". Mismatch produces a published explorer whose URL doesn't match the legacy slug.
• tolerate_extra_indicators — usually want True for explorers since you're cherry-picking indicators from larger upstream datasets.
• Checkbox dimensions must have exactly 2 choices. The na pattern (3 choices: na, off, on) works for radio and dropdown but not checkbox — the framework rejects it with Dimension choices for 'checkbox' must have exactly two choices. If you genuinely need a third "doesn't apply" state, either use a radio with three choices, or drop the na slot and let the framework hide/disable the toggle when the current dimension state has no matching <true> view.
• YAML 1.1 booleanizes unquoted no/yes/on/off/true/false slugs. A view.dimensions: { relative_to_world: no } reads back as {"relative_to_world": False}, which won't match the string slug "no" declared in dimensions[*].choices. Use semantic slugs (total/relative, absolute/share) or quote the strings — but choosing different slug names is the more durable fix.
• edit_views doesn't reach indicator-level display. Fields like numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, and per-indicator color live on view.indicators.y[i].display, not on view.config. edit_views only writes view-level config/metadata. For these, fall back to a c.views loop (see Step 6) or push them into the indicator's garden presentation.grapher_config so they apply at chart render time.

Reference examples

Full-YAML (each view hand-listed):

• etl/steps/viz/explorer/agriculture/latest/crop_yields.{py,config.yml} — large indicator-based explorer with many dimensions.
• etl/steps/viz/explorer/agriculture/latest/food_prices.{py,config.yml} — small grapher-chart-based migration (12 chart IDs unwrapped to 12 single-indicator views) with conditional dimensions ("na" pattern).
• etl/steps/viz/explorer/agriculture/latest/fertilizers.{py,config.yml} — checkbox dimension with choice_slug_true; multi-namespace dependencies.
• etl/steps/viz/explorer/food/latest/food_footprints.{py,config.yml} — hybrid (16 grapher-chart views + 29 CSV-backed views) showing dimension-filtered common_views for differing sourceDesc per view-type.
• etl/steps/viz/explorer/war/latest/countries_in_conflict_data.{py,config.yml} — uses na-named choices to model conditional dimensions.
• etl/steps/viz/explorer/emissions/latest/ipcc_scenarios.{py,config.yml} — moderate-size YAML-driven.

Table-driven (views auto-expanded from a dimensional table):

• etl/steps/viz/explorer/migration/latest/migration_flows.{py,config.yml} — passes tb=tb, indicator_names=[...], dimensions=[...] to create_explorer; YAML carries only the static config and dimension presentation. Includes add_display_settings(c) post-processing.
• etl/steps/viz/explorer/emissions/latest/co2.{py,dims.yaml,config.yml} — sidecar .dims.yaml for the 54-entry column→dimensions map; uses c.edit_views([...]) with both an unscoped default and a 4-dim-filtered override; uses a c.views loop with view.matches(...) for per-indicator numDecimalPlaces overrides.
• etl/steps/viz/explorer/emissions/latest/air_pollution.{py,config.yml} — table-driven with c.group_views(...) to add facet views, c.drop_views(...) to prune cross-products.
• etl/steps/viz/chart/minerals/latest/minerals.py — same APIs in the multidim channel; useful read for choice_renames.

Follow-up

Once an explorer is on paths.create_explorer(), it's a candidate for the Track-B port to MDIM (viz://chart/...) once feature parity is reached. See umbrella issue #6014.