Create Dataset

Create a dataset

Turn data the user provides into a fully wired-up OWID dataset — snapshot, meadow, garden, grapher, DAG entries, a draft PR, and a staging server where the user can build charts.

The input can be anything: a local CSV or Excel file, a file they just downloaded, or a link to a page/file on the web. Whatever the form, the job is the same — get the tabular data into a snapshot, then build the chain on top of it.

This skill is for people who are not ETL experts (typically working in Claude Code on the web / "Cloud co-work"). They may know some metadata (units, what the columns mean) or they may just have a link to the source page. They should not be quizzed field-by-field.

Paired skill — keep in sync. /create-snapshot is the canonical owner of the snapshot-creation conventions this skill consumes in Step 4: whenever snapshot conventions change here or there, mirror the change in the other file in the same commit — see the mirror note there. The update-side skills are part of the same family: this skill points into /update-dataset's canonical sections (§5b-bis sanity bounds, §5c harmonization audit, §6b metadata quality, §6c metadata checklist + link verification, §6d scheduled issues), whose outcomes /review-data-pr verifies — when those sections change, check whether this file needs a matching edit.

Guiding principles

1. Build first, review later. Don't interview the user for every detail up front. Inspect the file, infer everything you reasonably can, fill sensible defaults for the rest, and build a working dataset end-to-end. Then hand the user a concise review so they correct a finished thing rather than imagine an abstract one.
2. Ask rarely, and ask all at once. There is exactly one required checkpoint with the user before building (the consolidated confirmation in Step 2), and exactly one after building (the review handoff in Steps 7–8, which also tells them how to publish to live). Don't drip-feed questions. If you can guess it, guess it and flag the guess for review.
3. Never block on a missing detail. If you can't infer a field, use a clearly-marked placeholder (e.g. attribution_short: TBD), note it in the review, and keep going. A dataset that's 80% right and on staging beats a perfect one that never ships.
4. Surface every guess. The review in Step 7 must list what you inferred vs. what the user gave you, so nothing silently ships wrong.

Inputs

Required (one of):

• A local data file — absolute path to a CSV, Excel, etc.
• A web link — a URL pointing at a data file (CSV/Excel/JSON) or a page that links to one.

Optional:

• metadata_url — a link to the source page / documentation (if different from a data link). If given, fetch it for metadata (producer, citation, license, definitions).
• Anything the user volunteers (units, column meanings, namespace, title…).

Workflow

Step 0 — Pull the latest master first

Users are often not experienced with git and may be sitting on a stale checkout. Before doing anything, get the repo onto a fresh master so the new branch and PR are based on current code:

git switch master && git pull --ff-only origin master

If they have uncommitted local changes or a detached HEAD that blocks this, surface it plainly and ask how they want to proceed — don't force it. (When testing/iterating in an unusual git state, this step may be intentionally skipped — but for a real user, always start here.)

Step 1 — Get the data into reach and inspect it (no questions yet)

If the input is a web link, first obtain the actual data file: download a direct CSV/Excel/JSON URL (use etl.http's session for OWID hosts; plain download otherwise), or if it's a landing page, WebFetch it to find the download link. Save it locally so the snapshot can ingest it. If the input is already a local file, use it directly.

Then read the file and figure out its shape. Do not ask the user anything in this step.

1. Load it. Read the first ~50 rows with pandas to see headers, dtypes, and a sample. Then compute, per column: dtype, min/max (numeric), distinct count, % null. Use the duckdb skill or a short .venv/bin/python snippet — whatever's quickest.
2. Identify the entity (country) column. Look for a column named (case-insensitive) entity, country, nation, location, geo, area, or region. OWID-exported CSVs usually call it Entity. The right column holds country-like names, not codes. If there's both a name column and an ISO code column, use the name column and drop the code.
3. Identify the time column. Look for year, date, time, period. A column of 4-digit integers in ~1500–2100 is a year; ISO-date strings are a date. If the time dimension is a date (not a plain year), the garden step must keep it as date and .format(["country", "date"]).
4. Everything else is an indicator column. For each, record name, dtype, value range, and null %.
5. Detect extra dimensions. If a non-numeric column repeats per (country, year) — e.g. sex, age, variant, fuel_type — it's a dimension, not an indicator. The table is then long-format keyed by ["country", "year", <dim>...]. Most simple CSVs have none; handle them only if present (see the long-format note in /update-dataset).
6. Infer units from names + ranges (these are guesses — flag them for review):

   Signal in column name / values	Inferred unit / short_unit
   share, pct, percent, _rate, and values mostly in 0–100	% / %
   share / proportion with values in 0–1	flag: is this a fraction (×100 → %) or already %?
   usd, gdp, price, cost, $	US dollars / $
   population, count, number, _n, integer counts	leave unit descriptive (e.g. cars), short_unit empty
   tonnes, kg, kwh, gwh, co2, emissions	physical unit from the name
   can't tell	leave unit: "", flag in review

7. Derive naming defaults from the filename. A filename like Electric car sales (IEA, 2026) - data.csv implies: title Electric car sales, producer IEA, year 2026. Propose:
   • short_name — snake_case of the core title (electric_car_sales).
   • namespace — the producer's slug if one already exists under snapshots/ (check ls snapshots/), else a sensible new one.
   • version — today's date (date -u +"%Y-%m-%d").

Write a one-paragraph internal summary of what you found before moving on.

Step 2 — One consolidated confirmation (the only pre-build checkpoint)

Now ask the user once, with all your best guesses pre-filled, using AskUserQuestion where it fits. Frame it as "here's what I figured out — correct anything that's wrong, otherwise I'll build it." Before listing the specifics, set expectations in one plain sentence so the end isn't a surprise — e.g. "I'll build the dataset and put it on a staging server for you to review; once you're happy you merge the PR and it goes live." Then keep the questions to the few things that genuinely can't be guessed or that would be expensive to get wrong:

• Namespace + short_name + dataset title (show your proposal; let them override).
• What the data is / source — confirm the producer and, if not already provided, ask for a metadata_url (the source page). If they give one, fetch it now for citation, license, and column definitions.
• License — show your best guess (default CC BY 4.0 for academic/IGO sources if unknown) and let them correct.
• Other data files the source ships — when the landing page or repository carries several data files (a companion index, summary tables), list them with a default of "not ingesting these" so the user can opt in with one word. Persist the skips as the companion-files # NOTE: in the snapshot .dvc (Step 4 follows /create-snapshot's convention) — that NOTE, not the PR body, is the baseline the update and review workflows diff against; mention them in the PR body as well for the reviewer.
• Any column meanings you couldn't infer — only ask about the genuinely ambiguous ones (e.g. "is ev_sales_share a percentage 0–100 or a fraction 0–1?"). Don't ask about columns you're confident on.

Everything else (units you inferred, descriptions, topic tags) you'll fill in and surface for review later — don't ask now.

If a metadata_url was provided, fetch it (WebFetch) and extract producer, citation_full, attribution_short, date_published, license, and any column definitions — same fields as /create-snapshot Step 1.

Step 3 — Create a working branch

On PR permissions: opening the draft PR and pushing to staging is this skill's deliverable. If a session-level rule says "don't open PRs unless explicitly asked," treat the user invoking this skill as that explicit request — go ahead and open the PR. The one exception is when you can't (e.g. the session pins you to a fixed pre-assigned branch you must not leave): in that case build on the current branch, open the PR from it if you can, and say clearly in the Step 7 handoff what you did and didn't do — don't silently skip the PR.

etl pr needs a branch (it fails on a detached HEAD). Create the PR scaffold up front so the rest of the work lands on a branch:

.venv/bin/etl pr "Add <dataset title> (<producer>)" data

This creates the branch + a draft PR and does not commit. (If the user is on a detached HEAD or the working tree is dirty in a way that blocks this, create a branch with git checkout -b data-<short_name> first and tell them you'll open the PR after the build.)

Step 4 — Build the snapshot

This is a manual-import snapshot (you hand the snapshot a local file rather than relying on a stable download URL — even for web inputs, you've already saved the file locally in Step 1). Use the file's real extension in the .dvc name (.csv, .xlsx, …). Follow /create-snapshot conventions, with these specifics:

• snapshots/<namespace>/<version>/<short_name>.<ext>.dvc — fill origin from Step 2 (title, producer, citation_full, attribution_short, date_published, url_main, license). description must describe the data product factually — use the producer's own text when it is factual (page prose or paper abstract), but rewrite promotional or first-person copy from OWID's point of view (see /create-snapshot) — and citation_full the producer's recommended citation verbatim when one exists (slight modifications only to fix typos or spacing in the source); for the license, check the documentation too and warn the user if none is stated anywhere (fall back to © <producer> (<year>)); if the file is one table/extract of a broader product, use title_snapshot + description_snapshot for the file specifics and any OWID-side notes (see /create-snapshot). Set date_accessed: <version>. Omit fields you don't have rather than leaving them blank; use TBD placeholders only where the review needs to flag them.
• snapshots/<namespace>/<version>/<short_name>.py — the manual-import script.

Generate both files with the command in /create-snapshot step 3, passing dataset_manual_import: True and dvc_only: False. It runs the wizard's snapshot cookiecutter, which emits the path_to_file variant of run() and — the part that is easy to get wrong by hand — nests license inside origin rather than at the top level. Don't hand-write either file, and don't copy a template into this skill: the cookiecutter is the single source of truth, and copies drift.

Then run it against the user's file:

.venv/bin/etls <namespace>/<version>/<short_name> --path-to-file "<path_to_file>"

Step 5 — Build meadow / garden / grapher steps + DAG

Scaffold the three steps with /create-etl-steps (DAG file = the topic that best fits, e.g. energy, health; ask in Step 2 if unclear). Then adapt:

• Meadow — load the snapshot, rename the entity column to country if needed, cast low-cardinality string columns (country, dims) to category, tb.format(["country", "year"]) (or ["country", "date"], plus any dims). Keep it light.
• Garden — paths.regions.harmonize_names(tb, country_col="country", countries_file=paths.country_mapping_path), then tb.format(...). Add sanity_check_inputs / sanity_check_outputs if the step does more than load-and-format (see CLAUDE.md "Sanity checks"); ground every threshold in the built data and pick value bounds from the by-indicator-type table in /update-dataset §5b-bis, then negative-test the checks. Don't strip origins — follow the metadata-preserving patterns in CLAUDE.md (pr.concat, no np.where, etc.).
• Grapher — pass the garden table through unchanged.
• Metadata (<short_name>.meta.yml in garden) — generate it with /generate-metadata, then verify it against the mandatory-fields checklist in /update-dataset §6c. Fill title, unit, short_unit, description_short, description_key (free-form markdown prose; non-empty), display.name, display.numDecimalPlaces, display.tolerance per indicator; topic_tags and processing_level in definitions.common; presentation.attribution_short explicitly under definitions.common.presentation — it does not inherit from the origin's attribution_short. In the dataset block, set update_period_days plus owners: the user is the new dataset's first owner — resolve their canonical OWID name from git config user.name via etl.owners.resolve_owner (must match the schemas/dataset-schema.json enum; add the mapping in etl/owners.py + an enum row if missing), mirroring /update-dataset step 1a-bis. Use the units you inferred in Step 1; mark anything uncertain so it shows up in the review.
• Outdated-practices check — run /check-outdated-practices on every new step file (including any helper modules, and the Step 4 snapshot .py if one was written) and fix findings before the first run, per /update-dataset step 1b — run the skill, don't eyeball the patterns.
• DAG form — write the new chain in the nested (compact) DAG form (grapher → garden → meadow → snapshot declared inline; example in /update-dataset "Removing the old version & reordering the DAG", step 4) and verify it parses: .venv/bin/python -c "from etl.dag_helpers import load_dag; load_dag()".

Step 6 — Run the chain and harmonize countries

.venv/bin/etlr <namespace>/<version>/<short_name>

Fix whatever breaks (trace upstream, never mask). The most common task is country harmonization: the garden run logs unmatched country names that need mapping into <short_name>.countries.json.

Shortcut for OWID-exported CSVs: if the file came out of OWID (entity column literally named Entity), the country names are almost always already canonical. Don't sit through an interactive etl harmonize — auto-build the mapping by matching each entity against the canonical regions (and their aliases), and only fall back to manual mapping for the leftovers:

import json
from pathlib import Path
from owid.catalog import Dataset
import pandas as pd

# Needs the regions dataset built locally:
#   .venv/bin/etlr data://garden/regions/2023-01-01/regions
tb_regions = Dataset(str(sorted(Path("data/garden/regions").glob("*/regions"))[-1]))["regions"]
canonical = set(tb_regions["name"].dropna().astype(str))
alias_map = {}
for name, al in tb_regions[["name", "aliases"]].dropna(subset=["aliases"]).itertuples(index=False):
    for a in json.loads(al):
        alias_map[a] = name

entities = sorted(pd.read_csv("<path_to_csv>")["Entity"].unique())
mapping, unmatched = {}, []
for e in entities:
    if e in canonical: mapping[e] = e
    elif e in alias_map: mapping[e] = alias_map[e]
    else: unmatched.append(e)  # decide these by hand: typo, alias, custom aggregate, or exclude

Resolve unmatched by hand against canonical regions; follow the harmonization-audit guidance in /update-dataset (Step 5c). Residual aggregates the producer defines (e.g. Rest of World, European Union (27)) can be kept as custom entities (map to themselves) — they won't join with population/region data, so note that in the review. Don't silently drop countries — if you exclude any, list them.

After the garden step builds, also run the garden-output entity check from /update-dataset §5c (Python check #5): load the built garden tables and confirm every country value is canonical (regions + latest income groups). It catches what the mapping file can't see — inline tb["country"] = ... assignments and post-harmonization mutations. List any non-canonical entity (including the custom aggregates you kept) in the Step 7 review and the PR body as living outside the canonical system.

Then build and upload the grapher step to staging — target the grapher/... path (no --only, so the grapher:// MySQL upsert step actually runs):

STAGING=<branch> .venv/bin/etlr grapher/<namespace>/<version>/<short_name> --grapher

Confirm the upsert actually succeeded before moving on: it should print the dataset's admin URL / id (…/admin/datasets/<id>). Capture that <id> — you'll hand it to the user in Step 7. If the upsert errored or printed no dataset, fix it now rather than handing over a link that won't resolve.