Generate Metadata

OWID Metadata Generation

Practical guidelines for writing high-quality metadata in *.meta.yml files.

Core principle: Metadata is for the public. Every field should help someone understand the data. Write in plain language -- if a layperson can't understand it, rewrite it.

When to Use

• New garden/grapher datasets needing metadata
• Enriching incomplete or sparse metadata files
• Updating existing metadata for dataset refreshes

Don't use for: Snapshot metadata (different process) or quick single-field edits.

Canonical YAML Structure

definitions:
  common:
    processing_level: minor  # or major
    presentation:
      topic_tags:
        - <Topic>
      attribution_short: <Short Source Name>
    display:
      numDecimalPlaces: 1
  # Reusable text blocks
  my_note: &my_note |-
    Reusable text.

dataset:
  update_period_days: 365  # Must be accurate: 0 for datasets that will never update

tables:
  <table_name>:
    variables:
      <variable_name>:
        title: <Human-readable title>
        unit: <Full unit name>
        short_unit: <Symbol>
        description_short: |-
          <1-2 sentences>
        description_key: |-
          <Key information as free-form markdown: paragraphs, with sub-lists where helpful>
        description_from_producer: |-
          <Original text from source>
        description_processing: |-
          <What OWID did to process this>
        display:
          name: <Legend label>
        presentation:
          title_public: <Public-facing title>
          title_variant: <Disambiguating label>

Key rule from CLAUDE.md: The dataset: block carries only update_period_days and owners -- everything else is inherited from origin. Always set owners when it's missing (canonical name from the schemas/dataset-schema.json enum, resolved via etl.owners.resolve_owner; first entry = accountable owner).

Full field reference: For a complete list of all supported metadata fields (beyond what's covered here), read schemas/dataset-schema.json.

Field-by-Field Guidelines

Titles

Rules:

• Sentence-case, no trailing period
• Never include producer, year, or units in titles
• When display.name is set, also set title_public (see title hierarchy in docs/architecture/metadata/faqs.md)
• Prefer NOT setting title_public unless the title has dimension breakdowns or codes (e.g. SDG indicator numbers). The data page will show the curated chart title instead, which is usually better. Use display.name for cleaner legend/table labels.
• Prefer clear, reader-friendly titles over technically precise ones (e.g. "Share of people who think vaccines are safe" over "Share who strongly agree that...")
• title_variant disambiguates when multiple indicators share a similar title — use short phrases like "Historical data", "WHO", "Age-standardized", "Extrapolated". Watch for redundancy with attribution_short (avoid "V-Dem - V-Dem" duplication).

# GOOD
title: Number of neutron star mergers in the Milky Way
display:
  name: Neutron star mergers
# BAD
title: Number of neutron star mergers (NASA, 2023)

Units

• Always set unit explicitly, even to "" for dimensionless indicators (scores, indexes)
• short_unit is only needed when there's an actual unit to abbreviate. Omit it for dimensionless indicators -- it defaults to None and grapher won't show a unit label.
• Use "person" not "capita" (kilowatts per person)
• Choose human-friendly scales -- if most values are below 1 tonne per person, use "kilograms per person" instead
• short_unit should use SI abbreviations (g not grams, % not pct)

Decimal precision: 0 for counts, 1 for percentages, 2 for economic/per-capita values. Be consistent across related variables. Always set numDecimalPlaces explicitly -- it's a frequent source of review feedback.

Descriptions

description_short -- 1-2 sentences, ~200 characters ideally. Answers "What does this number measure?" Longer explanations belong in description_key.

• Use |- block scalar for multi-line; inline strings are fine for single sentences
• Don't mention units, sources, or processing (redundant or belongs elsewhere)
• Don't just repeat the title -- if it adds nothing beyond the title, omit it entirely
• Remove filler phrases like "Emissions are..." at the start
• Supports Markdown: [text](url) for links, [term](#dod:term) for OWID definition popups (e.g. [stunted](#dod:stunting))

# GOOD
description_short: |-
  The number of people living in extreme poverty, defined as living on less than $2.15 per day.
# BAD - repeats the title
description_short: |-
  Manufactured cigarettes sold in this country in this year.
# BAD - mentions sources (belongs in description_key)
description_short: |-
  The number of people living in extreme poverty, based on data and estimates from different sources.

description_key -- Free-form markdown text for the "About this data" panel: prose paragraphs, with markdown sub-lists only where a list genuinely helps. (A YAML list of bullet points is still accepted and renders as a markdown list, but prefer prose — see grapher's descriptionKey-to-string migration.)

• Plain language, no jargon (e.g. "livestock digestive processes" not "enteric fermentation"). Expand acronyms on first use.
• Add concrete examples for abstract scope descriptions (what countries are included, what events qualify)
• Order: data-specific points first, methodology second, caveats last
• Only describe data that actually exists in the indicator. State implications of limitations explicitly.
• Separate distinct facts with blank lines (paragraphs), so line breaks render meaningfully.
• No redundancy. The bullets are read as a set, directly under description_short and the chart's title/subtitle, so a sentence that repeats another at the same level of detail costs the reader attention and makes the panel look padded. Before adding one, read the whole rendered list and ask what it adds that isn't already there; if the answer is "it says an existing bullet more fully", edit that bullet rather than adding a second. Expanding description_short is not redundancy — the short line is a one-sentence summary, and unpacking it in the first bullet (the full definition, how it's measured, what's included) is exactly what the panel is for. What to avoid is a bullet that restates it and stops there. Two shapes to watch: a source or caveat sentence duplicating a caveat already made in different words, and a Jinja variant bullet restating the shared bullet for some dimension values only.

# GOOD - prose paragraphs, sub-list only where it helps
description_key: |-
  Extreme poverty is measured using the International Poverty Line of $2.15 per day in 2017 international dollars.

  This metric uses household survey data adjusted for purchasing power parity (PPP).
# BAD - unexpanded acronyms, jargon
description_key: |-
  Uses IPL of $2.15/day (2017 PPP).

description_from_producer -- Exact producer text, verbatim or minimally edited. Only if producer provides clear definitions. Can be set in definitions.common when the same producer description applies to all variables, with per-variable overrides as needed.

description_processing -- What OWID did. Only for major transformations (aggregations, per-capita calculations, combining sources). Don't document routine operations (country harmonization, dropping nulls).

• Document dropped data and aggregation choices. Keep in sync with code.
• Don't reference internal dataset names — describe what was done, not which datasets were used.
• Low-visibility field — surface important caveats in description_short or description_key too.

Processing Level and License

• minor: data largely unchanged (reformatting, unit conversion, harmonization) -> use most restrictive origin license
• major: significant transformations (calculations, combinations, imputations) -> use CC BY 4.0
• The test: is the data identical to the original? If not (e.g. processing of historical regions, per-capita calculations), it's major.

Set in definitions.common, override per-variable as needed.

Topic Tags

• 1-3 tags, most relevant first. First tag = primary (used in citations).
• Must match valid tags from the schema. To list them:

  .venv/bin/python -c "import json; tags=json.load(open('schemas/dataset-schema.json'))['properties']['tables']['additionalProperties']['properties']['variables']['additionalProperties']['properties']['presentation']['properties']['topic_tags']['items']['enum']; print('\n'.join(tags))"

• Set in definitions.common.presentation.topic_tags.

Presentation & Display

• presentation.attribution_short: Short source name ("WHO", "World Bank"). Set in common when uniform.
• presentation.grapher_config: Only set when you want a specific default chart view. Common sub-fields:
  • note: Chart footnotes — methodology caveats, sample sizes, inflation adjustments. Keep to 1-2 sentences.
  • selectedEntityNames: Pre-select countries/regions for the default view (e.g. ["United States", "China", "Europe"])
  • selectedEntityColors: Map entity names to hex colors (e.g. {"Africa": "#A2559C", "Asia": "#00847E"})
  • map: Map tab settings — colorScale with baseColorScheme (e.g. "YlOrRd"), binningStrategy ("manual"), and customNumericValues for bin thresholds
  • Set at variable level, or in definitions.common.presentation when all variables share the same chart defaults
• display.numDecimalPlaces: Set explicitly. Use metadata-export --decimals auto to auto-detect.
• display.tolerance: Number of years to allow gap-bridging on line charts (default 0). Set higher (e.g. 5-10) for sparse historical data where connecting distant points is acceptable.
• display.roundingMode: Use "significantFigures" with numSignificantFigures instead of numDecimalPlaces when values span many orders of magnitude.

YAML Efficiency Patterns

Use definitions.common when 3+ variables share the same field values. Remember: common does NOT merge -- it completely overrides. Use <<: *anchor for partial overrides.

Use anchors/aliases for identical blocks shared by 2+ variables. Define in definitions: at the top. Name anchors to indicate their target field (e.g. description_producer_refugee not description_refugee) so reviewers can tell which metadata field the text will end up in.

Use <<: *anchor merge to extend a shared mapping while overriding specific keys (e.g. <<: *common_display then numDecimalPlaces: 0).

Adding text to an existing file: follow the pattern it already uses. If its text lives in definitions: and variables reference {definitions.<key>}, add your text as new definitions at the top next to the related ones — inline prose under the variable renders fine but leaves the file with two authoring styles. In big datasets prefer definitions-at-top regardless of reuse: with hundreds of variables it is what keeps the file readable, since all the prose sits in one place and the variable blocks stay skimmable. Grep the existing definitions first: new text often duplicates a bullet already defined under another name, and reusing (or replacing that key's text, after checking which other variables reference it) beats a near-duplicate. Then widen the grep past the file — boilerplate travels, so the same caveat usually sits in other datasets' .meta.yml too (search distinctive 5–8 word fragments across etl/steps/, since near-duplicates differ by a word or two). When your wording supersedes theirs, propose the same fix there in a separate PR rather than editing another dataset's text inside yours; when a sibling already words it better, adopt that wording instead of minting a third variant. Details and the text-neutrality proof for such refactors are in .claude/skills/edit-faust-metadata/SKILL.md ("Writing new text into a garden .meta.yml").

Use Jinja templates for dimensional datasets (age, sex, cause breakdowns). Custom delimiters: <% %> for blocks, << >> for expressions.

# Jinja example: dimensional variable with conditional descriptions
definitions:
  description_short_time_spent: |-
    <% if who_category == "Alone" %>
    Time spent alone, by gender and age.
    <%- else %>
    Time spent with <<who_category.lower()>>, by gender and age.
    <%- endif %>

variables:
  time_spent:
    title: Time spent with <<who_category>> throughout life
    unit: hours per day
    short_unit: h
    description_short: "{definitions.description_short_time_spent}"
    display:
      name: With <<who_category>>

Write Jinja conditionals in definitions:, with the condition first and each branch on its own line, and call them by name from the variable. This applies whenever the conditional picks a whole value: a full field, or a description_key bullet. Keep the <% if %> out of the variable block itself, so the variable stays a short list of {definitions.…} references. Put each <% if %> / <%- elif %> / <%- else %> / <%- endif %> tag on its own line, with what it renders underneath, so a reviewer reads condition → text without scanning one long line:

definitions:
  description_key_gni_per_capita: GNI per capita is GNI divided by population, …
  description_key_gni_per_capita_by_sex: GNI is not measured separately by sex, …
  description_key_gni_per_capita_total_or_by_sex: |-
    <% if sex == 'total' %>
    {definitions.description_key_gni_per_capita}
    <%- else %>
    {definitions.description_key_gni_per_capita_by_sex}
    <%- endif %>

tables:
  undp_hdr_sex:
    variables:
      gni_pc:
        description_key:
          - "{definitions.description_key_gni}"
          - "{definitions.description_key_gni_per_capita_total_or_by_sex}"

❌ - "<% if sex == 'total' %>{definitions.a}<% else %>{definitions.b}<% endif %>", a one-line conditional inside the variable.

Name the conditional definition after what it chooses between (…_total_or_by_sex), not after one of the branches. Splitting the tags over several lines renders exactly like the one-line form. The Jinja environment sets trim_blocks and lstrip_blocks, and the rendered text is stripped (owid.catalog.core.jinja), so the tag lines and their newlines disappear. {definitions.…} references resolve before Jinja runs, so a definition can wrap other definitions. Each branch must still be a single line, because a wrapped branch puts a real newline into the text.

Dash the closing tags: <%- elif %>, <%- else %>, <%- endif %>. trim_blocks removes the newline after a tag but not the one that ends the branch line before it. As a whole value that newline is stripped anyway. But once the definition is spliced into a longer string (Intro {definitions.x} outro.), the plain form renders Intro A\n outro., and the - is what gives Intro A outro.. Since a definition can be reused anywhere, dash them by default. Leave the opening <% if %> undashed, because <%- there would also eat the space before it and glue the text together. If text follows the conditional, keep it on the <%- endif %> line (<%- endif %> Outro.): on the next line it gets glued on as AOutro..

The exception is a fragment inside a sentence or a title, such as title: GNI per capita<% if sex != "total" %> (<<sex>>)<% endif %>. Splitting that one leaves a line break or stray indentation in the rendered text, so keep it inline. Short fragments like this can still live in a definition (among_sex: <% if sex == "males" %> among men<% elif … %><% endif %>) when several fields reuse them. When you restructure an existing conditional, render every dimension value before and after to confirm nothing changed (the text-neutrality recipe in .claude/skills/edit-faust-metadata/SKILL.md).

A sentence written with one breakdown in mind renders on all the others, where the view's own filtering can make it false. Sweep every dimension value before shipping — check 6 of the quality suite below.

Use {definitions.xxx} string interpolation for reusing text fragments inline (e.g. '{definitions.methodology}' in a description_key bullet). Unlike YAML anchors which substitute entire nodes, this inserts text within strings. Use anchors for whole fields/blocks, interpolation for composing text.

Use shared.meta.yml when multiple .meta.yml files in the same directory share definitions or macros. These files contain only Jinja macros and reusable definitions — no actual variable metadata. Step-level .meta.yml files then import and call these macros. Used in large multi-file datasets like IHME GBD.

For full syntax details, see docs/architecture/metadata/structuring-yaml.md.