Add Ivs Indicators

Add indicators to the values-survey pipeline (IVS / WVS, no version bump)

Adds new question codes to OWID's values-survey pipeline without bumping the version — you extend the current ivs/<version> step in place. The dataset holds two tables, and you pick which one you're adding to:

• integrated_values_surveys — the IVS (WVS + EVS merged trend). Source: the IVS .dta.
• world_values_survey — WVS-only questions (asked in WVS but absent from IVS). Source: the WVS Time-Series .dta.

Both tables live in the same ivs/<v> garden + grapher dataset, so variable titles must be unique across both tables (Step 5/6). The two paths differ in only a handful of places; everything else in this skill is identical. Pick the target first, then read each step with the right column:

Find the active versions: ls etl/steps/data/garden/ivs/ (shared garden/grapher dataset → <v>, e.g. 2025-06-27) and ls snapshots/wvs/ (WVS snapshot/meadow → <wv>, e.g. 2026-06-30).

Why this pipeline is unusual

Neither snapshot is a downloaded file — each is a CSV produced by a Stata script (*_create_file.do) that collapses survey microdata into country×year response shares (we publish only the aggregated shares, not the microdata — this also respects the WVS "no redistribution" license). So adding indicators means editing the .do, having the user run it in Stata (Claude cannot run Stata), then flowing the new columns through meadow → garden → grapher.

End-to-end flow (same for both targets):

edit <ivs|wvs>_create_file.do  →  USER runs it in Stata → regenerates the csv
  → re-snapshot (same version)  →  meadow VARS_DICT  →  garden constants + checks
  →  garden .meta.yml  →  etlr (meadow→garden→grapher)  →  [IVS only: Notion Available flags]

Reference docs (kept out of git — *.pdf/*.xlsx are git-ignored)

Two documents make scale/wording verification easy. They normally sit in snapshots/ivs/<v>/ during an IVS update but are git-ignored, so they're not in the repo. If they aren't in the snapshot folder, ask the user for them (or download from worldvaluessurvey.org → Data and documentation → Data Download):

• Common EVS/WVS dictionary (…Common_EVS_WVS_Dictionary_IVS.xlsx) — maps each IVS code to its WVS-7 / EVS variable name (the Q-number) and label (sheet IVS_EVS_and_WVS_Variables). Path: Data Download → WVS/EVS Trend 1981-2022 → IVS documentation: IVS Common EVS-WVS dictionary.
• WVS-7 master questionnaire (…WVS-7_Master_Questionnaire_…English.pdf) — verbatim question stems + answer-category wording (look up by Q-number). Path: Data Download → Wave 7 (2017-2022) (matches the current IVS version) → Questionnaire link.
• EVS 2017 field questionnaire (ZA7500_q_gb.pdf, the Great Britain master) — the fallback for older EVS-only items that never made it into WVS-7 (so they're absent from the WVS-7 PDF). Look up by the EVS variable name / question number (e.g. the IVS E158 "concern about humankind" = EVS Q60, item v216 "the living conditions of all humans all over the world"). Source: europeanvaluesstudy.eu → Methodology, data, documentation → Survey 2017 → full release EVS2017 → participating countries → questionnaires.

To find which questionnaire a code lives in, check the dictionary's WVS-7 vs EVS columns: if the WVS-7 variable name is blank for that IVS code, it's EVS-only — go to the EVS questionnaire.

The 878 MB .dta is likewise git-ignored and must be present in snapshots/ivs/<v>/ locally to regenerate the CSV.

Step 0 — Verify scales against the .dta (NEVER from memory)

Model-recalled survey wording/scales are unreliable — verify. The authoritative code→category mapping for each variable lives in the .dta value labels. Read every code you're adding in one pass:

import pandas as pd
# IVS: snapshots/ivs/<v>/Integrated_values_surveys_1981-2022.dta   → convert_categoricals=True
# WVS: snapshots/wvs/<wv>/WVS_Time_Series_1981-2022_stata_v5_0.dta → convert_categoricals=False, then
#      inspect raw codes: WVS codes missing as NEGATIVES (-1 DK, -2 NA, -4 not asked, -5 missing); the
#      positive codes (and 0 where it's a real category, e.g. immigration G05x) are the substantive answers.
path = "snapshots/ivs/<v>/Integrated_values_surveys_1981-2022.dta"
# .dta is row-major: each read_stata() streams the whole 878 MB file once. Read EVERY code you
# need in ONE pass, then inspect each from the in-memory frame — NOT one read_stata() per code
# (that re-streams the 878 MB file per code: ~24 full passes for 24 codes, ~10x slower).
codes = ["C001", "C002", "H002_01"]  # all the codes you're adding
df = pd.read_stata(path, columns=codes, convert_categoricals=True)
for c in codes:
    print(c, list(df[c].cat.categories))   # order == code 1, 2, 3, … (the IVS coding)

(StataReader.value_labels() mapping is unreliable for these vars — use convert_categoricals=True.)

Key gotchas:

• IVS often collapses the raw WVS scale. E.g. C001/C002 are 5-point in the WVS-7 questionnaire but 3 categories in the IVS .dta (1 Agree · 2 Disagree · 3 Neither). Trust the .dta categories.
• Look-alike questions can have different scales. H002 neighborhood frequency = Very / Quite / Not / Not at all frequently; H008_02 ("felt unsafe at home") = Often / Sometimes / Rarely / Never — don't lump them into one block.
• Confirm the Q-number + verbatim wording from the dictionary + questionnaire (see Reference docs).

For the user-facing wording, prefer the fuller questionnaire text; for the answer/category names the .dta labels and questionnaire may differ slightly ("some respect" vs "fairly much respect") — they mean the same; pick per the user's preference.

Step 1 — Stata: snapshots/ivs/<v>/ivs_create_file.do

The .do follows one idiom per question group: define a global listing codes → a preserve-scoped block (a foreach loop for multi-item groups, or a custom block for a single question) that recodes to 0/1 dummies → collapse (mean) … [w=S017], by(year country) → save a tempfile → later merge 1:1 year country in the "Combine all the saved datasets" section. New codes must also be added to the master keep S002VS S002EVS S003 S017 $questions line.

WVS target? Edit snapshots/wvs/<wv>/wvs_create_file.do instead — same idiom, but the master keep is keep S002VS S003 S017 $questions (no S002EVS) and the missing-value handling differs: per question use keep if \var' >= -2 & `var' < .(drops -5/-4/-3 and system-missing; keeps substantive + DK + NA), thendont_know_<q> = (`var' == -1), no_answer_<q> = (`var' == -2), substantive dummies on the positive codes, and for avg_score questions gen avg_score_<q> = `var' if `var' >= 0` (so negative DK/NA are excluded from the mean). The structural-twin table below still applies — only the keep / DK / NA lines change.

Reuse the closest existing structural twin instead of inventing a block:

For a 10-pt block the three aggregates (>=7, 5\|6, <=4) + dont_know + no_answer partition to 100% — that's what check_sum_100 checks (avg_score is extra). For multinomial, the N option dummies + dont_know + no_answer sum to 100%. When several questions share the same option codes (e.g. E001/E002 "aims of country: 1st/2nd choice" both map 1–4 to the same four goals), put them in one loop.

Aggregate-name collisions: an aggregate must not collide with a category name. The closeness 4-pt scale has a category close (code 2), so the high aggregate (1|2) must be named feel_close, not close. Similarly check any not_* aggregate vs a not_* category before naming.

Aggregates and avg_score are NOT part of the check_sum_100 partition — only the mutually-exclusive categories + dont_know + no_answer sum to 100. Aggregates (worry_, feel_close_, agree_, at_least_weekly_…) are derived extras layered on top.

Stata name limits (this WILL bite you): local-macro / tempfile names are capped at 31 chars (variable names at 32). tempfile neighborhood_frequency_\var'_file(35) errors withr(198). Keep tempfile macro names short, e.g. nbhd_freq_`var'_file`.

Continuous indices (e.g. Y022 Welzel equality): keep the native scale; keep if Y022 < .; gen avg_score_<name> = Y022; collapse (mean) …; no DK/NA, no aggregate. Naming it avg_score_* makes the final replace \var' = `var'*100step skip it (that step multiplies everything **except**avg_score*`).

Each block's generated columns are {answer}_{CODE} for loops (meadow renames the CODE), or directly named for single-question custom blocks (e.g. secure_neighborhood, very_secure_neighborhood).

The user runs the edited .do in Stata to regenerate ivs.csv. You can sanity-check by reading the .dta in pandas and recomputing a couple of weighted means to diff against the produced CSV.

Step 2 — Re-snapshot (same version, no bump)

After the user regenerates ivs.csv (next to the .dta), first confirm the Stata run actually emitted your new columns before snapshotting — cheap insurance against a botched/partial .do run:

import pandas as pd
cols = set(pd.read_csv("snapshots/ivs/<v>/ivs.csv", nrows=0).columns)
# pre-rename raw names, i.e. {prefix}_{CODE} for loops + literal names for custom blocks
assert {"at_least_weekly_E248B", "agree_E217", "better_off_science_world", "concerned_humankind"} <= cols

Then re-snapshot:

.venv/bin/etls ivs/<v>/integrated_values_surveys --path-to-file snapshots/ivs/<v>/ivs.csv

This overwrites the .dvc md5/size in place (commit that diff with a 📊🤖 message). Delete the local ivs.csv after.

Step 3 — Meadow: etl/steps/data/meadow/ivs/<v>/integrated_values_surveys.py

Add entries to VARS_DICT ("CODE": "Readable label") only for codes used as column suffixes — i.e. the loop groups. The rename works by column.endswith(code), then snake_cases. Custom single-question blocks already produce final names, so they get no VARS_DICT entry. Check that no new code is a suffix of another (e.g. H002_01 vs H002_1 — fine; just be deliberate).

Labels: avoid : and other punctuation. rename_vars snake_cases in two steps — first a crude .str.lower().str.replace(" ", "_"), then tb.format() applies the real underscore(). A label like "Information source: Daily newspaper" ends up as information_source__daily_newspaper (the colon becomes a second underscore). Use colon-free labels ("Information source daily newspaper") so the final suffix is clean single-underscore (information_source_daily_newspaper), and put the nicely punctuated wording in the .meta.yml title/description_short instead. Codes with trailing letters (e.g. E248B, G007_18_B) work fine with endswith.

Step 4 — Garden: etl/steps/data/garden/ivs/<v>/integrated_values_surveys.py

Per categorical group add:

1. a suffix-list constant (the snake-cased item names),
2. a replace_dont_know_by_null(...) call — answers = the individual response categories (no aggregate, no DK/NA),
3. a check_sum_100(...) call — answers = individual categories + dont_know + no_answer.

check_sum_100 is the correctness gate: it fails loudly if a recode is wrong (categories must sum to 100%).

Derive the exact garden column suffixes (so the constants match meadow output) with the same transform meadow uses:

from owid.catalog.core.utils import underscore   # owid.catalog.utils path is deprecated
suffix = underscore("Information source daily newspaper")  # -> "information_source_daily_newspaper"

(apostrophes are dropped, hyphens → _; for colons see Step 3.)

avg_score and the replace(0, NaN) step. Only indices where 0 is a genuine value need to be excluded from the 0→null replacement (WELZEL_EQUALITY_INDEX_COLUMNS, the 0–1 Welzel index). An ordinary avg_score_* on a 1–N scale (frequency 1–5, closeness 1–4, agree 1–10, …) never equals 0, so the replacement is a no-op for it — do not add those to the exclusion set. A truly continuous block also gets no replace_dont_know/check_sum.

Step 5 — Metadata: etl/steps/data/garden/ivs/<v>/integrated_values_surveys.meta.yml

Every new column needs an entry: title, single-quoted description_short (quote the question stem + answer options, double internal apostrophes for YAML), and display.name + <<: *common-display. Source the wording from the questionnaire PDF; the code→category order from the .dta. Index/unit-less columns (the avg_score_*) override unit: "" / short_unit: "" and numDecimalPlaces: 2; share columns inherit unit: "%" from &common-display. Generate the entries programmatically (it's 100s of columns) and append to the variables: block. To re-run the generator after a wording fix, splice cleanly: find the first new key in the file, truncate from there, and re-append the regenerated block (don't blindly append twice).

Enumerate the possible answers in every description_short. Don't stop at naming the response(s) the indicator measures — append the full set of options the respondent could choose, so each indicator is self-describing. Follow the IVS pattern: for categorical / Likert scales end the sentence with Possible answers are "<a>", "<b>", … and "<z>". (list them in the source order from the .dta value labels); for an N-point numeric scale state the anchored range instead (e.g. on a scale from 1 ("never justifiable") to 10 ("always justifiable")). This applies to every column in the block — the aggregates (agree_agg_*, never_just_agg_*, …), each individual category, and the dont_know / no_answer / avg_score columns — they all carry the same possible-answer clause; only the measured-response part differs. Pull the exact option wording from the .dta value labels (verify per Step 0), never from memory — e.g. WVS D066_01's fifth point is literally Disagree strongly, and F114E labels only the 1 and 10 endpoints. This is a rule for new indicators you add — apply it as you write each entry.

Do not attempt an automated bulk back-fill of the possible-answers clause across the existing IVS indicators. Most already convey their answers (often inside the question quote, e.g. important-in-life ends "…very important, rather important, not very important or not at all important?"), so a blanket append is redundant; and mapping an existing shortName back to its source .dta code is trap-laden — suffixes collide (*_democratic_political_system is E117, not the 1–10 E236 *_democratic; *_secure_neighborhood is the H001 security scale, not the G007_18_B neighborhood-trust scale), and value-label wording can diverge from the pipeline's recode wording (E124 human-rights: labels say "There is a lot of respect…" while the columns use "a great deal of respect…").

Keep IVS and WVS at metadata parity. Any description_short house-style or convention you apply to one table's indicators, apply to the other's too (e.g. the possible-answers clause above), so the two blocks stay consistent, differing only in the survey-specific anchors (*_wvs) and the question wording.

Canonical shape of a variable entry (a WVS entry = the IVS entry plus the three *_wvs override lines; everything else is inherited from definitions.common, so don't re-specify it):

definitions:
  common:                                 # applied dataset-wide → inherited by BOTH tables
    presentation: {attribution_short: Integrated Values Surveys}   # IVS-worded
    processing_level: major
    description_key: [ …IVS-worded (merged WVS+EVS, IVS waves)… ]
    description_processing: | …IVS-worded…
    display: &common-display
      numDecimalPlaces: 1
      tolerance: 5
      entityAnnotationsMap: |-
        United Kingdom: England, Scotland, and Wales
    unit: "%"
    short_unit: "%"
  # WVS overrides (common is IVS-specific) — referenced per WVS variable:
  attribution_short_wvs: &attribution_short_wvs World Values Survey
  description_key_wvs: &description_key_wvs [ …WVS-worded… ]
  description_processing_wvs: &description_processing_wvs | …WVS-worded…

tables:
  integrated_values_surveys:              # IVS variable — inherits all of common
    variables:
      <short_name>:
        title: '<unique across BOTH tables>'
        description_short: '% of respondents … "<question stem>". Possible answers are "<a>", …, "<z>".'
        display:
          name: <short label>
          <<: *common-display
        presentation:
          topic_tags: *topic_tags_<topic>

  world_values_survey:                    # WVS variable — SAME shape + the 3 *_wvs overrides
    variables:
      <short_name>:
        title: '<unique across BOTH tables>'
        description_short: '% of respondents … "<question stem>". Possible answers are "<a>", …, "<z>".'
        description_key: *description_key_wvs                 # ← WVS-only override
        description_processing: *description_processing_wvs   # ← WVS-only override
        display:
          name: <short label>
          <<: *common-display
        presentation:
          attribution_short: *attribution_short_wvs           # ← WVS-only override
          topic_tags: *topic_tags_<topic>

      avg_score_<q>:                        # unit-less average column (either table)
        title: '…: average score'
        description_short: 'Average score … on a scale from 1 ("…") to 10 ("…").'
        display: {name: '…: average score', numDecimalPlaces: 2, <<: *common-display}
        unit: ""
        short_unit: ""
        presentation: {topic_tags: *topic_tags_<topic>}       # + attribution_short: *attribution_short_wvs on WVS

Battery questions — separate the prompt from the item. Many WVS/EVS questions read a generic prompt then list items (e.g. "…with the following statements? - Work is a duty…"). In description_short never leave a raw " - " separator and never let the item dangle straight after the "?" (e.g. …statements? Work is a duty…) — both read as two sentences mashed together. End the prompt's quote and re-open a quote for the item, picking the connector by what the item is:

Rationale: the deciding factor for statements is singular vs plural — a singular "the following statement?" already names its one statement so a connector is redundant (just drop the dash), while a plural battery needs the connector to say which item. The connector noun must fit the item: "when the thing was" reads badly, so use "regarding" (or a clean noun like "situation"/"characteristic"). Always use the comma form ", when the … was ", matching the pre-existing justifiable battery.

YAML quoting in a Python generator: keep RAW apostrophes in your strings and run them through one single-quote helper that doubles '→''. Do not hand-write 'Don''t know' in Python source — that's adjacent-string-literal concatenation and silently yields Dont know. After appending, re-parse the file with ruamel_load and assert the count + that a new avg_score_* entry shows unit == "" and the &common-display anchor resolved (e.g. numDecimalPlaces).

title must be UNIQUE across the whole dataset — and the dataset holds BOTH tables. Because integrated_values_surveys and world_values_survey share one grapher dataset, a WVS title must not clash with any IVS title (or vice-versa). WVS shares many concepts with IVS (worries, confidence, justifiable, …), so when you add a WVS question that duplicates an IVS concept, suffix the WVS title (e.g. … (World Values Survey)) to keep it unique; display.name may still repeat. Grapher enforces this at the --grapher upload (_adapt_table_for_grapher), not at the garden build, so a clash sails through check_sum_100 and only explodes at upload (AssertionError: Variable titles are not unique). The trap: an aggregate and a category that describe the same thing collide on title even when their column names differ. For example, the closeness category close_* and the high aggregate feel_close_* both wanted title: "Feel close to X". Fix: disambiguate the aggregate title ("Feel close to X (very close or close)") and keep the short label in display.name. Assert title uniqueness in the pre-flight (Step 6), don't wait for the upload to catch it: titles=[e["title"] for e in vars.values()]; assert len(titles)==len(set(titles)).

Audit gotcha: verify any stated scale range against the actual .do recode — an existing entry can say "6 to 10" where the recode is >= 7. Don't copy ranges blindly.