Fact Check Dataset

Step 4 — Phase 2: independent online cross-check (anomaly-led + anchors)

This is the half that catches the source's mistakes (failure class 2 above).

What to check:

• Every anomaly that survived your Phase-1 triage. Cap the WebSearch effort at ~10 values; list anything beyond the cap as unchecked in Part 2.
• Fixed anchors, regardless of anomalies: the World total (if the dataset has one), 2–3 major or topic-relevant countries, the latest year, and one mid-series historical year.
• One-year shifts in newly added years, for sources that compile national-office figures (OECD, Eurostat, UN agencies). Compare the new years for 1–2 countries with the national office's own figures: our value for year Y matching its figure for Y−1 means the series is shifted.

Independence rules (anti-circularity — read before searching). An independent source is a different producer measuring the same quantity (WHO vs. IHME, IEA vs. Energy Institute, IMF vs. World Bank, UN WPP vs. a national statistics office), or the primary source the producer aggregates. Never count as independent: ourworldindata.org itself; sites that republish OWID (Wikipedia charts and infoboxes frequently cite us — check the citation); mirrors of the same producer (tradingeconomics and friends scrape WB/IMF); or the producer's own secondary pages.

Procedure per value: WebSearch the quantity + entity + year → open 1–2 authoritative hits with WebFetch → record source, value, and link in the Part 2 table. Never cite a number straight from the search-results summary — summaries blend several sources and lag living pages; every figure that reaches a finding, a PR body, or a producer question must be quoted from a page you actually opened.

Measurement-artifact scrutiny (per source, not per value): search "<producer> completeness bias", "<producer> coverage <region>", "<indicator> revision history" and read what comes back. When you flag a comparability problem, name the specific mechanism by which the data misleads (e.g. "death registration completeness below 60% in region X inflates apparent improvement"); a bare "comparisons should be made with care" is banned.

Internal accounting identities beat external sources. When a producer publishes components and their aggregate, check that they reconcile — a contradiction inside the producer's own release is arithmetic, needs no independent source, and can't be waved away as a methodology difference. It also survives the common case where every external source is bot-blocked. This is confirmation route (b) in the Tolerance gate below — it settles a source error on its own, provided the guards hold: every term of the identity comes from the same release and vintage (never mix an old download's share with a new download's rate), the terms are defined so the identity holds by construction (shares that sum to 1, urban+rural weighted by the same population split), and the contradiction is far beyond rounding. Two habits that make this reusable: sweep the identity across all entities, not just the suspicious one, to prove the error is isolated rather than systematic; and check the producer's footnote table (WDIfootnote.csv and equivalents) — an unqualified bad value is a stronger finding than a flagged one. One more guard before anything reaches the corrections route: the contradiction proves an error exists among the identity's terms — it does not by itself say which term is wrong. Identify the bad term with evidence beyond the identity: the entity's own adjacent years (the term solved from the identity matches its own prior-year value, while the other terms sit in line with their own histories), the all-entity sweep isolating a single term, or a footnote. If nothing singles out one term, the finding is still a confirmed producer error — report it and hand it to producer follow-up, but don't guess which value to overwrite in corrections.yml.

A "no charts use this indicator" clearance goes stale the moment charts are remapped. Blast-radius checks are version-scoped; a chart stranded on an older version won't match a query filtered to the version you're updating, and will silently come into scope once the stranded-chart sweep runs. Re-run any such check after the indicator upgrade before relying on it.

A mismatch that matches exactly under swapped names is a label error, and it can be on either side. When a comparison against the producer's own table fails for two related rows (two subregions, two sexes, two variants) and each row's values equal the other row's to the last decimal, don't treat it as our pipeline being wrong. Settle which side has the labels crossed with an independent reconstruction — rebuild both aggregates from their members, or check the populations each label implies against the table's own totals — before touching anything. If it is the producer's document, that is a confirmed producer error to report; if it is ours, it is an entity-mapping bug.

Tolerance: rounding, vintage/revision drift, and methodology gaps of a few percent are not findings. The targets are magnitude errors (×10/×100/×1000), wrong-year values, sign errors, entity mix-ups, and stale pre-revision values. Declaring a confirmed source error requires one of two routes: (a) ≥2 independent sources that agree with each other and disagree with ours beyond methodology tolerance, or (b) an accounting-identity contradiction internal to the producer's own release, under the guards in Internal accounting identities beat external sources above.

Attribution before routing. Before routing any confirmed bad value, read the raw snapshot (from etl.snapshot import Snapshot; Snapshot("<ns>/<version>/<file>").read() — read() picks the reader from the file's format; use the format-specific read_csv/read_excel/read_json only when auto-detection needs overriding) to determine where it entered: present in the source file → source error (corrections route); absent → our processing introduced it (trace snapshot → meadow → garden and fix the step).

Step 5 — Phase 3: adversarial metadata review (top-N + anomalous indicators only)

Treat each prioritized indicator's user-facing text as a set of claims and attack them against the Phase-0 documentation:

• Does unit (and any display.unit/conversion) match the producer's stated unit? A (mils)/(000) marker in the source's column header demands a visible conversion in garden.
• Does the title / description_short overclaim scope — "global" when the source covers reporting countries only, "countries" when it's high-income countries?
• Does description_key state contested definitions as settled, or omit a caveat the producer's own docs (or your Phase-2 literature search) prominently state — coverage gaps, comparability breaks, denominator choices?
• Are causal or certainty words ("shows", "proves", "leads to", "drives") backed by the source's methodology, or do they smuggle in an interpretation?
• For categorical indicators built from label maps, list the distinct source labels and verify each maps explicitly — values routed to a fallback bucket ("unknown", "other") are silent misclassifications, because the fallback is an existing category and no validation fires. Recommend an observed labels ⊆ map keys assert where one is missing.
• For Jinja-templated metadata, spot-check the rendered text readers actually see: Dataset("data/grapher/<ns>/<version>/<short_name>").read(t, load_data=False)[col].metadata.

Lead with the concrete rewrite, not the objection. "Add a link" is a valid fix. Match the register and length of the original — prefer a word swap over an added clause.

Routing findings

Output report

Write to ai/adversarial-review-<short_name>-<YYYY-MM-DD>.md:

# Adversarial data review — <ns>/<version>/<short_name>

## TL;DR
(max 3 sentences: overall verdict + the one thing to fix first)

## Part 0 — Source verification
- Source(s) + documents accessed (URL, access status: read / paywalled / unreachable)
- What the source publishes (indicators, units, definitions, granularity, upstream data)
- What OWID adds (from description_processing + step code + corrections.yml)
- Field-placement check (.dvc description / description_processing / description_from_producer)
- Data-quality flags stated in the source's own docs
- Flags the docs SHOULD state but don't (from the artifact-literature search)
- Existing corrections.yml entries acknowledged

## Part 1 — Findings (numbered, ordered 🔴 → 🟡 → 🟢)
N. 🔴|🟡|🟢 [data-level|text-level] <one-line defect>
   Evidence: <quote or value + link>
   Fix: <concrete action + routing per the table above>
   Why: <one tight sentence>

## Part 2 — Cross-check appendix
- Values checked: indicator | entity | year | our value | independent value(s) | source link | verdict ✓/✗/~
  (mark each row as anchor or anomaly)
- SKIPPED indicators (name, views_365d rank, why skipped) — mandatory, never silent
- Anomalies beyond the WebSearch cap, listed as unchecked

Severity rubric (aligned with /review-data-pr): 🔴 = confirmed factual error (metadata contradicted by the producer's own docs, or a value confirmed wrong via route (a) or (b) in Step 4, with snapshot-level attribution); 🟡 = likely issue needing confirmation; 🟢 = informational or unverifiable.

In author flows, apply the 🔴 fixes immediately (they're why the skill ran before commit); leave 🟡/🟢 as report items for the user to triage.

Key constraints

• Always fetch and search — never judge data plausibility from memory or training data. The whole point is a fresh look at what the source and the wider literature actually say today.
• Establish source-vs-OWID attribution (Phase 0) before critiquing either layer; every finding names whose layer it targets.
• Severity proportional to verification depth — the HARD RULE in Step 2.
• Don't manufacture objections. A clean report is a valid outcome; if uncertain whether something is wrong, say "verify this" rather than asserting it.
• Separate data-level from text-level findings — a legitimate metric can sit under overclaiming text, and carefully hedged text can sit on top of broken data.
• Methodology differences are not errors. Name the specific mechanism before calling a mismatch an error.
• Factual accuracy only — no style/typo/spacing duplication (see Scope).
• corrections.yml override values on categorical columns must come from the source's current vocabulary — assigning a retired label fails with Cannot setitem on a Categorical; choose the current-vocabulary value that yields the same published output.
• No persistent files beyond the ai/ report — plus, in author flows, the metadata/corrections edits themselves. Ad-hoc analysis scripts run from the session and are not committed.