Find Data

find-data

Turn an idea, URL, or category into a phase2/datasets/<name>/ folder that the /data2story-pro pipeline can run on without crashing.

You are the gatekeeper before the 7-agent newsroom. Detective, Analyst, Editor, Designer, Programmer, Auditor, Inspector all assume the data is already there, parseable, and provenanced. Your job is to make sure that's true before they start.

Refuse to mark a folder ready until it passes 4 gates. See references/completeness_gates.md for the criteria. The gates are codified in tools/audit.py, not in this prose.

Prerequisites

Python deps for the tools/:

• pandas — required (audit.py reads/inspects CSV/JSON).
• openpyxl — required only when a source is .xlsx (audit.py's pd.ExcelFile); a mid-run missing-openpyxl is the usual cause of an XLSX audit error.
• urllib — stdlib, no install (fetch.py downloads, audit.py HEAD-checks).

One install line:

pip install pandas openpyxl

After editing anything under tools/, run the no-network regression suite: py tools/selftest.py (exit 0 = pass).

Resolve paths first

• SKILL_DIR = directory containing this SKILL.md
• WORKSPACE = ancestor that contains phase2/datasets/ (the parent repo root), if one exists
• DATASETS_ROOT = WORKSPACE/phase2/datasets when WORKSPACE exists; otherwise ./datasets (clone-relative, under the current working dir) — the open-source clone has no phase2/datasets
• BLOGS_ROOT = WORKSPACE/phase2/blogs when WORKSPACE exists
• OUT_DIR default = DATASETS_ROOT/<name> (so ./datasets/<name> on an OSS clone). Callers may override with an explicit --out, which always wins.
• INPUT = first positional argument from $ARGUMENTS
• Parse flags from $ARGUMENTS: --mode, --source, --out, --validate-only

Never hard-code machine paths. Resolve at runtime by walking up from SKILL_DIR.

Step 0 — Classify the input

Decision tree (the FIRST condition that matches wins):

1. --validate-only flag present → MODE = validate-only. INPUT must be an existing folder path.
2. INPUT is an existing folder path (and --validate-only absent) → still ask the user before re-fetching; default to validate-only behaviour with a confirmation.
3. INPUT starts with http:// or https:// → MODE = url.
4. INPUT (case-insensitive) exactly matches a DIP category from phase2/datasets/data_is_plural/dip_category_summary.md → MODE = category.
5. Otherwise → MODE = topic.

State your classification out loud in one line before doing anything else.

No local corpora (open-source clone): on a machine without the Economist/, Pudding/, tidytuesday/ clones and without the Data-is-Plural CSV under phase2/datasets/, browse_local.py and dip_query.py both return empty (no crash). In that case there is nothing to match locally, so topic/category discovery degrades straight to WebSearch (Step 1). No flag is needed — detect it from the empty browse_local.py + dip_query.py results. Keep the local-first path intact for machines that DO have the corpora.

Step 1 — Branch on mode

validate-only mode

Just audit. Skip discovery, skip fetch.

python "SKILL_DIR/tools/audit.py" "<folder>"

This writes <folder>/validate.json and prints the gate summary. Read it back with Read, surface the verdict, and stop.

If overall.ready_for_data2story is true → print the /data2story-pro <folder> command for the user to copy.

If it's false → list the specific gate failures and recommend remediations.

url mode

Determine OUT_DIR (use --out if given, else derive a slug from the URL path's last meaningful segment, store under DATASETS_ROOT/<slug>/ — i.e. ./datasets/<slug>/ on an open-source clone with no phase2/datasets).

Branch on URL shape:

fetch.py is source-aware: if the URL points to TheEconomist/graphic-detail-data, the-pudding/data, or rfordatascience/tidytuesday AND the local clone already contains that path, it copies from the local clone (no network). Otherwise it fetches via the GitHub API. This happens automatically — do not handle it yourself.

URL normalization (share-page vs direct-download)

Several hosts serve a human-facing page at the obvious URL, not the raw bytes, so a naive download returns HTML instead of a .csv/.json/.xlsx. fetch.py auto-rewrites GitHub /blob/ URLs to raw.githubusercontent.com and, as a backstop, warns on stderr if a data-extension download comes back as an HTML page (<!doctype html> / <html). The other common share-hosts are not auto-rewritten — pass the direct-download form yourself:

If a download trips the HTML warning, you almost certainly handed fetch.py a share/page URL — fix the URL to its direct-download form and re-fetch.

After fetch: check whether the folder has a README.md.

• If yes → leave it alone, jump to Step 2.
• If no → generate one from tools/README_template.md. Auto-fill what you can (title from folder name, column headers from the first CSV, source URL from the manifest). Leave the codebook definition cells as {TODO} for the user to fill in. Tell the user explicitly that README is a stub.

category mode

INPUT is one of the 14 DIP categories. Run both browse and DIP query in parallel:

python tools/browse_local.py "<INPUT>" --top 10
python tools/dip_query.py --category "<INPUT>" --top 10

Both emit JSON. Merge:

• Local candidates (already-cloned folders) score higher than DIP leads (which still need fetching).
• For each local candidate, run a cheap pre-score (does Gate 1 trivially pass — is there ≥ 1 CSV in the folder?).

Present top 5 to the user as a numbered list via AskUserQuestion:

• For each candidate, show: source · title · date · top-3 files · 1-line readme excerpt
• Options: pick 1–5, or "all" (theme mode), or "skip" (abort)

Then route the pick:

• Local candidate → copy folder into OUT_DIR (or in-place audit if user accepts).
• DIP lead → extract URL from the candidate's links array. If multiple URLs, pick the one with a data-file extension (csv/xlsx/json) or the GitHub one. Route to url mode for that URL.