Crawler Pep

New PEP Crawler

Create a new PEP crawler. The user will provide a target path, source data URL, and/or a GitHub issue URL: $ARGUMENTS

If given a GitHub issue URL, fetch it first to extract the data source URL and any context about the dataset.

Read upfront. These are the rules; this skill is the procedure for applying them, and repeats only the few rules the docs don't cover yet:

1. zavod/docs/peps.md — the PEP model, properties, position naming, categorisation, occupancy dates and status, historical terms.
2. .claude/docs/crawler-guide.md — shared crawler patterns, YAML template, lookups.
3. .claude/skills/crawler-pep/examples.md → "Reference crawler" — a complete, reviewed crawler. Yours should look like it: the shortest code that still handles every source field explicitly. Each helper, constant, guard or extra request you add beyond it needs a reason a reviewer can see — first drafts fail review far more often from added machinery than from missing it.

Open the rest of examples.md when your source differs from the reference (mixed datasets, several positions, multi-term, subnational). Ground the crawler in these files, not in other crawlers: the codebase is old and many have drifted from current practice.

Step 1: Understand the source

Do this before writing any code. A wrong endpoint or a misread date produces a crawler that looks finished and is worthless.

• Find the underlying JSON/XML endpoint before parsing HTML — page source, network calls, JS bundles. Parliament sites very often have an API behind the rendered page.
• Prove the source is blocked before reaching for Zyte. In order: a browser http.user_agent; a language cookie or Accept-Language; a format suffix (.json) or Accept: header. Zyte costs you ci_test: false.
• Establish the pagination contract from the response — the per-page cap, the total or next link. An API that silently caps page size truncates without error.
• Enumerate every field the source returns and decide each one: emit, or ignore.
• Find the terms the source exposes — a term switcher, an ElectionId parameter, a /legislaturas endpoint — and whether dates are per person or per term.
• Wikidata QIDs: check on Wikidata that the item is instance of (P31): position and applies to jurisdiction (P1001) matches the country; a plausible label is not enough. Never pass one QID to two positions — it becomes the entity ID, and they'd collapse into one.
• Citizenship: spawn a subagent (WebSearch/WebFetch) to find the legal document (constitution, electoral law) that requires citizenship for this specific position. Cite its URL in a comment next to person.add("citizenship", ...), or next to its omission if not required.
• Term-bounded source (fixed mandates, per-term pages)? Note a structural signature (page URL, file name, term id) and fail in crawl() when it changes, so a new term can't go unnoticed.
• Check the records against reality. Count them by role and by term, and compare with the seats the body actually has. Could a record's dates be stale (e.g. a re-elected member still carrying their previous term)?

Checkpoint: show the user a short recon note and wait before writing code — endpoints and pagination; every field with its decision; the terms exposed; counts by role and term against the seats; anything the source contradicts itself on, with the simplest options. A wrong assumption corrected here costs one message; found in review it costs a rewrite. In an automated run with no user, put the note at the top of your final report and proceed.