New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Crawler Constants To Yml

by opensanctions in opensanctions/opensanctions

Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever…

MITAuto-check: notesData & Analytics

Install Crawler Constants To Yml

skills CLI
$ npx skills add opensanctions/opensanctions --skill crawler-constants-to-yml -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install opensanctions/opensanctions crawler-constants-to-yml --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.claude/skills/crawler-constants-to-yml .claude/skills/crawler-constants-to-yml && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
crawler-constants-to-yml
GitHub stars
834
Token cost
~1.4k tokens
SKILL.md length
563 words
Files
1
Skills in repo
11
Repo updated
First seen
Licence
MIT

At a glance

Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever…

  • Works in 3 steps: Classify the constant → pick its YAML home → Apply the change → Verify — always run the crawler
  • Put in the yml a crawler constant such as GENDERS
  • SKILL.md covers 1. Classify the constant →…, 2. Apply the change, 3. Verify — always run the… and Gotchas
  • Calls ruff, mypy and python3

What it does

Crawler Constants To Yml is an agent skill from opensanctions/opensanctions. Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever possible, otherwise http / config / dates metadata. Use when asked to move, hoist, or "put in the yml" a crawler constant such as GENDERS or HEADERS.

Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Data & Analytics, covering Web scraping and Translation. The repository describes itself as: An open database of international sanctions data, persons of interest and politically exposed persons. The licence is MIT.

When your agent uses it

  • Put in the yml a crawler constant such as GENDERS
  • Tasks that involve Web scraping
  • Tasks that involve Translation

Example prompts

  • “put in the yml”
  • “/crawler-constants-to-yml”

Requirements

  • Python 3
  • Pre-approved tools (allowed-tools): Read, Edit, Bash, Grep, Glob

Workflow steps

3 steps, taken from the step headings in SKILL.md.

  1. Classify the constant → pick its YAML home
  2. Apply the change
  3. Verify — always run the crawler

What it can do on your machine

Read from SKILL.md and the folder at commit 123b662. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves these tools, so the agent can use them without asking each time:

    • Read
    • Edit
    • Bash
    • Grep
    • Glob

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • ruff
    • mypy
    • python3

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Crawler Constants To Yml loads about 1.4k tokens when it runs. Until then it costs about 92 tokens; SKILL.md has 563 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~92
When it runs · the whole SKILL.md, loaded when a task matches
~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check: notes

The automated check noted patterns worth knowing about, such as sudo or a known installer.

  • NotePre-approves every shell command (allowed-tools: Bash)SKILL.md
    allowed-tools: Read, Edit, Bash, Grep, Glob

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from opensanctions/opensanctions at commit 123b662, republished under its MIT licence (© opensanctions). 563 words, ~1,391 tokens.

Download SKILL.mdSave it as .claude/skills/crawler-constants-to-yml/SKILL.md (or your agent's skills folder).
name
crawler-constants-to-yml
description
Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever possible, otherwise http / config / dates metadata. Use when asked to move, hoist, or "put in the yml" a crawler constant such as GENDERS or HEADERS.
allowed-tools
Read, Edit, Bash, Grep, Glob
argument-hint
[dataset .yml or crawler.py path]

Move crawler constants into the dataset YAML

Hoist a module-level constant out of a crawler and into its .yml. The guiding rule: whatever the framework can handle from the YAML should live in the YAML — a lookup, http, config, or dates block — leaving the crawler code free of one-off maps and conditionals. Target: $ARGUMENTS (the .yml, or the sibling crawler.py).

Read the relevant zavod/docs before editing — do not re-derive the mechanics here:

  • Lookups (type vs named, matching, config keys): zavod/docs/best_practices/datapatch_lookups.md
  • Headers / bot-blocking / http.user_agent: zavod/docs/best_practices/http_operations.md
  • Date formats and month-name translation: zavod/docs/best_practices/dates_meta.md

1. Classify the constant → pick its YAML home

Constant in codeBelongs inMechanism
Map of source value → clean value for a typed property (gender, country, …)lookups: type.<type>Auto-applied on entity.add(prop, raw). Pass the raw value; delete the dict.
Map/translation of a value for a non-type property, or a categorical labellookups: <name>context.lookup_value("<name>", raw, warn_unmatched=True)
Request User-Agenthttp.user_agentApplied to the session; drop headers= if it held only the UA.
Static structural/config dict (column labels, field maps used for validation)config:Read via context.dataset.config["key"]
Date formats / non-English month namesdates.formats, dates.monthsConsumed by h.apply_date

Type-property names map to their type.<type> per the table in datapatch_lookups.md (e.g. gender → type.gender, citizenship/country → type.country, email → type.email).

If a constant genuinely cannot move (see gotchas), leave it in code — do not force it.

2. Apply the change

Type lookups (the common GENDERS case). Add the lookup and feed the raw value in:

yaml
lookups:
  type.gender:
    options:
      - match: Masculino   # exact source values, one per real value
        value: male
      - match: Femenino
        value: female
python
person.add("gender", record["sexo"])   # was: GENDERS.get(record["sexo"])

Delete the dict, its .get(...), and any or ""/None-guard it needed. Match the source value exactly — vocabularies differ per source (Femenino vs Feminino, Hombre/Mujer vs Masculino/Femenino); never copy a sibling dataset's lookup blind.

Named lookups. For a value that isn't a typed property (e.g. a seat/membership type), use an explicit lookup and store the result — if the old code fetched the value only to validate it, recording the mapped value is a strict improvement:

python
seat_type = context.lookup_value("membership_type", raw, warn_unmatched=True)
occupancy.add("summary", seat_type)

Headers. Move only the User-Agent to http.user_agent. For any other header, first prove whether the server actually needs it (§3) — drop inert ones, keep genuinely-required ones inline (the http block supports only user_agent). A format-selecting Accept header can sometimes be replaced by a URL suffix/param in data.url (e.g. …/atual.json), removing the header entirely.

Show full SKILL.md (198 more words)Show less

3. Verify — always run the crawler

bash
ruff check <crawler.py> && mypy --strict <crawler.py>
zavod crawl <dataset.yml>
# issues.log must be clean (or only carry warnings you expect):
python3 -c "import json;[print(json.loads(l)['level'],json.loads(l).get('message')) for l in open('data/datasets/<name>/issues.log')]" | sort | uniq -c
# confirm the value still lands, at the expected coverage:
grep -a ',<Schema>:<prop>,' data/datasets/<name>/statements.pack | awk -F, '{print $3}' | sort | uniq -c

When probing which headers a server requires, drop them one at a time against the live endpoint and keep only those whose removal changes the status/response.

Gotchas

  • A type lookup changes failure behavior — that's the point. The old DICT.get(x) returned None and silently dropped unmapped values; the lookup instead lets unmapped values reach the type cleaner, which warns. After the change, confirm coverage in the statement count and that no new Rejected property value warnings appear.
  • Lookups normalize/order-independently; don't put order-significant data in one. Keep ordered structural lists (e.g. left-to-right column keys) in code; move only the order-independent label map to config:. Making correctness depend on YAML key order is a trap for the next editor.
  • config: is for static config, not value cleaning. Reach for a lookups: block when you're normalizing property values; use config: for parsing/structure knobs.
  • Not every header fits the YAML. Accept, Content-Type, Referer, etc. have no http field — required ones stay as an inline headers= dict on the fetch call.
  • Don't preserve cargo-cult headers. If a comment claims the server "needs" a header set but §3 shows a header is inert, drop it rather than moving it.

© opensanctions, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in .claude/skills/crawler-constants-to-yml of opensanctions/opensanctions.

Open the folder on GitHubat commit 123b662

Compare with similar skills

Crawler Constants To Yml next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Crawler Constants To Yml compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Crawler Constants To Yml this skillopensanctions/opensanctions834—~1.4kAutomated safety check: NotesMIT
Tmuxtrpc-group/trpc-agent-go1.9k23 repos~868Automated safety check: PassApache-2.0
Ketch1broseidon/ketch7021 repos~3.9kAutomated safety check: PassMIT
Crawl4AI Web Scrapingsmallnest/goclaw5991 repos~2.5kAutomated safety check: PassMIT
Boss Zhipin Scrapereatmoreduck/boss-zhipin-scraper1.5k—~2.6kAutomated safety check: PassMIT
Axyusukebe/ax7191 repos~918Automated safety check: PassMIT

Similar skills

  • Tmux

    trpc-group/trpc-agent-go

    Remote-control tmux sessions for interactive CLIs by sending keystrokes and scraping pane output.

    1.9k GitHub starsUsed in 23 repos~868 tokens
    Data & AnalyticsAuto-check passed

  • Ketch

    1broseidon/ketch

    Research skill for ketch — a fast stateless CLI for web search, OSS code search, curated library docs, page scraping, and site crawling; an optional MCP server exists for operators who want it, but…

    702 GitHub starsUsed in 1 repo~3.9k tokens
    Data & AnalyticsAuto-check passed

  • Crawl4AI Web Scraping

    smallnest/goclaw

    Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.

    599 GitHub starsUsed in 1 repo~2.5k tokens
    Data & AnalyticsAuto-check passed

  • Boss Zhipin Scraper

    eatmoreduck/boss-zhipin-scraper

    Scrape BOSS直聘 (job listing site) via Chrome CDP. An agent skill from eatmoreduck/boss-zhipin-scraper.

    1.5k GitHub stars~2.6k tokensUpdated 11 days ago
    Data & AnalyticsAuto-check passed

  • Ax

    yusukebe/ax

    Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML.

    719 GitHub starsUsed in 1 repo~918 tokens
    Data & AnalyticsAuto-check passed

  • Anakinscraper

    Anakin-Inc/anakin

    Scrape any website into clean markdown or structured JSON. An agent skill from Anakin-Inc/anakin.

    4.5k GitHub stars~859 tokensUpdated 1 mo ago
    Data & AnalyticsAuto-check passed

More from opensanctions/opensanctions

All 11 skills in this repo

  • Crawler Pep

    opensanctions/opensanctions

    Scaffold a new PEP (Politically Exposed Persons) crawler — members of a parliament, legislature, senate, chamber of deputies, cabinet, judiciary, or an asset-declaration register — from a source URL…

    834 GitHub stars~1.9k tokensUpdated today
    Auto-check: notes

  • Dataset Metadata

    opensanctions/opensanctions

    Bring a dataset .yml's metadata in line with house conventions (title, summary, description, coverage, publisher, maintainer comments).

    834 GitHub stars~604 tokensUpdated today
    Auto-check: notes

  • Legislature Metadata

    opensanctions/opensanctions

    Refactor the title, description and coverage frequency of a legislature/parliament PEP dataset .yml into the house style.

    834 GitHub stars~953 tokensUpdated today
    Auto-check passed

  • Name Framework Migration First Step

    opensanctions/opensanctions

    Migrate ad-hoc name cleaning in a crawler to h.reviewnames (Step 1 of the name framework migration).

    834 GitHub stars~1.1k tokensUpdated today
    Auto-check passed

  • Refactor Crawler

    opensanctions/opensanctions

    Rewrite messy or AI-generated crawler code into clean, production-ready style that follows the zavod best practices.

    834 GitHub stars~1.1k tokensUpdated today
    Auto-check: notes

  • Release Datasets

    opensanctions/opensanctions

    Release one or more datasets by adding them to a topical collection, bumping coverage.start, and verifying.

    834 GitHub stars~566 tokensUpdated today
    Auto-check passed

Categories

Questions about Crawler Constants To Yml

What does Crawler Constants To Yml do?

Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever…. Crawler Constants To Yml is an agent skill from opensanctions/opensanctions.yml — as datapatch lookups wherever possible, otherwise http / config / dates metadata.

When should I use Crawler Constants To Yml?

Crawler Constants To Yml fits situations like: put in the yml a crawler constant such as GENDERS; tasks that involve Web scraping; tasks that involve Translation.

How do I install Crawler Constants To Yml in Claude Code?

Run `npx skills add opensanctions/opensanctions --skill crawler-constants-to-yml -a claude-code`. Or copy the skill folder (.claude/skills/crawler-constants-to-yml in opensanctions/opensanctions) into .claude/skills/crawler-constants-to-yml in your project. Claude Code loads it when a task matches its description.

How do I install Crawler Constants To Yml in Codex?

Run `npx skills add opensanctions/opensanctions --skill crawler-constants-to-yml -a codex`. Or copy the skill folder (.claude/skills/crawler-constants-to-yml in opensanctions/opensanctions) into .agents/skills/crawler-constants-to-yml in your project. Codex loads it when a task matches its description.

Can I use Crawler Constants To Yml in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add opensanctions/opensanctions --skill crawler-constants-to-yml -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/crawler-constants-to-yml, .gemini/skills/crawler-constants-to-yml, .github/skills/crawler-constants-to-yml and .opencode/skills/crawler-constants-to-yml in your project.

What does Crawler Constants To Yml need to run?

Going by SKILL.md and its folder, Crawler Constants To Yml needs the command-line tools its instructions call (ruff, mypy and python3). Our summary lists: Python 3. Its frontmatter pre-approves these tools: Read, Edit, Bash, Grep, Glob.

Does Crawler Constants To Yml access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Crawler Constants To Yml safe to install?

Our automated static check of SKILL.md found notes only (pre-approves every shell command (allowed-tools: bash)), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.

What licence does Crawler Constants To Yml use?

Crawler Constants To Yml is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Crawler Constants To Yml use?

About 1.4k tokens (SKILL.md is roughly 5.6k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Crawler Constants To Yml?

Skills that share tags, products or a category with Crawler Constants To Yml: Tmux (trpc-group/trpc-agent-go, 1.9k stars), Ketch (1broseidon/ketch, 702 stars), Crawl4AI Web Scraping (smallnest/goclaw, 599 stars) and Boss Zhipin Scraper (eatmoreduck/boss-zhipin-scraper, 1.5k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Crawler Constants To Yml?

opensanctions (a GitHub organization) maintains it in opensanctions/opensanctions, which has 834 GitHub stars. The repository holds 11 skills in this directory. The repository was last updated on October 9, 2026.

Source: opensanctions/opensanctions on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.