Chrome Devtools
einverne/dotfiles
Browser automation, debugging, and performance analysis using Puppeteer CLI scripts.
Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL.
Install Debug Crawler
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install opensanctions/opensanctions debug-crawler --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.claude/skills/debug-crawler .claude/skills/debug-crawler && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .claude/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawlerType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install opensanctions/opensanctions debug-crawler --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.claude/skills/debug-crawler .agents/skills/debug-crawler && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .agents/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install opensanctions/opensanctions debug-crawler --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.claude/skills/debug-crawler .cursor/skills/debug-crawler && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .cursor/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/opensanctions/opensanctions.git --path .claude/skills/debug-crawler--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install opensanctions/opensanctions debug-crawler --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.claude/skills/debug-crawler .gemini/skills/debug-crawler && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .gemini/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install opensanctions/opensanctions debug-crawlerInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .github/skills && cp -r skills-src/.claude/skills/debug-crawler .github/skills/debug-crawler && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .github/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add opensanctions/opensanctions --skill debug-crawler -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install opensanctions/opensanctions debug-crawler --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/opensanctions/opensanctions.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.claude/skills/debug-crawler .opencode/skills/debug-crawler && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "debug-crawler" agent skill from https://github.com/opensanctions/opensanctions/tree/main/.claude/skills/debug-crawler into .opencode/skills/debug-crawler/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "debug-crawler", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
debug-crawler- GitHub stars
- 832
- Token cost
- ~1.2k tokens
- SKILL.md length
- 513 words
- Files
- 1
- Skills in repo
- 11
- Repo updated
- First seen
- Licence
- MIT
At a glance
Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL.
- Works in 4 steps: Get the diagnostic report → Inspect the current source data → Diagnose → …
- Tasks that involve Web scraping
- SKILL.md covers Step 1: Get the diagnostic…, Step 2: Inspect the current…, Step 3: Diagnose and Step 4: Fix and verify
- Calls python; reaches api.zyte.com; needs OPENSANCTIONS_ZYTE_API_KEY and ZYTE_API_KEY
What it does
Debug Crawler is an agent skill from opensanctions/opensanctions. Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL. Covers pulling the diagnostic report, inspecting source data via Zyte, and common failure patterns including sources that are blocked, geo-blocked, 403/429-throttled, or behind a JavaScript challenge or anti-bot protection.
Its SKILL.md is about 1.2k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Data & Analytics, covering Web scraping. It works with JavaScript. The repository describes itself as: An open database of international sanctions data, persons of interest and politically exposed persons. The licence is MIT.
When your agent uses it
- Tasks that involve Web scraping
Example prompts
- “/debug-crawler”
Requirements
- Python 3
- A credential in OPENSANCTIONS_ZYTE_API_KEY
- A credential in ZYTE_API_KEY
- Pre-approved tools (allowed-tools): Read, Edit, Glob, Grep, Bash, WebFetch
Workflow steps
4 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit ce59ef9. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves these tools, so the agent can use them without asking each time:
ReadEditGlobGrepBashWebFetch
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
python
From the folder's file list and the shell code blocks in SKILL.md.
Network
Hosts in commands or code, which the agent is likely to contact:
api.zyte.com
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names these keys or tokens, usually read from environment variables:
OPENSANCTIONS_ZYTE_API_KEYZYTE_API_KEY
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Debug Crawler loads about 1.2k tokens when it runs. Until then it costs about 87 tokens; SKILL.md has 513 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check: notes
The automated check noted patterns worth knowing about, such as sudo or a known installer.
- NotePre-approves every shell command (allowed-tools: Bash)SKILL.md
allowed-tools: Read, Edit, Glob, Grep, Bash, WebFetch
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from opensanctions/opensanctions at commit ce59ef9, republished under its MIT licence (© opensanctions). 513 words, ~1,190 tokens.
Download SKILL.mdSave it as
.claude/skills/debug-crawler/SKILL.md (or your agent's skills folder).- name
- debug-crawler
- description
- Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL. Covers pulling the diagnostic report, inspecting source data via Zyte, and common failure patterns including sources that are blocked, geo-blocked, 403/429-throttled, or behind a JavaScript challenge or anti-bot protection.
- allowed-tools
- Read, Edit, Glob, Grep, Bash, WebFetch
- argument-hint
- <dataset name or issues.json URL>
Debug a Failing Crawler
The user has provided a dataset name or issues.json artifact URL: $ARGUMENTS
(In an artifact URL, the dataset name is the path segment after /artifacts/.)
Fix the failing crawler. Do not refactor or standardise it.
.claude/docs/crawler-guide.md is the hub for how crawlers are normally written, and
links the relevant zavod/docs best-practice guides.
Step 1: Get the diagnostic report
python -m contrib.maintenance.diagnose <dataset_name>Read the crawler's .yml and crawler.py from the paths the report resolves. Note
the row data on each issue — for source-value issues the keys are slugified
column names, values are cell contents.
Step 2: Inspect the current source data
The source has likely changed. Use OPENSANCTIONS_ZYTE_API_KEY (already set in the
environment) to fetch via Zyte when direct access times out or is blocked:
python3 -c "
import requests, os
from base64 import b64decode
ZYTE_API_KEY = os.environ['OPENSANCTIONS_ZYTE_API_KEY']
url = '<the Source data URL from the diagnostic report>'
resp = requests.post(
'https://api.zyte.com/v1/extract',
auth=(ZYTE_API_KEY, ''),
json={'url': url, 'httpResponseBody': True, 'httpResponseHeaders': True},
timeout=60
)
resp.raise_for_status()
content = b64decode(resp.json()['httpResponseBody'])
# then parse content as appropriate for the source format
"If the fix is to move the crawler onto Zyte (the source is now blocked, geo-blocked,
throttled, or behind a JavaScript challenge), see
zavod/docs/best_practices/http_operations.md for choosing the right helper
(fetch_html for browser rendering, fetch_text / fetch_json / fetch_resource
otherwise) and set ci_test: false on the dataset.
Step 3: Diagnose
Compare what the source actually contains against what the crawler expects.
If the question is "since when?"
Only when the diagnosis actually turns on how the dataset changed over time — counts
drifted outside the assertions: bounds, or you need to know since when runs have been
failing to line it up against a source or crawler change. Don't walk the history as a
matter of course; the diagnostic report already covers the latest run and the last
successful one, which is what most failures need.
python -m contrib.maintenance.versions <dataset_name> -n 30One row per archived run, newest first, with entity and target counts; add
--schema Person (repeatable) to see where a count moved. .claude/docs/archive-investigation.md
goes further, into individual past runs and deltas — follow it only in an interactive
session with a human, who likely has the Google Cloud credentials it needs. The command
above works over plain HTTPS.
Show full SKILL.md (199 more words)Show less
Common failures
| Symptom | Cause | Fix |
|---|---|---|
| Expected field/column not found | Source renamed or restructured columns | Update the crawler to match the new structure |
| First page parses fine, later pages fail | Per-page header handling no longer matches source | Adjust header-reading logic to match current source |
| 403 / empty response from Zyte | Source geo-restricts content | Add 'geolocation': 'US' (or the relevant country code) to the Zyte request, and the matching geolocation= to the crawler's fetch_resource / fetch_html call |
| Assertion on entity count fails | Source grew or shrank | Verify the count is real — the report's assertion table shows the drift vs the last successful run; check the linked delta.json for what changed. Update assertions: bounds if changes can be explained by e.g. sanctions expiring, but never widen the envelope to fit a collapsed count (that's a broken crawl, not drift). |
Unexpected keys in audit_data | New columns added to source | Pop and handle (or explicitly ignore) the new fields |
Step 4: Fix and verify
zavod crawl datasets/<path>/<dataset_name>.ymlCheck data/datasets/<dataset_name>/issues.log for remaining warnings. Then export
and confirm the delta is plausible:
zavod export datasets/<path>/<dataset_name>.ymlA healthy run shows:
- No errors in the crawl log
- Delta (added/deleted/modified) consistent with elapsed time since the last run
- Entity counts within the
assertions:bounds in the.yml
© opensanctions, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in .claude/skills/debug-crawler of opensanctions/opensanctions.
Open the folder on GitHubat commit ce59ef9
Compare with similar skills
Debug Crawler next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Debug Crawler this skillopensanctions/opensanctions | 832 | — | ~1.2k | Automated safety check: Notes | MIT | |
| Chrome Devtoolseinverne/dotfiles | 121 | 2 repos | ~1.6k | Automated safety check: Notes | Apache-2.0 | |
| Web Scrapingplatonai/Browser4 | 1.2k | — | ~1.1k | Automated safety check: Pass | Apache-2.0 | |
| Selenium Opinion Crawler123321kk/opinion-agent-ultimate | 107 | — | ~631 | Automated safety check: Pass | None | |
| Meowhub Browserzhaojiaqi/MeowHub | 111 | — | ~1.6k | Automated safety check: Pass | GPL-3.0 | |
| Web Unblockeroxylabs/agent-skills | 875 | — | ~865 | Automated safety check: Pass | MIT |
Similar skills
Web Scraping
platonai/Browser4
Extracts data from web pages using browser automation and CSS/JavaScript selectors.
Selenium Opinion Crawler
123321kk/opinion-agent-ultimate
browser-based page capture and text extraction for public-opinion research.
Meowhub Browser
zhaojiaqi/MeowHub
Browse the web using Browserless.io cloud browser service. An agent skill from zhaojiaqi/MeowHub.
Web Unblocker
oxylabs/agent-skills
Bypasses anti-bot protections using Oxylabs Web Unblocker, an AI-powered proxy that handles fingerprinting, JavaScript rendering, and retries automatically.
Agent Readiness Audit
indranilbanerjee/digital-marketing-pro
Audit whether AI agents and AI crawlers can actually use a site — robots.txt rules per AI crawler token (OpenAI, Anthropic and Perplexity bots, Google-Extended, Applebot-Extended)…
More from opensanctions/opensanctions
All 11 skills in this repoCrawler Pep
opensanctions/opensanctions
Scaffold a new PEP (Politically Exposed Persons) crawler — members of a parliament, legislature, senate, chamber of deputies, cabinet, judiciary, or an asset-declaration register — from a source URL…
Crawler Constants To Yml
opensanctions/opensanctions
Move hardcoded lookup/config constants (gender maps, header dicts, value translations, column-label maps, date formats) out of a crawler and into the dataset .yml — as datapatch lookups wherever…
Dataset Metadata
opensanctions/opensanctions
Bring a dataset .yml's metadata in line with house conventions (title, summary, description, coverage, publisher, maintainer comments).
Legislature Metadata
opensanctions/opensanctions
Refactor the title, description and coverage frequency of a legislature/parliament PEP dataset .yml into the house style.
Name Framework Migration First Step
opensanctions/opensanctions
Migrate ad-hoc name cleaning in a crawler to h.reviewnames (Step 1 of the name framework migration).
Refactor Crawler
opensanctions/opensanctions
Rewrite messy or AI-generated crawler code into clean, production-ready style that follows the zavod best practices.
Related
Works with
Categories
Questions about Debug Crawler
What does Debug Crawler do?
Investigate a failing crawler and propose a fix, starting from a dataset name or an issues.json artifact URL. Debug Crawler is an agent skill from opensanctions/opensanctions.json artifact URL.
When should I use Debug Crawler?
Debug Crawler fits situations like: tasks that involve Web scraping.
How do I install Debug Crawler in Claude Code?
Run `npx skills add opensanctions/opensanctions --skill debug-crawler -a claude-code`. Or copy the skill folder (.claude/skills/debug-crawler in opensanctions/opensanctions) into .claude/skills/debug-crawler in your project. Claude Code loads it when a task matches its description.
How do I install Debug Crawler in Codex?
Run `npx skills add opensanctions/opensanctions --skill debug-crawler -a codex`. Or copy the skill folder (.claude/skills/debug-crawler in opensanctions/opensanctions) into .agents/skills/debug-crawler in your project. Codex loads it when a task matches its description.
Can I use Debug Crawler in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add opensanctions/opensanctions --skill debug-crawler -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/debug-crawler, .gemini/skills/debug-crawler, .github/skills/debug-crawler and .opencode/skills/debug-crawler in your project.
What does Debug Crawler need to run?
Going by SKILL.md and its folder, Debug Crawler needs the command-line tools its instructions call (python) and credentials named OPENSANCTIONS_ZYTE_API_KEY and ZYTE_API_KEY. Our summary lists: Python 3; A credential in OPENSANCTIONS_ZYTE_API_KEY; A credential in ZYTE_API_KEY. Its frontmatter pre-approves these tools: Read, Edit, Glob, Grep, Bash, WebFetch.
Does Debug Crawler access the network?
SKILL.md names 1 domain. In commands or code: api.zyte.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.
Is Debug Crawler safe to install?
Our automated static check of SKILL.md found notes only (pre-approves every shell command (allowed-tools: bash)), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.
What licence does Debug Crawler use?
Debug Crawler is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Debug Crawler use?
About 1.2k tokens (SKILL.md is roughly 4.8k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Debug Crawler?
Skills that share tags, products or a category with Debug Crawler: Chrome Devtools (einverne/dotfiles, 121 stars), Web Scraping (platonai/Browser4, 1.2k stars), Selenium Opinion Crawler (123321kk/opinion-agent-ultimate, 107 stars) and Meowhub Browser (zhaojiaqi/MeowHub, 111 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Debug Crawler?
opensanctions (a GitHub organization) maintains it in opensanctions/opensanctions, which has 832 GitHub stars. The repository holds 11 skills in this directory. The repository was last updated on October 8, 2026.
Source: opensanctions/opensanctions on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.