Crawl4AI Web Scraping
smallnest/goclaw
Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.
Working with Owl — OWID's lightweight, single-folder pipeline runner for small datasets (an alternative to the full snapshot → meadow → garden → grapher chain).
$ npx skills add owid/etl --skill tool-owl -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install owid/etl tool-owl --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.claude/skills/tool-owl .claude/skills/tool-owl && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .claude/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/owid/etl/tree/master/.claude/skills/tool-owlType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add owid/etl --skill tool-owl -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install owid/etl tool-owl --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.claude/skills/tool-owl .agents/skills/tool-owl && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .agents/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add owid/etl --skill tool-owl -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install owid/etl tool-owl --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.claude/skills/tool-owl .cursor/skills/tool-owl && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .cursor/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/owid/etl.git --path .claude/skills/tool-owl--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add owid/etl --skill tool-owl -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install owid/etl tool-owl --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.claude/skills/tool-owl .gemini/skills/tool-owl && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .gemini/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install owid/etl tool-owlInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add owid/etl --skill tool-owl -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .github/skills && cp -r skills-src/.claude/skills/tool-owl .github/skills/tool-owl && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .github/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add owid/etl --skill tool-owl -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install owid/etl tool-owl --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/owid/etl.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.claude/skills/tool-owl .opencode/skills/tool-owl && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "tool-owl" agent skill from https://github.com/owid/etl/tree/master/.claude/skills/tool-owl into .opencode/skills/tool-owl/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "tool-owl", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
tool-owlWorking with Owl — OWID's lightweight, single-folder pipeline runner for small datasets (an alternative to the full snapshot → meadow → garden → grapher chain).
Tool Owl is an agent skill from owid/etl. Working with Owl — OWID's lightweight, single-folder pipeline runner for small datasets (an alternative to the full snapshot → meadow → garden → grapher chain). Use whenever the user mentions Owl, owlsteps/, the owl CLI (owl new/run/snapshot/viz), or wants to create, migrate, run, or debug a small dataset with Owl.
Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Data & Analytics, covering Data pipelines and ETL. The repository describes itself as: A compute graph for loading and transforming OWID's data. The licence is MIT.
7 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit bf5dc8e. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
makeFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Tool Owl loads about 1.4k tokens when it runs. Until then it costs about 85 tokens; SKILL.md has 496 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from owid/etl at commit bf5dc8e, republished under its MIT licence (© owid). 496 words, ~1,350 tokens.
.claude/skills/tool-owl/SKILL.md (or your agent's skills folder).Owl is OWID's lightweight pipeline runner: one step folder (owl_steps/<namespace>/<dataset>/vYYYYMMDD/) holds the snapshot, transform, and metadata, and writes a normal owid.catalog dataset under data/garden/. It replaces the full snapshot → meadow → garden → grapher chain for small datasets.
The authoritative reference is lib/owl/owl/README.md — read it before doing anything with Owl. It covers the commands, project layout, @Snapshot/@Dataset/@Action API, snapshot capture helpers, meta.yml, loading existing ETL datasets, the Grapher action, and current limitations. Do not duplicate it from memory — open it.
The framework is small and readable; for API details beyond the README, go to the source:
| File | What's in it |
|---|---|
lib/owl/owl/cli.py | owl new / run / snapshot / viz commands and flags |
lib/owl/owl/snapshot.py | @Snapshot, SnapshotCapture (download/add/write_bytes/write_text), lock behavior |
lib/owl/owl/dataset.py | @Dataset, @Action, ColumnMeta, DatasetMeta, metadata merging, staleness |
lib/owl/owl/catalog.py | load_snapshot (Table + origins), export |
lib/owl/owl/etl_dataset.py | ETLDataset (read-only deps on main-ETL datasets) |
/create-etl-steps, /update-dataset) — mature, high-traffic, or multi-stage datasets that need real DAG integration. Owl is experimental, has limited metadata validation, and no DAG integration.If a request doesn't clearly fit Owl, say so and recommend classic ETL instead of forcing it.
.venv/bin/owl new <namespace>/<dataset> [--date YYYY-MM-DD], then fill in step.py and meta.yml, then owl snapshot → owl run. Follow the README's "Creating a new step", "Snapshots", and "Datasets" sections.
Collapse the snapshot → meadow → garden → grapher chain into one Owl step folder:
owl new, or hand-create vYYYYMMDD/), using a --date matching the source's access date.@Snapshot (e.g. snap.download(URL, suffix=".csv")).@Dataset — load with load_snapshot(...), harmonize, tb.format([...])..meta.yml into the Owl meta.yml: the datasets.<name> block accepts the standard catalog shape (definitions, tables → variables), so most metadata moves over verbatim. Snapshot origin goes under snapshots.<name>.origin.@Action(kind="grapher", default=False) that calls upsert_dataset(...)..py/.dvc, meadow, garden + meta, grapher) and their active DAG entries.owl snapshot + owl run, then verify the output dataset matches the old one.owl run <pattern> (regex; rebuilds stale steps + upstream deps — no --force needed after edits), owl run <pattern> --grapher for Grapher actions, owl run <pattern> --force to rebuild regardless, owl viz <pattern> to render the dependency DAG. owl snapshot is the only way to fetch data — run never fetches.
step.py pattern (real datasets)For anything beyond a toy dataset, use the catalog-Table pattern (preserves origins/metadata properly), rather than the lightweight inline-DatasetMeta scaffold:
from pathlib import Path
from owid.catalog import Table
from owl import Action, Dataset, Snapshot, SnapshotCapture
from owl.catalog import load_snapshot
from owl.grapher import upsert_dataset
from etl.data_helpers import geo
URL_DOWNLOAD = "https://example.com/data.csv"
COUNTRIES_FILE = Path(__file__).with_name("my_dataset.countries.json")
@Snapshot
def raw_data(snap: SnapshotCapture) -> None:
snap.download(URL_DOWNLOAD, suffix=".csv")
@Dataset
def my_dataset(raw_data: Snapshot) -> Table:
tb = load_snapshot(raw_data) # Table with origins from meta.yml `origin`
tb = tb.rename(columns={"country_name": "country"})
tb = geo.harmonize_countries(df=tb, countries_file=COUNTRIES_FILE)
tb = tb.format(["country", "year"])
return tb
@Action(kind="grapher", default=False)
def upsert_to_grapher(my_dataset: Dataset) -> None:
upsert_dataset(my_dataset)Dependencies wire by parameter name (a raw_data parameter resolves to the @Snapshot/@Dataset named raw_data in the module).
step.pynp.where, no pd.concat/pd.to_numeric (use pr.* from owid.catalog.processing), no pd.DataFrame(tb). See the "Preserving metadata/origins" rules in CLAUDE.md.make check before committing. Don't commit, push, or open PRs unless explicitly told to.© owid, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .claude/skills/tool-owl of owid/etl.
Open the folder on GitHubat commit bf5dc8e
Tool Owl next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Tool Owl this skillowid/etl | 158 | — | ~1.4k | Automated safety check: Pass | MIT | |
| Crawl4AI Web Scrapingsmallnest/goclaw | 598 | 1 repos | ~2.5k | Automated safety check: Pass | MIT | |
| Glue 09 10 Migrationaws-samples/aws-glue-samples | 1.5k | — | ~2.4k | Automated safety check: Pass | MIT-0 | |
| Migrate Glue Devendpoint To Interactive Sessionsaws-samples/aws-glue-samples | 1.5k | — | ~3.6k | Automated safety check: Pass | MIT-0 | |
| Dbt Databricks PR Readydatabricks/dbt-databricks | 380 | — | ~2.8k | Automated safety check: Pass | Apache-2.0 | |
| Mz Dbt ReleaseMaterializeInc/materialize | 6.4k | — | ~1.2k | Automated safety check: Pass | Custom licence |
smallnest/goclaw
Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.
aws-samples/aws-glue-samples
Upgrade an AWS Glue ETL job from Glue version 0.9 or 1.0 to Glue 4.0.
aws-samples/aws-glue-samples
Migrate a legacy AWS Glue development endpoint to a Glue interactive session, following the official AWS migration checklist.
databricks/dbt-databricks
A skill your agent uses for an open dbt-databricks pull request, including your own PR or a fork PR, to assess merge readiness and optionally repair selected gaps on the PR head branch.
MaterializeInc/materialize
Cut a dbt-materialize PyPI release: bump the version in version.py and setup.py, date the Unreleased CHANGELOG entry, and open the release PR with a Ship: <url body.
liam-machine/erd-studio
Friendly, step-by-step setup for ERD Studio in an existing dbt project, for people who may be new to dbt or data modelling.
owid/etl
Find every OWID surface that references a chart, indicator, MDIM, or explorer — articles (links vs embeds), explorers, narrative charts, data insights, static viz, key-chart slots, MDIM views.
owid/etl
Add a scatter view (with GDP per capita on x) to existing OWID charts via the admin API, mirroring the admin UI's "Add scatter type" defaults, then retire the old standalone "X vs.
owid/etl
Add new survey question codes (e.g. An agent skill from owid/etl.
owid/etl
Build or refresh an OWID static visualization end to end — resolve what data it needs from an old static viz image, an indicator, or a grapher chart; check both the ETL catalog and the producer's…
owid/etl
Propose redirects from (soon-to-sunset) grapher charts to the matching views of published MDIMs.
owid/etl
Take (soon-to-sunset) OWID explorers to redirected MDIMs, end to end.
Categories
Working with Owl — OWID's lightweight, single-folder pipeline runner for small datasets (an alternative to the full snapshot → meadow → garden → grapher chain). Tool Owl is an agent skill from owid/etl. Working with Owl — OWID's lightweight, single-folder pipeline runner for small datasets (an alternative to the full snapshot → meadow → garden → grapher chain).
Tool Owl fits situations like: the user mentions Owl; the owl CLI (owl new/run/snapshot/viz); wants to create; debug a small dataset with Owl.
Run `npx skills add owid/etl --skill tool-owl -a claude-code`. Or copy the skill folder (.claude/skills/tool-owl in owid/etl) into .claude/skills/tool-owl in your project. Claude Code loads it when a task matches its description.
Run `npx skills add owid/etl --skill tool-owl -a codex`. Or copy the skill folder (.claude/skills/tool-owl in owid/etl) into .agents/skills/tool-owl in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add owid/etl --skill tool-owl -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/tool-owl, .gemini/skills/tool-owl, .github/skills/tool-owl and .opencode/skills/tool-owl in your project.
Going by SKILL.md and its folder, Tool Owl needs the command-line tools its instructions call (make). Our summary lists: Python 3.
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Tool Owl is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.4k tokens (SKILL.md is roughly 5.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Tool Owl: Crawl4AI Web Scraping (smallnest/goclaw, 598 stars), Glue 09 10 Migration (aws-samples/aws-glue-samples, 1.5k stars), Migrate Glue Devendpoint To Interactive Sessions (aws-samples/aws-glue-samples, 1.5k stars) and Dbt Databricks PR Ready (databricks/dbt-databricks, 380 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
owid (a GitHub organization) maintains it in owid/etl, which has 158 GitHub stars. The repository holds 35 skills in this directory. The repository was last updated on October 8, 2026.
Source: owid/etl on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.