Agent skill

Web Scraper

by guia-matthieu in guia-matthieu/clawfu-skills

Extract structured data from websites. An agent skill from guia-matthieu/clawfu-skills.

Install Web Scraper

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/automation/web-scraper .claude/skills/web-scraper && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .claude/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/automation/web-scraper .agents/skills/web-scraper && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .agents/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/automation/web-scraper .cursor/skills/web-scraper && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .cursor/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/guia-matthieu/clawfu-skills.git --path skills/automation/web-scraper

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/automation/web-scraper .gemini/skills/web-scraper && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .gemini/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/automation/web-scraper .github/skills/web-scraper && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .github/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install guia-matthieu/clawfu-skills web-scraper --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/guia-matthieu/clawfu-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/automation/web-scraper .opencode/skills/web-scraper && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "web-scraper" agent skill from https://github.com/guia-matthieu/clawfu-skills/tree/main/skills/automation/web-scraper into .opencode/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: web-scraper
GitHub stars: 150
Token cost: ~981 tokens
SKILL.md length: 247 words
Files: 3 (incl. scripts)
Skills in repo: 28
Repo updated: Oct 1, 2026
First seen: Oct 7, 2026
Licence: MIT

At a glance

Extract structured data from websites. An agent skill from guia-matthieu/clawfu-skills.

Works in 5 steps: Check robots.txt - Respect site's… → Rate limit - Don't overload servers (1-2… → Identify yourself - Use descriptive… → …
: collecting competitor pricing
SKILL.md covers When to Use This Skill, What Claude Does vs What You…, Dependencies and Commands, plus 6 more sections
Runs Python scripts from its folder; calls python and pip; reaches competitor.com

What it does

Web Scraper is an agent skill from guia-matthieu/clawfu-skills. Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

Its SKILL.md is about 980 tokens, which your agent loads only when the skill is triggered. The skill folder holds 3 other files, including scripts (for example `scripts/main.py`).

It sits in Data & Analytics, covering Web scraping and Schema markup. The repository describes itself as: 172 expert marketing skills for AI agents — ClawFu MCP Server. The licence is MIT.

When your agent uses it

: collecting competitor pricing
Scraping product listings
Extracting contact information
Gathering research data

Example prompts

“/web-scraper”

Requirements

Python 3

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

Check robots.txt - Respect site's scraping policy
Rate limit - Don't overload servers (1-2 req/sec)
Identify yourself - Use descriptive User-Agent
Cache requests - Don't re-scrape unchanged pages
Terms of Service - Check if scraping is allowed

What it can do on your machine

Read from SKILL.md and the folder at commit 4108f5c. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Ships 2 files in scripts/ (Python), which the agent can run.
Shell commands in SKILL.md call:
- python
- pip
From the folder's file list and the shell code blocks in SKILL.md.
Network
Hosts in commands or code, which the agent is likely to contact:
- competitor.com
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Web Scraper loads about 981 tokens when it runs. Until then it costs about 51 tokens; SKILL.md has 247 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~51
When it runs · the whole SKILL.md, loaded when a task matches: ~981

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from guia-matthieu/clawfu-skills at commit 4108f5c, republished under its MIT licence (© guia-matthieu). 247 words, ~981 tokens.

Download SKILL.mdSave it as .claude/skills/web-scraper/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.

name: web-scraper
description: Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes
license: MIT
metadata.author: ClawFu
metadata.version: 1.0.0
metadata.mcp-server: @clawfu/mcp-skills

Web Scraper

Extract structured data from websites using BeautifulSoup and requests - turn any webpage into usable data.

When to Use This Skill

Competitor research - Scrape pricing, features, positioning
Lead generation - Extract contact info from directories
Content audit - Pull headings, links, meta data
Price monitoring - Track competitor pricing changes
Data collection - Gather research data from multiple sources

What Claude Does vs What You Decide

Claude Does	You Decide
Structures analysis frameworks	Strategic priorities
Synthesizes market data	Competitive positioning
Identifies opportunities	Resource allocation
Creates strategic options	Final strategy selection
Suggests implementation approaches	Execution decisions

Dependencies

bash

pip install beautifulsoup4 requests pandas click lxml

Commands

Scrape Elements

bash

python scripts/main.py scrape https://example.com --selector "h1,h2,p"
python scripts/main.py scrape https://example.com --selector ".product-price"

Extract Links

bash

python scripts/main.py links https://example.com
python scripts/main.py links https://example.com --internal-only

Extract Emails

bash

python scripts/main.py emails https://example.com
python scripts/main.py emails https://example.com --depth 2

Extract Structured Data

bash

python scripts/main.py structured https://example.com/article --schema article
python scripts/main.py structured https://example.com/product --schema product

Examples

Example 1: Scrape Competitor Pricing

bash

python scripts/main.py scrape https://competitor.com/pricing --selector ".price,.plan-name"

# Output:
# Extracted 6 elements
# 1. Starter - $29/mo
# 2. Pro - $99/mo
# 3. Enterprise - Contact us

Example 2: Extract Article Content

bash

python scripts/main.py structured https://blog.example.com/post --schema article

# Output: article_data.json
# {
#   "title": "How to Scale Your Startup",
#   "author": "Jane Doe",
#   "date": "2024-01-15",
#   "content": "...",
#   "word_count": 1523
# }

CSS Selector Reference

Selector	Description	Example
`tag`	Element type	`h1`, `p`, `div`
`.class`	Class name	`.price`, `.title`
`#id`	Element ID	`#main-content`
`tag.class`	Tag with class	`div.product`
`tag[attr]`	Has attribute	`a[href]`
`parent > child`	Direct child	`ul > li`
`tag1, tag2`	Multiple	`h1, h2, h3`

Ethical Scraping Guidelines

Check robots.txt - Respect site's scraping policy
Rate limit - Don't overload servers (1-2 req/sec)
Identify yourself - Use descriptive User-Agent
Cache requests - Don't re-scrape unchanged pages
Terms of Service - Check if scraping is allowed

Skill Boundaries

What This Skill Does Well

Structuring strategic analysis
Identifying market opportunities
Creating strategic frameworks
Synthesizing competitive data

What This Skill Cannot Do

Replace market research
Guarantee strategic success
Know proprietary competitor info
Make executive decisions

competitor-monitor - Monitor competitor changes
pdf-extractor - Extract from PDFs

Skill Metadata

Mode: centaur

yaml

category: automation
subcategory: data-extraction
dependencies: [beautifulsoup4, requests, pandas]
difficulty: intermediate
time_saved: 5+ hours/week

Files

SKILL.md and 2 other files (scripts) in skills/automation/web-scraper of guia-matthieu/clawfu-skills.

SKILL.md
scripts/main.py
scripts/requirements.txt

Open the folder on GitHubat commit 4108f5c

Compare with similar skills

Web Scraper next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Web Scraper compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Web Scraper this skillguia-matthieu/clawfu-skills	150	—	~981	Automated safety check: Pass	MIT	7 days ago
Axyusukebe/ax	719	1 repos	~918	Automated safety check: Pass	MIT	2 mo ago
Meowhub Browserzhaojiaqi/MeowHub	111	—	~1.6k	Automated safety check: Pass	GPL-3.0	4 mo ago
Chatgpt SearchSeifBenayed/cloclo	114	—	~1.7k	Automated safety check: Notes	MIT	6 mo ago
Agent Readiness Auditindranilbanerjee/digital-marketing-pro	855	1 repos	~3.9k	Automated safety check: Pass	MIT	5 days ago
SEO Firecrawlseranking/seo-skills	160	—	~2.3k	Automated safety check: Pass	MIT	3 mo ago

Similar skills

Ax
yusukebe/ax
Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML.
719 GitHub starsUsed in 1 repo~918 tokens
Data & AnalyticsAuto-check passed
Meowhub Browser
zhaojiaqi/MeowHub
Browse the web using Browserless.io cloud browser service. An agent skill from zhaojiaqi/MeowHub.
111 GitHub stars~1.6k tokensUpdated 4 mo ago
Data & AnalyticsAuto-check passed
Chatgpt Search
SeifBenayed/cloclo
Search ChatGPT and extract the full response + hydration JSON that powers the UI.
114 GitHub stars~1.7k tokensUpdated 6 mo ago
Data & AnalyticsAuto-check: notes
Agent Readiness Audit
indranilbanerjee/digital-marketing-pro
Audit whether AI agents and AI crawlers can actually use a site — robots.txt rules per AI crawler token (OpenAI, Anthropic and Perplexity bots, Google-Extended, Applebot-Extended)…
855 GitHub starsUsed in 1 repo~3.9k tokens
Data & AnalyticsAuto-check passed
SEO Firecrawl
seranking/seo-skills
Ad-hoc web scraping, site mapping, and full-site crawling via Firecrawl MCP.
160 GitHub stars~2.3k tokensUpdated 3 mo ago
Data & AnalyticsAuto-check passed
Facebook Page Posts
browser-act/skills
Scrapes posts from any public Facebook Page timeline, returning structured data including post text, author info, engagement metrics (likes/comments/shares), reaction breakdowns…
6.1k GitHub stars~2.3k tokensUpdated 1 mo ago
Data & AnalyticsAuto-check passed

Questions about Web Scraper

What does Web Scraper do?

Extract structured data from websites. An agent skill from guia-matthieu/clawfu-skills. Web Scraper is an agent skill from guia-matthieu/clawfu-skills. Extract structured data from websites.

When should I use Web Scraper?

Web Scraper fits situations like: : collecting competitor pricing; scraping product listings; extracting contact information; gathering research data.

How do I install Web Scraper in Claude Code?

Run `npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a claude-code`. Or copy the skill folder (skills/automation/web-scraper in guia-matthieu/clawfu-skills) into .claude/skills/web-scraper in your project. Claude Code loads it when a task matches its description.

How do I install Web Scraper in Codex?

Run `npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a codex`. Or copy the skill folder (skills/automation/web-scraper in guia-matthieu/clawfu-skills) into .agents/skills/web-scraper in your project. Codex loads it when a task matches its description.

Can I use Web Scraper in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add guia-matthieu/clawfu-skills --skill web-scraper -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/web-scraper, .gemini/skills/web-scraper, .github/skills/web-scraper and .opencode/skills/web-scraper in your project.

What does Web Scraper need to run?

Going by SKILL.md and its folder, Web Scraper needs Python for the scripts in its folder and the command-line tools its instructions call (python and pip). Our summary lists: Python 3.

Does Web Scraper access the network?

SKILL.md names 1 domain. In commands or code: competitor.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is Web Scraper safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Web Scraper use?

Web Scraper is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Web Scraper use?

About 981 tokens (SKILL.md is roughly 3.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Web Scraper?

Skills that share tags, products or a category with Web Scraper: Ax (yusukebe/ax, 719 stars), Meowhub Browser (zhaojiaqi/MeowHub, 111 stars), Chatgpt Search (SeifBenayed/cloclo, 114 stars) and Agent Readiness Audit (indranilbanerjee/digital-marketing-pro, 855 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Web Scraper?

guia-matthieu (a GitHub user) maintains it in guia-matthieu/clawfu-skills, which has 150 GitHub stars. The repository holds 28 skills in this directory. The repository was last updated on October 1, 2026.

Source: guia-matthieu/clawfu-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Install Web Scraper

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Web Scraper

When to Use This Skill

What Claude Does vs What You Decide

Dependencies

Commands

Scrape Elements

Extract Links

Extract Emails

Extract Structured Data

Examples

Example 1: Scrape Competitor Pricing

Example 2: Extract Article Content

CSS Selector Reference

Ethical Scraping Guidelines

Skill Boundaries

What This Skill Does Well

What This Skill Cannot Do

Related Skills

Skill Metadata

Files

Compare with similar skills

Similar skills

More from guia-matthieu/clawfu-skills

Related

Questions about Web Scraper