Agent skill

Web Scraper

Scrape, crawl, and extract data from websites. An agent skill from shobcoder/shob.

Install Web Scraper

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/web-scraper .claude/skills/web-scraper && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .claude/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/shobcoder/shob/tree/main/skills/web-scraper

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/web-scraper .agents/skills/web-scraper && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .agents/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/web-scraper .cursor/skills/web-scraper && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .cursor/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/shobcoder/shob.git --path skills/web-scraper

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/web-scraper .gemini/skills/web-scraper && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .gemini/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/web-scraper .github/skills/web-scraper && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .github/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add shobcoder/shob --skill web-scraper -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install shobcoder/shob web-scraper --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/shobcoder/shob.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/web-scraper .opencode/skills/web-scraper && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "web-scraper" agent skill from https://github.com/shobcoder/shob/tree/main/skills/web-scraper into .opencode/skills/web-scraper/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "web-scraper", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: web-scraper
GitHub stars: 577
Token cost: ~700 tokens
SKILL.md length: 171 words
Files: 1
Skills in repo: 6
Repo updated: Sep 18, 2026
First seen: Oct 7, 2026
Licence: MIT

At a glance

Scrape, crawl, and extract data from websites. An agent skill from shobcoder/shob.

Works in 5 steps: Start with simpler extraction before… → Use specific prompts for targeted data → Respect website terms of service → …
Users ask to scrape web pages
SKILL.md covers Overview, When to Use, Tools Available and Usage Patterns, plus 3 more sections
Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Web Scraper is an agent skill from shobcoder/shob. Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.

Its SKILL.md is about 700 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Data & Analytics, covering Web scraping. The repository describes itself as: Shob – an AI agent that delivers high-quality coding & automation work. The licence is MIT.

When your agent uses it

Users ask to scrape web pages
Extract content
Collect data from the internet

Example prompts

“/web-scraper”

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

Start with simpler extraction before complex patterns
Use specific prompts for targeted data
Respect website terms of service
Add delays between requests when scraping multiple pages
Handle errors gracefully with try/catch

What it can do on your machine

Read from SKILL.md and the folder at commit 14831ba. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md (its code samples are javascript).
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Web Scraper loads about 700 tokens when it runs. Until then it costs about 42 tokens; SKILL.md has 171 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~42
When it runs · the whole SKILL.md, loaded when a task matches: ~700

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from shobcoder/shob at commit 14831ba, republished under its MIT licence (© shobcoder). 171 words, ~700 tokens.

Download SKILL.mdSave it as .claude/skills/web-scraper/SKILL.md (or your agent's skills folder).

name: web-scraper
description: Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.

Web Scraper

Overview

Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction.

When to Use

Extract text content from web pages
Crawl entire websites
Collect structured data
Research and gather information
Monitor website changes
Extract tables and lists

Tools Available

Content Extraction

javascript

// Use extract_content_from_websites for structured extraction
// Supports batch processing of multiple URLs
// Returns JSON format with extracted content

Task Format

javascript

{
    tasks: [
        {
            url: "https://example.com",
            prompt: "Extract specific information",
            task_name: "optional_name"
        }
    ]
}

Usage Patterns

Simple Content Extraction

javascript

// Extract main content from a page
const result = await extract_content_from_websites({
    tasks: [{
        url: "https://news.example.com/article",
        prompt: "Extract the title, author, date, and main content"
    }]
});

Batch URL Processing

javascript

// Process multiple URLs in parallel
const urls = [
    "https://site.com/page1",
    "https://site.com/page2",
    "https://site.com/page3"
];

const results = await extract_content_from_websites({
    tasks: urls.map((url, i) => ({
        url,
        prompt: "Extract all product information, prices, and descriptions",
        task_name: `product_${i}`
    }))
});

Data Mining

javascript

// Extract structured data like prices, reviews, specifications
const data = await extract_content_from_websites({
    tasks: [{
        url: "https://ecommerce.example.com/products",
        prompt: "Extract product name, price, rating, and availability for all products listed"
    }]
});

Extraction Modes

Auto Mode (Default)

Attempts HTTP GET first
Falls back to browser rendering for CSR pages
Best for most websites

Curl Only Mode

Fast direct HTTP requests
Best for static HTML pages
May fail on JavaScript-heavy sites

Browser Only Mode

Full browser rendering
Handles dynamic content
Slower but more comprehensive

Best Practices

Start with simpler extraction before complex patterns
Use specific prompts for targeted data
Respect website terms of service
Add delays between requests when scraping multiple pages
Handle errors gracefully with try/catch

Data Handling

Returns JSON format for easy processing
Handles batch operations efficiently
Supports pagination when needed
Maintains data structure in results

Files

Just SKILL.md in skills/web-scraper of shobcoder/shob.

Open the folder on GitHubat commit 14831ba

Compare with similar skills

Web Scraper next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Web Scraper compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Web Scraper this skillshobcoder/shob	577	—	~700	Automated safety check: Pass	MIT	19 days ago
Tmuxtrpc-group/trpc-agent-go	1.8k	23 repos	~868	Automated safety check: Pass	Apache-2.0	today
Ketch1broseidon/ketch	696	1 repos	~3.9k	Automated safety check: Pass	MIT	today
Crawl4AI Web Scrapingsmallnest/goclaw	598	1 repos	~2.5k	Automated safety check: Pass	MIT	6 mo ago
Boss Zhipin Scrapereatmoreduck/boss-zhipin-scraper	1.5k	—	~2.6k	Automated safety check: Pass	MIT	8 days ago
Axyusukebe/ax	719	1 repos	~918	Automated safety check: Pass	MIT	2 mo ago

Similar skills

Tmux
trpc-group/trpc-agent-go
Remote-control tmux sessions for interactive CLIs by sending keystrokes and scraping pane output.
1.8k GitHub starsUsed in 23 repos~868 tokens
Data & AnalyticsAuto-check passed
Ketch
1broseidon/ketch
Research skill for ketch — a fast stateless CLI for web search, OSS code search, curated library docs, page scraping, and site crawling; an optional MCP server exists for operators who want it, but…
696 GitHub starsUsed in 1 repo~3.9k tokens
Data & AnalyticsAuto-check passed
Crawl4AI Web Scraping
smallnest/goclaw
Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.
598 GitHub starsUsed in 1 repo~2.5k tokens
Data & AnalyticsAuto-check passed
Boss Zhipin Scraper
eatmoreduck/boss-zhipin-scraper
Scrape BOSS直聘 (job listing site) via Chrome CDP. An agent skill from eatmoreduck/boss-zhipin-scraper.
1.5k GitHub stars~2.6k tokensUpdated 8 days ago
Data & AnalyticsAuto-check passed
Ax
yusukebe/ax
Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML.
719 GitHub starsUsed in 1 repo~918 tokens
Data & AnalyticsAuto-check passed
Anakinscraper
Anakin-Inc/anakin
Scrape any website into clean markdown or structured JSON. An agent skill from Anakin-Inc/anakin.
4.5k GitHub stars~859 tokensUpdated 1 mo ago
Data & AnalyticsAuto-check passed

Questions about Web Scraper

What does Web Scraper do?

Scrape, crawl, and extract data from websites. An agent skill from shobcoder/shob. Web Scraper is an agent skill from shobcoder/shob. Scrape, crawl, and extract data from websites.

When should I use Web Scraper?

Web Scraper fits situations like: users ask to scrape web pages; extract content; collect data from the internet.

How do I install Web Scraper in Claude Code?

Run `npx skills add shobcoder/shob --skill web-scraper -a claude-code`. Or copy the skill folder (skills/web-scraper in shobcoder/shob) into .claude/skills/web-scraper in your project. Claude Code loads it when a task matches its description.

How do I install Web Scraper in Codex?

Run `npx skills add shobcoder/shob --skill web-scraper -a codex`. Or copy the skill folder (skills/web-scraper in shobcoder/shob) into .agents/skills/web-scraper in your project. Codex loads it when a task matches its description.

Can I use Web Scraper in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add shobcoder/shob --skill web-scraper -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/web-scraper, .gemini/skills/web-scraper, .github/skills/web-scraper and .opencode/skills/web-scraper in your project.

What does Web Scraper need to run?

SKILL.md names no scripts, command-line tools or credentials: Web Scraper is instructions for the agent only.

Does Web Scraper access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Web Scraper safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Web Scraper use?

Web Scraper is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Web Scraper use?

About 700 tokens (SKILL.md is roughly 2.8k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Web Scraper?

Skills that share tags, products or a category with Web Scraper: Tmux (trpc-group/trpc-agent-go, 1.8k stars), Ketch (1broseidon/ketch, 696 stars), Crawl4AI Web Scraping (smallnest/goclaw, 598 stars) and Boss Zhipin Scraper (eatmoreduck/boss-zhipin-scraper, 1.5k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Web Scraper?

shobcoder (a GitHub organization) maintains it in shobcoder/shob, which has 577 GitHub stars. The repository holds 6 skills in this directory. The repository was last updated on September 18, 2026.

Source: shobcoder/shob on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Install Web Scraper

Facts

At a glance

What it does

When your agent uses it

Example prompts

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Web Scraper

Overview

When to Use

Tools Available

Content Extraction

Task Format

Usage Patterns

Simple Content Extraction

Batch URL Processing

Data Mining

Extraction Modes

Auto Mode (Default)

Curl Only Mode

Browser Only Mode

Best Practices

Data Handling

Files

Compare with similar skills

Similar skills

More from shobcoder/shob

Related

Questions about Web Scraper