Crawl4AI Web Scraping

What it does

When your agent uses it

• Scraping a website into clean markdown
• Extracting structured data from JavaScript-heavy pages
• Crawling many URLs in a batch
• Building an automated web data pipeline

Example prompts

• “Crawl our docs site and save each page as markdown.”
• “Extract product names and prices from this listing page with a schema, without using an LLM.”
• “Crawl these twenty URLs in a batch and give me the links found on each.”

Requirements

• Python with the crawl4ai package, installed with pip and crawl4ai-setup

Workflow steps

2 steps, taken from the step headings in SKILL.md.

1. 1Schema-Based CSS Extraction (Most Efficient)
2. 2LLM-Based Extraction

What it can do on your machine

Read from SKILL.md and the folder at commit e05c79d. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 4 files in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • python
  • pip

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • shop.com
  • site1.com
  • site2.com
  • site3.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Crawl4AI Web Scraping loads about 2.5k tokens when it runs, and up to ~63k if it reads all its reference files. Until then it costs about 71 tokens; SKILL.md has 439 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~71

When it runs · the whole SKILL.md, loaded when a task matches

~2.5k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~63k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from smallnest/goclaw at commit e05c79d, republished under its MIT licence (© smallnest). 439 words, ~2,466 tokens.

name

crawl4ai

description

This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.

version

1.0.0

author

Claude

always

true

pip install crawl4ai
crawl4ai-setup

# Verify installation
crawl4ai-doctor

# Basic crawling - returns markdown
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache

# See more examples
crwl --example

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:500])

asyncio.run(main())

# Basic markdown
crwl https://docs.example.com -o markdown

# Filtered markdown (removes noise)
crwl https://docs.example.com -o markdown-fit

# With content filter
crwl https://docs.example.com -f filter_bm25.yml -o markdown-fit

# filter_bm25.yml (relevance-based)
type: "bm25"
query: "machine learning tutorials"
threshold: 1.0

from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

bm25_filter = BM25ContentFilter(user_query="machine learning", bm25_threshold=1.0)
md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)

config = CrawlerRunConfig(markdown_generator=md_generator)
result = await crawler.arun(url, config=config)

print(result.markdown.fit_markdown)  # Filtered
print(result.markdown.raw_markdown)  # Original

# Generate schema once (uses LLM)
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# Use schema for extraction (no LLM)
crwl https://shop.com -e extract_css.yml -s product_schema.json -o json

{
  "name": "products",
  "baseSelector": ".product-card",
  "fields": [
    {"name": "title", "selector": "h2", "type": "text"},
    {"name": "price", "selector": ".price", "type": "text"},
    {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
  ]
}

# extract_llm.yml
type: "llm"
provider: "openai/gpt-4o-mini"
instruction: "Extract product names and prices"
api_token: "your-token"

crwl https://shop.com -e extract_llm.yml -o json

crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000"

# crawler.yml
wait_for: "css:.ajax-content"
scan_full_page: true
page_timeout: 60000
delay_before_return_html: 2.0

for url in url1 url2 url3; do crwl "$url" -o markdown; done

urls = ["https://site1.com", "https://site2.com", "https://site3.com"]
results = await crawler.arun_many(urls, config=config)

# login_crawler.yml
session_id: "user_session"
js_code: |
  document.querySelector('#username').value = 'user';
  document.querySelector('#password').value = 'pass';
  document.querySelector('#submit').click();
wait_for: "css:.dashboard"

# Login
crwl https://site.com/login -C login_crawler.yml

# Access protected content (session reused)
crwl https://site.com/protected -c "session_id=user_session"

# browser.yml
headless: true
proxy_config:
  server: "http://proxy:8080"
  username: "user"
  password: "pass"
user_agent_mode: "random"

crwl https://example.com -B browser.yml

# Search Google and get results as JSON
python scripts/google_search.py "your search query" 20

# Example
python scripts/google_search.py "2026年Go语言展望" 20

crwl https://docs.example.com -o markdown > docs.md

# Generate schema once
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# Monitor (no LLM costs)
crwl https://shop.com -e extract_css.yml -s schema.json -o json

# Multiple sources with filtering
for url in news1.com news2.com news3.com; do
  crwl "https://$url" -f filter_bm25.yml -o markdown-fit
done

# First view content
crwl https://example.com -o markdown

# Then ask questions
crwl https://example.com -q "What are the main conclusions?"
crwl https://example.com -q "Summarize the key points"

crwl https://example.com -c "wait_for=css:.dynamic-content,page_timeout=60000"

crwl https://example.com -B browser.yml

# browser.yml
headless: false
viewport_width: 1920
viewport_height: 1080
user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

# Debug: see full output
crwl https://example.com -o all -v

# Try different wait strategy
crwl https://example.com -c "wait_for=js:document.querySelector('.content')!==null"

# Verify session
crwl https://site.com -c "session_id=test" -o all | grep -i session

© smallnest, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 15 other files (scripts, references) in internal/builtin_skills/crawl4ai-skill of smallnest/goclaw.

• SKILL.md
• .claude/settings.local.json
• README.md
• references/cli-guide.md
• references/complete-sdk-reference.md
• references/sdk-guide.md
• scripts/basic_crawler.py
• scripts/batch_crawler.py
• scripts/extraction_pipeline.py
• scripts/google_search.py
• tests/README.md
• tests/run_all_tests.py
• tests/test_advanced_patterns.py
• tests/test_basic_crawling.py
• tests/test_data_extraction.py
• tests/test_markdown_generation.py

Open the folder on GitHubat commit e05c79d

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in smallnest/goclaw, which our catalogue first saw on October 7, 2026.

• aiskillstore/marketplaceskills/smallnest/crawl4ai