Crawl4AI SEO Site Crawler

What it does

When your agent uses it

• Running a full SEO inventory of a site's pages
• Finding orphan pages and weakly linked sections
• Auditing breadcrumbs, menus and navigation consistency
• Comparing a shortlist of landing pages or competitor pages

Example prompts

• “Crawl example.com and give me an inventory of titles, H1s and canonicals.”
• “Find orphan pages and weakly linked hubs on our blog.”
• “Compare our pricing landing page with the three competitor pages I listed, on on-page signals.”

Requirements

• Python 3 with uv
• crawl4ai and playwright installed, as checked by doctor.py

Workflow steps

3 steps, taken from the first numbered list in SKILL.md.

1. 1Проверь окружение
2. 2Определи цель исследования
3. 3Собери launch params

What it can do on your machine

Read from SKILL.md and the folder at commit 8bbeead. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 6 files in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • python3
  • uv

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use uv, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Crawl4AI SEO Site Crawler loads about 1.7k tokens when it runs, and up to ~4.1k if it reads all its reference files. Until then it costs about 173 tokens; SKILL.md has 516 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~173

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~4.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from artwist-polyakov/polyakov-claude-skills at commit 8bbeead, republished under its MIT licence (© artwist-polyakov). 516 words, ~1,740 tokens.

name

crawl4ai-seo

description

SEO-краулер сайтов на базе Crawl4AI. Полный аудит страниц: title, meta, H1, canonical, breadcrumbs, навигация, внутренние ссылки. Инвентаризация сайта, навигационный аудит, сравнение лендингов, анализ конкурентов. Работает для Google и Яндекс SEO (Cyrillic URL, коммерческие факторы, региональность). Связка с yandex-search-api, yandex-metrika, yandex-webmaster, scrapedo-web-scraper. Triggers: crawl4ai, seo crawl, site audit, page inventory, site inventory, on-page audit, internal links, internal linking audit, navigation audit, landing comparison, competitor analysis, competitor pages, orphan pages, technical seo, аудит сайта, краулер, перелинковка, навигационный аудит.

# 1. Создать job
python3 scripts/init_job.py --domain https://example.com --project client-a --label full-audit

# 2. Собрать seed URL из sitemap/robots
python3 scripts/seed_urls.py --job-id <job_id>

# 3. Запустить crawl
uv run --script scripts/crawl_batch.py --job-id <job_id>

# 4. Построить навигационный отчёт
python3 scripts/build_navigation_report.py \
  --seed-job-id <seed_job_id> \
  --crawl-job-id <crawl_job_id> \
  --report-dir reports/<project>/<label>

# 5. Сравнить страницы
python3 scripts/compare_pages.py --job-id <job_id>

python3 scripts/seed_urls.py --domain https://example.com --project client-a --label quick-check
uv run --script scripts/crawl_batch.py --job-id <job_id>

cache/jobs/<job_id>/
  launch_params.json      # raw params этого запуска
  resolved_config.json    # effective config после merge defaults + params
  manifest.json           # operational state и artifact paths
  seed.json               # seed URLs и метаданные discovery
  pages.ndjson            # одна запись на страницу
  links.ndjson            # одна запись на link edge
  summary.json            # компактная сводка
  markdown/               # сохранённый markdown контент

© artwist-polyakov, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 12 other files (scripts, references) in plugins/crawl4ai-seo/skills/crawl4ai-seo of artwist-polyakov/polyakov-claude-skills.

• SKILL.md
• .gitignore
• cache/.gitkeep
• config/README.md
• config/defaults.example.json
• references/OUTPUT_SCHEMA.md
• references/WORKFLOWS.md
• scripts/build_navigation_report.py
• scripts/common.py
• scripts/compare_pages.py
• scripts/crawl_batch.py
• scripts/doctor.py
• scripts/seed_urls.py

Open the folder on GitHubat commit 8bbeead