Crawlberg

<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:2400d592929aaf450317f228f695638b76d0f7987f5996106372e06b9abce70d
Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c
Schema-Version: v1
-->

Crawlberg

Crawlberg is a Rust-native web crawler and scraper. It fetches static HTML with reqwest, falls back to headless Chrome when a page needs JS or trips a WAF, and converts every result to clean Markdown via the built-in HTML→Markdown engine.

Use this skill when the user wants to:

• Scrape a single URL to Markdown plus structured metadata.
• Crawl a site following links bounded by depth, page count, and concurrency.
• Enumerate URLs from sitemaps without paying for rendering.
• Drive a real browser (click, type, scroll) and capture the resulting DOM.
• Run the same operations from another agent harness via MCP tools.

Installation

The plugin shells out to a crawlberg binary on PATH. Install one of:

brew install xberg-io/tap/crawlberg
# or run without a persistent install (the CLI proxy package self-installs the binary):
npx @xberg-io/crawlberg-cli --help
uvx --from crawlberg-cli crawlberg --help
# or build from source:
cargo install crawlberg-cli --features all

The serve and mcp subcommands are gated behind non-default cargo features (api and mcp). The Homebrew tap is built with all features, so both subcommands work out of the box. A from-source build must pass --features mcp (and --features api for serve), or --features all, to include them.

Verify:

crawlberg --version

Headless fallback needs Chrome/Chromium reachable locally (chromiumoxide launches it on demand). Skip the install if you only plan to use --browser-mode never.

Command map

crawlberg scrape <url>          # single page → JSON or Markdown
crawlberg crawl <url...>        # follow links, BFS, depth-bounded
crawlberg map <url>             # enumerate URLs via sitemaps + link extraction
crawlberg interact <url>        # browser actions: click, type, scroll
crawlberg batch-scrape <url...> # scrape many URLs concurrently
crawlberg batch-crawl <url...>  # crawl many seed URLs concurrently
crawlberg download <url>        # download a document, report file metadata
crawlberg citations <input>     # markdown links → numbered citations (text or @file.md)
crawlberg version               # print the crawlberg version as JSON
crawlberg mcp                   # MCP server (stdio) — auto-registered (`mcp` feature)
crawlberg serve                 # REST API server (`api` feature)

crawl also handles batching implicitly: pass multiple seed URLs and it fans out via batch_crawl internally. The explicit batch-scrape and batch-crawl subcommands expose the same concurrency for many independent URLs.

Per-subcommand flags:

Shared flags

The --config flag accepts the full CrawlConfig schema. Anything you set explicitly on the CLI overrides the corresponding JSON field.

These shared flags apply to the crawl/scrape-family subcommands. --format, --browser-mode, --browser-endpoint, and --config cover scrape, crawl, map, interact, batch-scrape, and batch-crawl; download takes --timeout, --browser-mode, --browser-endpoint, --max-size, and --config (no --format). --respect-robots-txt applies to scrape, crawl, map, batch-scrape, and batch-crawl. citations and version take no shared flags.

Scrape a single page

crawlberg scrape https://example.com --format markdown

JSON output (default) carries the rendered Markdown, page metadata (PageMetadata), links by category, images, feeds, JSON-LD blocks, and HTTP response metadata. Use Markdown output when piping into a file the user will read.

See the scraping-html-to-markdown skill for the full flag surface.

Crawl a site

crawlberg crawl https://example.com \
  --depth 3 --max-pages 200 --concurrent 8 --rate-limit 250 \
  --stay-on-domain --respect-robots-txt --format markdown

Crawling is BFS by default, bounded by --depth, --max-pages, and --concurrent. Per-domain politeness is enforced by --rate-limit (milliseconds between requests to the same origin).

See the crawling-a-site skill for the recommended defaults and the full flag surface.

Map URLs

crawlberg map https://example.com --limit 500 --search docs --format markdown

map reads sitemap.xml (and nested sitemaps), then falls back to link extraction from the seed page. It does not render pages — use it to plan a crawl or to feed URLs into another tool.

Browser interaction

crawlberg interact https://example.com \
  --actions '[{"type":"click","selector":"#load-more"},
              {"type":"wait","milliseconds":500},
              {"type":"scrape"}]'

Action types are click, type, press, scroll, wait, screenshot, executeJs, and scrape (to wait for an element, use wait with a selector field). The result wraps the final HTML under interaction.final_html. See the automating-the-browser skill for the full action schema and limits.