Dingtalk Message
agentscope-ai/ReMe
钉钉消息发送技能。支持企业内部机器人（批量单聊/群聊）和 Webhook 自定义机器人两种接入方式，支持多机器人管理，支持文本、Markdown、链接、ActionCard、FeedCard等多种消息类型。
A skill your agent uses when the user wants to follow links across a domain and capture every reachable page as Markdown.
Install Crawling A Site
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-site --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .claude/skills/crawling-a-site && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .claude/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-siteType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-site --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .agents/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .agents/skills/crawling-a-site && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .agents/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-site --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .cursor/skills/crawling-a-site && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .cursor/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/hashgraph-online/awesome-codex-plugins.git --path plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-site --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .gemini/skills/crawling-a-site && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .gemini/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-siteInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .github/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .github/skills/crawling-a-site && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .github/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install hashgraph-online/awesome-codex-plugins crawling-a-site --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/hashgraph-online/awesome-codex-plugins.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site .opencode/skills/crawling-a-site && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "crawling-a-site" agent skill from https://github.com/hashgraph-online/awesome-codex-plugins/tree/main/plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site into .opencode/skills/crawling-a-site/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "crawling-a-site", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
crawling-a-site- GitHub stars
- 1.3k
- Token cost
- ~1.4k tokens
- SKILL.md length
- 468 words
- Files
- 1
- Skills in repo
- 714
- Repo updated
- First seen
- Licence
- Apache-2.0
At a glance
A skill your agent uses when the user wants to follow links across a domain and capture every reachable page as Markdown.
- The user wants to follow links across a domain and capture every reachable page as Markdown
- SKILL.md covers Quick recipe, Flag surface, When to pick which flags and Output, plus 3 more sections
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
- Tasks that involve Rate limiting
What it does
Crawling A Site is an agent skill from hashgraph-online/awesome-codex-plugins. Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers crawlberg crawl with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Backend & APIs, covering Rate limiting and Markdown. The repository describes itself as: A curated list of awesome OpenAI Codex / ChatGPT plugins, skills, and resources. The 1 Codex Marketplace. See live plugins at: https://hol.org/plugins/best-codex-plugins. The licence is Apache-2.0.
When your agent uses it
- The user wants to follow links across a domain and capture every reachable page as Markdown
- Tasks that involve Rate limiting
- Tasks that involve Markdown
Example prompts
- “/crawling-a-site”
What it can do on your machine
Read from SKILL.md and the folder at commit 9e7b281. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md (its code samples are bash).
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Crawling A Site loads about 1.4k tokens when it runs. Until then it costs about 60 tokens; SKILL.md has 468 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from hashgraph-online/awesome-codex-plugins at commit 9e7b281, republished under its Apache-2.0 licence (© hashgraph-online). 468 words, ~1,353 tokens.
Download SKILL.mdSave it as
.claude/skills/crawling-a-site/SKILL.md (or your agent's skills folder).- name
- crawling-a-site
- description
- Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14
Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c
Schema-Version: v1
-->Crawling a site
Reach for crawlberg crawl when one URL is not enough — the user wants
the docs site, the blog, the marketing pages, or the whole domain.
Quick recipe
crawlberg crawl https://example.com \
--depth 3 \
--max-pages 200 \
--concurrent 8 \
--rate-limit 250 \
--stay-on-domain \
--respect-robots-txt \
--format markdownDefaults you should usually override:
--depth 2is shallow — set it explicitly.--max-pagesis unbounded by default; cap it for any unknown site.--concurrent 10is aggressive for small hosts; drop to 4-8 for third-party sites.
Flag surface
| Flag | Default | Purpose |
|---|---|---|
--depth, -d | 2 | Maximum hop count from the seed URL. |
--max-pages, -n | — | Hard cap on pages fetched. Set this on any unknown site. |
--concurrent, -c | 10 | Parallel in-flight requests. |
--rate-limit | 200 | Milliseconds between requests to the same origin. |
--stay-on-domain | off | Skip links that leave the seed domain. |
--respect-robots-txt | off | Honour robots.txt. Pass it for any third-party host. |
--proxy | — | HTTP, HTTPS, or SOCKS5 proxy URL. |
--user-agent | — | Override the request UA. Be honest. |
--timeout | 30000 | Per-request timeout in ms. |
--browser-mode | auto | auto, always, never — see the headless-fallback skill. |
--browser-endpoint | — | External CDP ws:// URL. |
--format | json | json or markdown. |
--config | — | Inline JSON or @file.json for the full CrawlConfig. |
Multiple seed URLs are accepted positionally — the engine fans out with
batch_crawl and aggregates results.
When to pick which flags
Docs sites you own
crawlberg crawl https://docs.example.com \
--depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \
--stay-on-domain --format markdown > docs.mdHigher concurrency and lower rate limits are fine on infrastructure you control.
Third-party sites
crawlberg crawl https://blog.unknown.example \
--depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \
--stay-on-domain --respect-robots-txt --format markdownStay shallow, cap pages, throttle hard, obey robots.
Multi-seed batch
crawlberg crawl \
https://example.com/blog \
https://example.com/docs \
https://example.com/pricing \
--depth 2 --max-pages 100 --stay-on-domain --format jsonJSON output for batch is an array of { seed_url, result } entries — each
result is a full crawl payload or { error: ... }.
Output
Markdown mode
---
URL: https://example.com/page-one
---
# Page One
… markdown content …
---
URL: https://example.com/page-two
---
…JSON mode
Top-level CrawlResult with pages: [...]. Each page carries the rendered
Markdown plus metadata, links, images, JSON-LD, and HTTP response info. Read
result.pages[i].markdown.content for the Markdown string.
Show full SKILL.md (194 more words)Show less
Politeness checklist
- Pass
--respect-robots-txton every third-party crawl. - Cap
--max-pages— a runaway BFS can issue tens of thousands of requests. - Bump
--rate-limitfor hosts that show signs of stress (5xx, slowdowns). - Identify yourself via
--user-agent crawlberg (contact@example.com).
Common pitfalls
- No pages returned. The seed page may be JS-only — the engine falls
back to headless automatically in
--browser-mode auto, butnevermode will silently produce an empty crawl. Re-run with--browser-mode alwaysor check the headless-fallback skill. - Crawl leaves the domain. Pass
--stay-on-domain. Combine withallow_subdomains: truein--configJSON to include subdomains. - Slow crawl. The default rate limit is 200 ms per origin — multiple
seed URLs on the same host still share the bucket. Spread seeds across
hosts or raise
--concurrentfor unrelated origins. - Memory growth. Each page carries full Markdown plus structured data.
Stream JSON output to a file rather than holding it in memory; set
--max-pagesaggressively if downstream cannot keep up.
When to reach for map instead
If the user only needs the list of URLs (sitemap analysis, link planning,
seeding another tool), use crawlberg map <url> — it skips rendering and
returns a flat MapResult with hundreds of URLs in seconds.
© hashgraph-online, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site of hashgraph-online/awesome-codex-plugins.
Open the folder on GitHubat commit 9e7b281
Compare with similar skills
Crawling A Site next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Crawling A Site this skillhashgraph-online/awesome-codex-plugins | 1.3k | — | ~1.4k | Automated safety check: Pass | Apache-2.0 | |
| Dingtalk Messageagentscope-ai/ReMe | 3.6k | — | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Add Hosted Keysimstudioai/sim | 30k | — | ~3.4k | Automated safety check: Pass | Apache-2.0 | |
| Upstash Ratelimit TSupstash/ratelimit-js | 2k | — | ~313 | Automated safety check: Pass | MIT | |
| Repo2skillzhangyanxs/repo2skill | 246 | — | ~3.6k | Automated safety check: Pass | None | |
| Better Auth Security Best PracticesEpicenterHQ/epicenter | 4.8k | — | ~896 | Automated safety check: Pass | Custom licence |
Similar skills
Add Hosted Key
simstudioai/sim
Add hosted API key support to a tool so Sim provides the key (metered and billed to the workspace) when a user has not brought their own.
- Official
Upstash Ratelimit TS
upstash/ratelimit-js
Lightweight guidance for using the Redis Rate Limit TypeScript SDK, including setup steps, basic usage, and pointers to advanced algorithm, features, pricing, and traffic‑protection docs.
Repo2skill
zhangyanxs/repo2skill
Convert GitHub/GitLab/Gitee repositories into comprehensive OpenCode Skills using embedded LLM calls with multiple mirrors and rate limit handling
Better Auth Security Best Practices
EpicenterHQ/epicenter
Better Auth security hardening: rate limits, secrets, CSRF, trusted origins, cookies, sessions, OAuth tokens, and audit logging.
Dload Fetch Tool
php-internal/dload
Get a CLI tool — native binary or PHAR — from a GitHub release into a project folder with dload (vendor/bin/dload).
More from hashgraph-online/awesome-codex-plugins
All 714 skills in this repoAnime Reaction Gif
hashgraph-online/awesome-codex-plugins
Create original anime-style reaction stickers as looping GIFs and MP4 previews, using generated character pose sheets and timed key poses.
Calibredb
hashgraph-online/awesome-codex-plugins
Manage and query Calibre libraries with the calibredb CLI (local paths or Calibre Content server URLs).
Rust API Test Harness
hashgraph-online/awesome-codex-plugins
A skill your agent uses when adding, changing, testing, or debugging Rust HTTP APIs and services, especially when Codex needs black-box integration tests, random-port app startup, real database test…
Art
hashgraph-online/awesome-codex-plugins
Make a studio's game look like something at build time — a cover from a real frame of the game (free), painted covers, backdrops, textures and character plates from image models through the…
Calle
hashgraph-online/awesome-codex-plugins
Use CALL-E from Codex through the calle CLI. An agent skill from hashgraph-online/awesome-codex-plugins.
Game Balance Economy
hashgraph-online/awesome-codex-plugins
Balance game difficulty, resources, rewards, probability, progression, economies, and dominant strategies.
Related
Categories
Questions about Crawling A Site
What does Crawling A Site do?
A skill your agent uses when the user wants to follow links across a domain and capture every reachable page as Markdown. Crawling A Site is an agent skill from hashgraph-online/awesome-codex-plugins. Use when the user wants to follow links across a domain and capture every reachable page as Markdown.
When should I use Crawling A Site?
Crawling A Site fits situations like: the user wants to follow links across a domain and capture every reachable page as Markdown; tasks that involve Rate limiting; tasks that involve Markdown.
How do I install Crawling A Site in Claude Code?
Run `npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a claude-code`. Or copy the skill folder (plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site in hashgraph-online/awesome-codex-plugins) into .claude/skills/crawling-a-site in your project. Claude Code loads it when a task matches its description.
How do I install Crawling A Site in Codex?
Run `npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a codex`. Or copy the skill folder (plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/crawling-a-site in hashgraph-online/awesome-codex-plugins) into .agents/skills/crawling-a-site in your project. Codex loads it when a task matches its description.
Can I use Crawling A Site in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add hashgraph-online/awesome-codex-plugins --skill crawling-a-site -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/crawling-a-site, .gemini/skills/crawling-a-site, .github/skills/crawling-a-site and .opencode/skills/crawling-a-site in your project.
What does Crawling A Site need to run?
SKILL.md names no scripts, command-line tools or credentials: Crawling A Site is instructions for the agent only.
Does Crawling A Site access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Crawling A Site safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Crawling A Site use?
Crawling A Site is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Crawling A Site use?
About 1.4k tokens (SKILL.md is roughly 5.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Crawling A Site?
Skills that share tags, products or a category with Crawling A Site: Dingtalk Message (agentscope-ai/ReMe, 3.6k stars), Add Hosted Key (simstudioai/sim, 30k stars), Upstash Ratelimit TS (upstash/ratelimit-js, 2k stars) and Repo2skill (zhangyanxs/repo2skill, 246 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Crawling A Site?
hashgraph-online (a GitHub organization) maintains it in hashgraph-online/awesome-codex-plugins, which has 1,255 GitHub stars. The repository holds 714 skills in this directory. The repository was last updated on October 9, 2026.
Source: hashgraph-online/awesome-codex-plugins on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.