Agent Readiness Audit
indranilbanerjee/digital-marketing-pro
Audit whether AI agents and AI crawlers can actually use a site — robots.txt rules per AI crawler token (OpenAI, Anthropic and Perplexity bots, Google-Extended, Applebot-Extended)…
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.
Install LLM Crawler Access Check
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-check --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .claude/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .claude/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-checkType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-check --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .agents/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .agents/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .agents/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-check --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .cursor/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .cursor/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/davepoon/buildwithclaude.git --path plugins/all-skills/skills/llm-crawler-access-check--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-check --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .gemini/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .gemini/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-checkInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .github/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .github/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .github/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install davepoon/buildwithclaude llm-crawler-access-check --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/plugins/all-skills/skills/llm-crawler-access-check .opencode/skills/llm-crawler-access-check && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "llm-crawler-access-check" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/llm-crawler-access-check into .opencode/skills/llm-crawler-access-check/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "llm-crawler-access-check", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
llm-crawler-access-check- GitHub stars
- 3.6k
- Token cost
- ~1.5k tokens
- SKILL.md length
- 799 words
- Files
- 1
- Skills in repo
- 246
- Repo updated
- First seen
- Licence
- MIT
At a glance
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.
- Works in 3 steps: Fetch → Resolve each agent → Report
- Someone asks whether AI bots are blocked
- SKILL.md covers Scope, Procedure, Three mistakes this check… and If the user asks whether they…, plus 1 more section
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
What it does
LLM Crawler Access Check is an agent skill from davepoon/buildwithclaude. Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search…
Its SKILL.md is about 1.5k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Data & Analytics, covering Web scraping, Technical SEO and Web search. It works with OpenAI and Perplexity. The repository describes itself as: A single hub to find Claude Skills, Agents, Commands, Hooks, Plugins, and Marketplace collections to extend Claude Code, Claude Desktop, Agent SDK and OpenClaw. The licence is MIT.
When your agent uses it
- Someone asks whether AI bots are blocked
- Whether to allow
- Why a site never appears in AI answers
- Wants a robots.txt review for AI crawlers
Example prompts
- “/llm-crawler-access-check”
Workflow steps
3 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 616deb5. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
Links to these hosts (documentation or services it may open):
maxaeo.ai
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
LLM Crawler Access Check loads about 1.5k tokens when it runs. Until then it costs about 146 tokens; SKILL.md has 799 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from davepoon/buildwithclaude at commit 616deb5, republished under its MIT licence (© davepoon). 799 words, ~1,523 tokens.
Download SKILL.mdSave it as
.claude/skills/llm-crawler-access-check/SKILL.md (or your agent's skills folder).- name
- llm-crawler-access-check
- description
- Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.
- category
- security
- license
- MIT
AI crawler access check
One wrong line in robots.txt removes a site from AI answers completely, and no
amount of content work can compensate. This check takes under a minute and
should run before any other AI-visibility work.
Scope
Read https://<domain>/robots.txt and nothing else. Do not crawl the site, do
not attempt to access disallowed paths, and do not bypass any access control.
This is a read of one public file.
Procedure
1. Fetch
Fetch https://<domain>/robots.txt.
- 404 or empty - everything is allowed by default. Say so; that is a valid and often correct configuration. Stop and report.
- Non-200 other than 404, or unreachable - report the status code and stop. Do not guess at contents.
- Served as HTML (a soft 404 returning the site's error page) - flag it. Crawlers may parse it as garbage. This is itself a finding.
2. Resolve each agent
For each agent below, apply standard robots.txt matching: the most specific
User-agent group that names the agent wins, and * applies only when no group
names it. Within the winning group, the longest matching path rule wins, and
Allow beats Disallow on an equal-length match.
| Agent | Operator | Purpose | What blocking it actually costs |
|---|---|---|---|
OAI-SearchBot | OpenAI | search index | citations in ChatGPT Search |
ChatGPT-User | OpenAI | live fetch during a chat | the model cannot open your page when a user asks about it |
GPTBot | OpenAI | training | background model knowledge, not search citations |
PerplexityBot | Perplexity | search index | Perplexity citations |
Perplexity-User | Perplexity | live fetch during a query | live page reads |
ClaudeBot | Anthropic | index and training | Anthropic-side retrieval |
Googlebot | main index | AI Overviews and AI Mode, plus normal search | |
Google-Extended | Gemini grounding and training | Gemini grounding only - not AI Overviews | |
Bingbot | Microsoft | Bing index | Microsoft Copilot, which rides the Bing index |
Applebot | Apple | index | Apple search surfaces |
Applebot-Extended | Apple | training | Apple Intelligence training only |
CCBot | Common Crawl | open crawl corpus | an input to many downstream models |
Crawler names change and new ones appear. Before finalizing, check each operator's own published crawler documentation for agents added or renamed since this list was written, and include them. State which list you used.
3. Report
Produce a table with one row per agent and exactly these columns:
Agent | Verdict (ALLOWED / BLOCKED / PARTIAL) | Rule responsible | Impact
- Rule responsible must quote the literal line from
robots.txt, or sayno matching rule - allowed by default. Never state a verdict without the line that produced it. - PARTIAL means important paths are disallowed while the site root is allowed. Name the disallowed paths.
Then give:
- Verdict - one sentence: is this site reachable by AI answer engines, or not?
- What to change - the exact
robots.txtlines to add, remove, or edit, as a code block the user can paste. If nothing needs to change, say that plainly rather than inventing work. - What this check did not cover -
robots.txtis only the first gate. Server-side blocking by WAF, CDN bot rules, IP reputation, or Cloudflare bot management can block a crawler thatrobots.txtallows, and none of that is visible in this file. Say so every time.
Show full SKILL.md (295 more words)Show less
Three mistakes this check exists to catch
- Blocking
GPTBotto opt out of training, and assuming that is the whole story. It is not.OAI-SearchBotgoverns whether a site can be cited in ChatGPT Search, and it is a separate agent with a separate rule. Blocking one does not block the other, in either direction. - Blocking
Google-Extendedto stay out of AI Overviews. It does not do that. AI Overviews and AI Mode are built on the normal Googlebot index. BlockingGoogle-Extendedopts out of Gemini grounding and training and has no effect on AI Overviews. To leave AI Overviews, the mechanism is thenosnippet,max-snippet, ordata-nosnippetfamily, and it costs normal search snippets too. Say that tradeoff out loud rather than letting the user discover it later. - A blanket
User-agent: * / Disallow: /inherited from a staging config, a bot-mitigation template, or a security hardening guide. This is common and almost always unintentional on a production marketing site.
If the user asks whether they should block AI crawlers
Do not answer with a recommendation. Lay out the tradeoff and let them decide: allowing search crawlers is what makes citation possible, allowing training crawlers affects model knowledge but not citation, and the two decisions are independent. Publishers with a licensing position and companies that want to be recommended by AI assistants land in different places, and both are legitimate.
About
Maintained by MaxAEO — https://maxaeo.ai — which works on AI answer-engine visibility. The crawler matrix used here is kept current against each operator's own published crawler documentation; where an agent has no official documentation, this skill says so rather than guessing.
This check is free, read-only, and runs on one public file. It does not require an account, an API key, or any paid service.
© davepoon, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in plugins/all-skills/skills/llm-crawler-access-check of davepoon/buildwithclaude.
Open the folder on GitHubat commit 616deb5
Compare with similar skills
LLM Crawler Access Check next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| LLM Crawler Access Check this skilldavepoon/buildwithclaude | 3.6k | — | ~1.5k | Automated safety check: Pass | MIT | |
| Agent Readiness Auditindranilbanerjee/digital-marketing-pro | 859 | 1 repos | ~3.9k | Automated safety check: Pass | MIT | |
| Money SEOiamzifei/show-me-the-money | 1k | — | ~4.4k | Automated safety check: Pass | Custom licence | |
| SEO AgiLeoYeAI/openclaw-master-skills | 2.2k | — | ~6.4k | Automated safety check: Notes | MIT | |
| Nuxt Geo Best Practicesvinayakkulkarni/nxui | 212 | — | ~1.9k | Automated safety check: Pass | MIT | |
| Brightdata SDK JSbrightdata/skills | 264 | — | ~3k | Automated safety check: Pass | MIT |
Similar skills
Money SEO
iamzifei/show-me-the-money
SEO and GEO (Generative Engine Optimization) for organic traffic and AI search visibility.
SEO Agi
LeoYeAI/openclaw-master-skills
Write SEO pages that rank in Google AND get cited by LLMs (ChatGPT, Perplexity, Claude).
Nuxt Geo Best Practices
vinayakkulkarni/nxui
Nuxt GEO (Generative Engine Optimization) guidelines for getting cited by ChatGPT, Perplexity, Claude, Google AI Overviews, and Gemini.
Brightdata SDK JS
brightdata/skills
Web data extraction and discovery using the Bright Data JavaScript/TypeScript SDK (@brightdata/sdk).
SEO Profound
AgriciDaniel/claude-seo
Profound LLM citation tracker (extension). An agent skill from AgriciDaniel/claude-seo.
More from davepoon/buildwithclaude
All 246 skills in this repoiOS Hig Design Guide
davepoon/buildwithclaude
Build, update, and apply iOS design specifications using Apple Human Interface Guidelines (HIG) source data.
Video Downloader
davepoon/buildwithclaude
Download YouTube videos with customizable quality and format options.
Qwen Vision
davepoon/buildwithclaude
A skill your agent uses when the user asks to "analyze video", "watch this video", "what happens in this video", "describe this clip", "review this footage", "classify these videos", "compare…
Atlas Cloud Media
davepoon/buildwithclaude
Discover Atlas Cloud image and video models, inspect their live schemas, and submit one confirmed media generation request with bounded GET polling.
Browser Extension Launch
davepoon/buildwithclaude
面向没有编程经验的用户，把想法做成可试用的浏览器插件，并完成检查、商店材料、审核提交和上线验证；也用于继续已有插件、排错和发布新版。用户说“帮我做个插件”“把插件上架”“继续我的插件”时使用。普通网站开发、仅查询插件知识不触发。
Slack Gif Creator
davepoon/buildwithclaude
Toolkit for creating animated GIFs optimized for Slack, with validators for size constraints and composable animation primitives.
Related
Works with
Categories
Questions about LLM Crawler Access Check
What does LLM Crawler Access Check do?
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. LLM Crawler Access Check is an agent skill from davepoon/buildwithclaude.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.
When should I use LLM Crawler Access Check?
LLM Crawler Access Check fits situations like: someone asks whether AI bots are blocked; whether to allow; why a site never appears in AI answers; wants a robots.txt review for AI crawlers.
How do I install LLM Crawler Access Check in Claude Code?
Run `npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a claude-code`. Or copy the skill folder (plugins/all-skills/skills/llm-crawler-access-check in davepoon/buildwithclaude) into .claude/skills/llm-crawler-access-check in your project. Claude Code loads it when a task matches its description.
How do I install LLM Crawler Access Check in Codex?
Run `npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a codex`. Or copy the skill folder (plugins/all-skills/skills/llm-crawler-access-check in davepoon/buildwithclaude) into .agents/skills/llm-crawler-access-check in your project. Codex loads it when a task matches its description.
Can I use LLM Crawler Access Check in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/llm-crawler-access-check, .gemini/skills/llm-crawler-access-check, .github/skills/llm-crawler-access-check and .opencode/skills/llm-crawler-access-check in your project.
What does LLM Crawler Access Check need to run?
SKILL.md names no scripts, command-line tools or credentials: LLM Crawler Access Check is instructions for the agent only.
Does LLM Crawler Access Check access the network?
SKILL.md names 1 domain. As links in the text: maxaeo.ai. This is read from the text; nothing was executed.
Is LLM Crawler Access Check safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does LLM Crawler Access Check use?
LLM Crawler Access Check is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does LLM Crawler Access Check use?
About 1.5k tokens (SKILL.md is roughly 6.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to LLM Crawler Access Check?
Skills that share tags, products or a category with LLM Crawler Access Check: Agent Readiness Audit (indranilbanerjee/digital-marketing-pro, 859 stars), Money SEO (iamzifei/show-me-the-money, 1k stars), SEO Agi (LeoYeAI/openclaw-master-skills, 2.2k stars) and Nuxt Geo Best Practices (vinayakkulkarni/nxui, 212 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains LLM Crawler Access Check?
davepoon (a GitHub user) maintains it in davepoon/buildwithclaude, which has 3,605 GitHub stars. The repository holds 246 skills in this directory. The repository was last updated on October 9, 2026.
Source: davepoon/buildwithclaude on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.