Docs Management
xiaoyuge886/aigc
Single source of truth and librarian for ALL Claude official documentation.
When the user wants to configure, audit, or optimize robots.txt.
Install Robots Txt
$ npx skills add kostja94/marketing-skills --skill robots-txt -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install kostja94/marketing-skills robots-txt --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/seo/technical/robots .claude/skills/robots-txt && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .claude/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robotsType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add kostja94/marketing-skills --skill robots-txt -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install kostja94/marketing-skills robots-txt --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/seo/technical/robots .agents/skills/robots-txt && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .agents/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add kostja94/marketing-skills --skill robots-txt -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install kostja94/marketing-skills robots-txt --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/seo/technical/robots .cursor/skills/robots-txt && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .cursor/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/kostja94/marketing-skills.git --path skills/seo/technical/robots--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add kostja94/marketing-skills --skill robots-txt -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install kostja94/marketing-skills robots-txt --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/seo/technical/robots .gemini/skills/robots-txt && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .gemini/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install kostja94/marketing-skills robots-txtInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add kostja94/marketing-skills --skill robots-txt -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/seo/technical/robots .github/skills/robots-txt && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .github/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add kostja94/marketing-skills --skill robots-txt -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install kostja94/marketing-skills robots-txt --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/kostja94/marketing-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/seo/technical/robots .opencode/skills/robots-txt && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/kostja94/marketing-skills/tree/main/skills/seo/technical/robots into .opencode/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
robots-txt- GitHub stars
- 1k
- Token cost
- ~1.9k tokens
- SKILL.md length
- 795 words
- Files
- 1
- Skills in repo
- 102
- Repo updated
- First seen
- Licence
- MIT
At a glance
When the user wants to configure, audit, or optimize robots.txt.
- Works in 3 steps: Site URL: Base domain (e.g.,… → Indexing scope: Full site, partial, or… → AI crawler strategy: Allow…
- Wants to configure
- SKILL.md covers Scope (Technical SEO), Initial Assessment, Best Practices and Output Format, plus 1 more section
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
What it does
Robots Txt is an agent skill from kostja94/marketing-skills. When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
Its SKILL.md is about 1.9k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Data & Analytics, covering Technical SEO and Web scraping. The repository describes itself as: Agent Skills for Marketing — SEO, Social, Influencer & More. 160+ open-source skills for SEO, content, 40+ page types, paid ads, channels, and strategies. Add project context… The licence is MIT.
When your agent uses it
- Wants to configure
- Optimize robots.txt
- The user mentions robots.txt
- Crawl directives
Example prompts
- “robots.txt,”
- “crawler rules,”
- “block crawlers,”
- “/robots-txt”
Workflow steps
3 steps, taken from the first numbered list in SKILL.md.
- Site URL: Base domain (e.g., https://example.com)
- Indexing scope: Full site, partial, or specific paths to exclude
- AI crawler strategy: Allow search/indexing vs. block training data crawlers
What it can do on your machine
Read from SKILL.md and the folder at commit 8dd89c5. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
Links to these hosts (documentation or services it may open):
developers.google.com
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Robots Txt loads about 1.9k tokens when it runs. Until then it costs about 92 tokens; SKILL.md has 795 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from kostja94/marketing-skills at commit 8dd89c5, republished under its MIT licence (© kostja94). 795 words, ~1,934 tokens.
Download SKILL.mdSave it as
.claude/skills/robots-txt/SKILL.md (or your agent's skills folder).- name
- robots-txt
- description
- When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
- metadata.version
- 1.2.0
SEO Technical: robots.txt
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
- Crawler access: Path-level crawl control; AI crawler allow/block strategy
- Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions.
Initial Assessment
Project context: Read root contextus.md when present and load only the modules relevant to this task. Without Contextus, use available project material or user-provided facts and ask for missing information; do not create a parallel context system.
Identify:
- Site URL: Base domain (e.g.,
https://example.com) - Indexing scope: Full site, partial, or specific paths to exclude
- AI crawler strategy: Allow search/indexing vs. block training data crawlers
Best Practices
Purpose and Limitations
| Point | Note |
|---|---|
| Purpose | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) |
| Advisory | Rules are advisory; malicious crawlers may ignore |
| Public | robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing |
Crawl vs Index vs Link Equity (Quick Reference)
| Tool | Controls | Prevents indexing? |
|---|---|---|
| robots.txt | Crawl (path-level) | No—blocked URLs may still appear in SERP |
| noindex (meta / X-Robots-Tag) | Index (page-level) | Yes. See indexing |
| nofollow | Link equity only | No—does not control indexing |
When to Use robots.txt vs noindex
| Use | Tool | Example |
|---|---|---|
| Path-level (whole directory) | robots.txt | Disallow: /admin/, Disallow: /api/, Disallow: /staging/ |
| Page-level (specific pages) | noindex meta / X-Robots-Tag | Login, signup, thank-you, 404, legal. See indexing for full list |
| Critical | Do NOT block in robots.txt | Pages that use noindex—crawlers must access the page to read the directive |
Paths to block in robots.txt: /admin/, /api/, /staging/, temp files. Paths to use noindex (allow crawl): /login/, /signup/, /thank-you/, etc.—see indexing.
Location and Format
| Item | Requirement |
|---|---|
| Path | Site root: https://example.com/robots.txt |
| Encoding | UTF-8 plain text |
| Standard | RFC 9309 (Robots Exclusion Protocol) |
Core Directives
| Directive | Purpose | Example |
|---|---|---|
User-agent: | Target crawler | User-agent: Googlebot, User-agent: * |
Disallow: | Block path prefix | Disallow: /admin/ |
Allow: | Allow path (can override Disallow) | Allow: /public/ |
Sitemap: | Declare sitemap absolute URL | Sitemap: https://example.com/sitemap.xml |
Clean-param: | Strip query params (Yandex) | See below |
Path Format Rules (Critical)
Different directives use different path formats — a common source of errors:
| Directive | Format | Correct | Wrong |
|---|---|---|---|
| Disallow / Allow | Root-relative path only (starts with /) | Disallow: /admin/ | Disallow: https://example.com/admin/ |
| Sitemap | Absolute URL only | Sitemap: https://example.com/sitemap.xml | Sitemap: /sitemap.xml |
Wildcards: * matches any character sequence (Disallow: /tmp/*). $ marks exact URL ending (Allow: /news/.html$).
Priority: More specific paths take precedence. Allow: /shop/shoes/ overrides Disallow: /shop/. Path matching is case-sensitive: Disallow: /PDF/ does not match /pdf/.
Show full SKILL.md (341 more words)Show less
Critical: Do Not Block
| Do not block | Reason |
|---|---|
| CSS, JS, images | Google needs them to render pages; blocking breaks indexing |
/_next/ (Next.js) | Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See indexing |
| Pages that use noindex | Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that |
Only block: paths that don't need crawling: /admin/, /api/, /staging/, temp files.
AI Crawler Strategy
robots.txt is effective for all measured AI crawlers. Set rules per user-agent; check each vendor's docs for current tokens.
| User-agent | Purpose | Typical | Notes |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | Allow | Respects robots.txt |
| GPTBot | OpenAI training | Disallow | Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed |
| ChatGPT-User | User-initiated browsing | N/A | No longer respects robots.txt (Dec 2025); use server-side controls instead |
| Claude-SearchBot | Claude search | Allow | Respects robots.txt |
| Claude-User | Anthropic user-initiated browsing | Allow | Respects robots.txt (unlike ChatGPT-User) |
| ClaudeBot | Anthropic training | Disallow | Respects robots.txt |
Deprecated:
anthropic-ai— retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References toanthropic-aiin robots.txt have no effect. | PerplexityBot | Perplexity search | Allow | Respects robots.txt | | Google-Extended | Gemini training | Disallow | Respects robots.txt | | CCBot | Common Crawl (LLM training) | Disallow | Respects robots.txt | | Bytespider | ByteDance | Disallow | Respects robots.txt | | Meta-ExternalAgent | Meta | Disallow | Respects robots.txt | | AppleBot | Apple (Siri, Spotlight); renders JS | Allow for indexing | Respects robots.txt |
Allow vs Disallow: Allow search/indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot); Disallow training-only bots (GPTBot, ClaudeBot, CCBot) if you don't want content used for model training.
Important — ChatGPT-User exemption: As of December 2025, ChatGPT-User no longer respects robots.txt directives. OpenAI considers it a proxy for human-initiated browsing. If you need to block it, use server-side controls (WAF rules, IP rate-limiting), not robots.txt. See site-crawlability for AI crawler optimization (SSR, URL management).
Clean-param (Yandex)
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&ref&fbclid&gclidOutput Format
- Current state (if auditing)
- Recommended robots.txt (full file)
- Compliance checklist
- References: Google robots.txt
Related Skills
- indexing: Full noindex page-type list; when to use noindex vs robots.txt; GSC indexing diagnosis
- page-metadata: Meta robots (noindex, nofollow) implementation
- xml-sitemap: Sitemap URL to reference in robots.txt
- site-crawlability: Broader crawl and structure guidance; AI crawler optimization
- rendering-strategies: SSR, SSG, CSR; content in initial HTML for crawlers
© kostja94, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in skills/seo/technical/robots of kostja94/marketing-skills.
Open the folder on GitHubat commit 8dd89c5
Compare with similar skills
Robots Txt next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Robots Txt this skillkostja94/marketing-skills | 1k | — | ~1.9k | Automated safety check: Pass | MIT | |
| Docs Managementxiaoyuge886/aigc | 198 | — | ~7.8k | Automated safety check: Notes | MIT | |
| SEO Site Health Auditrampstackco/claude-skills | 945 | — | ~2.5k | Automated safety check: Pass | MIT | |
| Shopify Admin Agentic Crawler Access40RTY-ai/shopify-admin-skills | 194 | — | ~1.8k | Automated safety check: Pass | MIT | |
| SEO Sitemapseranking/seo-skills | 161 | — | ~2.3k | Automated safety check: Pass | MIT | |
| LLM Crawler Access Checkdavepoon/buildwithclaude | 3.6k | — | ~1.5k | Automated safety check: Pass | MIT |
Similar skills
SEO Site Health Audit
rampstackco/claude-skills
Triage technical SEO findings from Ahrefs Site Audit (and similar crawlers) by SEO impact, not just severity.
Shopify Admin Agentic Crawler Access
40RTY-ai/shopify-admin-skills
Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.
SEO Sitemap
seranking/seo-skills
Pull a domain's XML sitemap (and sitemap-of-sitemaps), then compare against the most recent SE Ranking website audit.
LLM Crawler Access Check
davepoon/buildwithclaude
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.
Ax
yusukebe/ax
Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML.
More from kostja94/marketing-skills
All 102 skills in this repoGrokipedia Recommendations
kostja94/marketing-skills
When the user wants to add recommendations, links, or content to Grokipedia.
AI Traffic Tracking
kostja94/marketing-skills
When the user wants to track AI search traffic in GA4 or GSC.
Analytics Tracking
kostja94/marketing-skills
When the user wants to set up, audit, or optimize analytics tracking (GA4, events, conversions).
Brand Visual Generator
kostja94/marketing-skills
When the user wants to define, audit, or apply visual identity (typography, colors, spacing, design tokens, frontend aesthetics).
Canonical Tag
kostja94/marketing-skills
When the user wants to configure canonical URLs, fix duplicate content, or consolidate URL signals.
Content Strategy
kostja94/marketing-skills
When the user wants to plan content for SEO, create content calendar, or build topic clusters.
Related
Categories
Questions about Robots Txt
What does Robots Txt do?
When the user wants to configure, audit, or optimize robots.txt. Robots Txt is an agent skill from kostja94/marketing-skills.txt.
When should I use Robots Txt?
Robots Txt fits situations like: wants to configure; optimize robots.txt; the user mentions robots.txt; crawl directives.
How do I install Robots Txt in Claude Code?
Run `npx skills add kostja94/marketing-skills --skill robots-txt -a claude-code`. Or copy the skill folder (skills/seo/technical/robots in kostja94/marketing-skills) into .claude/skills/robots-txt in your project. Claude Code loads it when a task matches its description.
How do I install Robots Txt in Codex?
Run `npx skills add kostja94/marketing-skills --skill robots-txt -a codex`. Or copy the skill folder (skills/seo/technical/robots in kostja94/marketing-skills) into .agents/skills/robots-txt in your project. Codex loads it when a task matches its description.
Can I use Robots Txt in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add kostja94/marketing-skills --skill robots-txt -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/robots-txt, .gemini/skills/robots-txt, .github/skills/robots-txt and .opencode/skills/robots-txt in your project.
What does Robots Txt need to run?
SKILL.md names no scripts, command-line tools or credentials: Robots Txt is instructions for the agent only.
Does Robots Txt access the network?
SKILL.md names 1 domain. As links in the text: developers.google.com. This is read from the text; nothing was executed.
Is Robots Txt safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Robots Txt use?
Robots Txt is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Robots Txt use?
About 1.9k tokens (SKILL.md is roughly 7.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Robots Txt?
Skills that share tags, products or a category with Robots Txt: Docs Management (xiaoyuge886/aigc, 198 stars), SEO Site Health Audit (rampstackco/claude-skills, 945 stars), Shopify Admin Agentic Crawler Access (40RTY-ai/shopify-admin-skills, 194 stars) and SEO Sitemap (seranking/seo-skills, 161 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Robots Txt?
kostja94 (a GitHub user) maintains it in kostja94/marketing-skills, which has 1,025 GitHub stars. The repository holds 102 skills in this directory. The repository was last updated on October 6, 2026.
Source: kostja94/marketing-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.