Site Architecture
AvdLee/RocketSimApp
When the user wants to plan, map, or restructure their website's page hierarchy, navigation, URL structure, or internal linking.
A skill your agent uses when applies to any public website. An agent skill from thedaviddias/Front-End-Checklist.
Install Robots Txt
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install thedaviddias/Front-End-Checklist robots-txt --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/robots-txt .claude/skills/robots-txt && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .claude/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txtType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install thedaviddias/Front-End-Checklist robots-txt --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/robots-txt .agents/skills/robots-txt && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .agents/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install thedaviddias/Front-End-Checklist robots-txt --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/robots-txt .cursor/skills/robots-txt && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .cursor/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/thedaviddias/Front-End-Checklist.git --path skills/robots-txt--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install thedaviddias/Front-End-Checklist robots-txt --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/robots-txt .gemini/skills/robots-txt && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .gemini/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install thedaviddias/Front-End-Checklist robots-txtInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/robots-txt .github/skills/robots-txt && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .github/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install thedaviddias/Front-End-Checklist robots-txt --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/thedaviddias/Front-End-Checklist.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/robots-txt .opencode/skills/robots-txt && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "robots-txt" agent skill from https://github.com/thedaviddias/Front-End-Checklist/tree/main/skills/robots-txt into .opencode/skills/robots-txt/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "robots-txt", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
robots-txt- GitHub stars
- 74k
- Token cost
- ~483 tokens
- SKILL.md length
- 216 words
- Files
- 2 (incl. references)
- Skills in repo
- 390
- Repo updated
- First seen
- Licence
- MIT
At a glance
A skill your agent uses when applies to any public website. An agent skill from thedaviddias/Front-End-Checklist.
- Applies to any public website
- SKILL.md covers Quick Reference, Check, Fix and Explain, plus 1 more section
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
- Auditing technical SEO foundations
What it does
Robots Txt is an agent skill from thedaviddias/Front-End-Checklist. Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.
Its SKILL.md is about 480 tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files, including reference files (for example `references/rule.md`).
It sits in Marketing & SEO, covering Technical SEO and Test coverage. The repository describes itself as: 🗂 The essential checklist for modern web development, for humans and AI agents. The licence is MIT.
When your agent uses it
- Applies to any public website
- Auditing technical SEO foundations
- Diagnosing crawl coverage gaps
Example prompts
- “/robots-txt”
What it can do on your machine
Read from SKILL.md and the folder at commit e8d14d0. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
Links to these hosts (documentation or services it may open):
frontendchecklist.io
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Robots Txt loads about 483 tokens when it runs, and up to ~1.3k if it reads all its reference files. Until then it costs about 32 tokens; SKILL.md has 216 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from thedaviddias/Front-End-Checklist at commit e8d14d0, republished under its MIT licence (© thedaviddias). 216 words, ~483 tokens.
Download SKILL.mdSave it as
.claude/skills/robots-txt/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.- name
- robots-txt
- description
- Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.
- metadata.category
- seo
- metadata.priority
- high
- metadata.difficulty
- beginner
- metadata.estimatedTime
- 10
- metadata.source
- frontendchecklist.io
- metadata.url
- https://frontendchecklist.io/rules/seo/robots-txt
Publish a robots.txt file
robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.
Quick Reference
- Serve a valid
robots.txtat/robots.txton the production domain, returning HTTP 200 - Include a
Sitemap:directive pointing to your XML sitemap - Never disallow crawling of CSS/JS assets that render your pages
- Avoid blocking all crawlers with
Disallow: /on a live site
Check
Fetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives.
Fix
Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering.
Explain
Explain how robots.txt controls crawler access, why an accidental Disallow: / can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.
Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
For full implementation details, code examples, and framework-specific guidance,
see references/rule.md.
Rule page: https://frontendchecklist.io/rules/seo/robots-txt
© thedaviddias, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 1 other file (references) in skills/robots-txt of thedaviddias/Front-End-Checklist.
- SKILL.md
- references/rule.md
Open the folder on GitHubat commit e8d14d0
Compare with similar skills
Robots Txt next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Robots Txt this skillthedaviddias/Front-End-Checklist | 74k | — | ~483 | Automated safety check: Pass | MIT | |
| Site ArchitectureAvdLee/RocketSimApp | 806 | 11 repos | ~3.3k | Automated safety check: Pass | Custom licence | |
| Hreflang and International SEOAgriciDaniel/claude-seo | 19k | 5 repos | ~3.4k | Automated safety check: Pass | MIT | |
| Google SEO APIsAgriciDaniel/claude-seo | 19k | 1 repos | ~4.2k | Automated safety check: Pass | MIT | |
| GEO-First SEO Audit Toolzubair-trabzada/geo-seo-claude | 11k | — | ~2.8k | Automated safety check: Notes | MIT | |
| SEO Ops Structural Checklisttigerless-labs/seo-ops | 700 | — | ~3.3k | Automated safety check: Notes | None |
Similar skills
Hreflang and International SEO
AgriciDaniel/claude-seo
Audits, validates and generates hreflang tags for multi-language and multi-region sites in HTML, HTTP headers or XML sitemaps, flagging common code and return-tag mistakes.
Google SEO APIs
AgriciDaniel/claude-seo
Pulls real Google data for SEO work: Search Console, PageSpeed Insights, CrUX field data, the Indexing API and GA4 organic traffic, through /seo google commands.
GEO-First SEO Audit Tool
zubair-trabzada/geo-seo-claude
Audits a website for AI search visibility across ChatGPT, Claude, Perplexity and Google AI Overviews while checking traditional SEO, schema and E-E-A-T content quality.
SEO Ops Structural Checklist
tigerless-labs/seo-ops
Checks a site against a deterministic set of structural SEO and GEO checks, either running a crawler-eye report against a URL or reviewing code against the same checklist.
SEO
Bhanunamikaze/Agentic-SEO-Skill
Deterministic LLM-first SEO audits for websites, blog posts, and GitHub repositories.
More from thedaviddias/Front-End-Checklist
All 390 skills in this repoContent Dates Audit
thedaviddias/Front-End-Checklist
Audits article and blog pages for visible publish dates, Article JSON-LD with datePublished and dateModified, and Open Graph time tags, then fixes what is missing.
FAQPage Schema Markup
thedaviddias/Front-End-Checklist
Adds, checks and fixes FAQPage JSON-LD on pages with visible question-and-answer sections so it matches what readers see and can qualify for rich results.
Favicon Audit and Setup
thedaviddias/Front-End-Checklist
Checks that a site's favicon is linked, reachable and large enough for Google search results, and sets up ICO, SVG and Apple touch icon files when they are missing.
Content Freshness Signals
thedaviddias/Front-End-Checklist
Audits article pages for freshness signals, covering the Last-Modified header, Article JSON-LD dateModified and a visible last-updated date, and fixes mismatches.
Geo Meta Tags Audit
thedaviddias/Front-End-Checklist
Audits and fixes geo.region, geo.placename and geo.position meta tags on regional pages, noting where they help (Bing) and where they do not (Google).
Improve a Front-End Checklist Rule
thedaviddias/Front-End-Checklist
Scores and rewrites a Front-End Checklist rule MDX file against a scored quality rubric, replacing generic stub prompts with specific, actionable ones.
Related
Categories
Questions about Robots Txt
What does Robots Txt do?
A skill your agent uses when applies to any public website. An agent skill from thedaviddias/Front-End-Checklist. Robots Txt is an agent skill from thedaviddias/Front-End-Checklist. Use when applies to any public website.
When should I use Robots Txt?
Robots Txt fits situations like: applies to any public website; auditing technical SEO foundations; diagnosing crawl coverage gaps.
How do I install Robots Txt in Claude Code?
Run `npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a claude-code`. Or copy the skill folder (skills/robots-txt in thedaviddias/Front-End-Checklist) into .claude/skills/robots-txt in your project. Claude Code loads it when a task matches its description.
How do I install Robots Txt in Codex?
Run `npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a codex`. Or copy the skill folder (skills/robots-txt in thedaviddias/Front-End-Checklist) into .agents/skills/robots-txt in your project. Codex loads it when a task matches its description.
Can I use Robots Txt in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add thedaviddias/Front-End-Checklist --skill robots-txt -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/robots-txt, .gemini/skills/robots-txt, .github/skills/robots-txt and .opencode/skills/robots-txt in your project.
What does Robots Txt need to run?
SKILL.md names no scripts, command-line tools or credentials: Robots Txt is instructions for the agent only.
Does Robots Txt access the network?
SKILL.md names 1 domain. As links in the text: frontendchecklist.io. This is read from the text; nothing was executed.
Is Robots Txt safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Robots Txt use?
Robots Txt is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Robots Txt use?
About 483 tokens (SKILL.md is roughly 1.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 768 tokens, read only when the agent opens those files.
What are the alternatives to Robots Txt?
Skills that share tags, products or a category with Robots Txt: Site Architecture (AvdLee/RocketSimApp, 806 stars), Hreflang and International SEO (AgriciDaniel/claude-seo, 19k stars), Google SEO APIs (AgriciDaniel/claude-seo, 19k stars) and GEO-First SEO Audit Tool (zubair-trabzada/geo-seo-claude, 11k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Robots Txt?
thedaviddias (a GitHub user) maintains it in thedaviddias/Front-End-Checklist, which has 74,421 GitHub stars. The repository holds 390 skills in this directory. The repository was last updated on October 6, 2026.
Source: thedaviddias/Front-End-Checklist on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.