Agent skill

Shopify Admin Agentic Crawler Access

by 40RTY-ai in 40RTY-ai/shopify-admin-skills

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.

MITAuto-check passedData & Analytics

Install Shopify Admin Agentic Crawler Access

skills CLI
$ npx skills add 40RTY-ai/shopify-admin-skills --skill shopify-admin-agentic-crawler-access -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install 40RTY-ai/shopify-admin-skills shopify-admin-agentic-crawler-access --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/40RTY-ai/shopify-admin-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/agentic/shopify-admin-agentic-crawler-access .claude/skills/shopify-admin-agentic-crawler-access && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
shopify-admin-agentic-crawler-access
GitHub stars
194
Token cost
~1.8k tokens
SKILL.md length
554 words
Files
1
Skills in repo
116
Repo updated
First seen
Licence
MIT

At a glance

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.

  • Works in 3 steps: OPERATION: themes — query → COMPUTE (no API): construct the new… → OPERATION: themeFilesUpsert — mutation
  • Tasks that involve Web scraping
  • SKILL.md covers Purpose, Prerequisites, Parameters and Safety, plus 6 more sections
  • Calls shopify

What it does

Shopify Admin Agentic Crawler Access is an agent skill from 40RTY-ai/shopify-admin-skills. Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.

Its SKILL.md is about 1.8k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts. Compatibility notes: Claude Code, Cursor, Codex, Gemini CLI

It sits in Data & Analytics, covering Web scraping and Technical SEO. It works with Shopify. The repository describes itself as: Community-maintained AI agent skills for operating Shopify stores — workflows, optimization, reports and more. The licence is MIT.

When your agent uses it

  • Tasks that involve Web scraping
  • Tasks that involve Technical SEO

Example prompts

  • “/shopify-admin-agentic-crawler-access”

Requirements

  • Compatibility (from SKILL.md): Claude Code, Cursor, Codex, Gemini CLI

Workflow steps

3 steps, taken from the first numbered list in SKILL.md.

  1. OPERATION: themes — query
  2. COMPUTE (no API): construct the new template. If no custom template exists, start from the Shopify default ({{ robots.default_groups }})…
  3. OPERATION: themeFilesUpsert — mutation

What it can do on your machine

Read from SKILL.md and the folder at commit 6765cb4. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • shopify

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Claude Code, Cursor, Codex, Gemini CLI

    From compatibility in the SKILL.md frontmatter.

Context cost

Shopify Admin Agentic Crawler Access loads about 1.8k tokens when it runs. Until then it costs about 58 tokens; SKILL.md has 554 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~58
When it runs · the whole SKILL.md, loaded when a task matches
~1.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from 40RTY-ai/shopify-admin-skills at commit 6765cb4, republished under its MIT licence (© 40RTY-ai). 554 words, ~1,795 tokens.

Download SKILL.mdSave it as .claude/skills/shopify-admin-agentic-crawler-access/SKILL.md (or your agent's skills folder).
name
shopify-admin-agentic-crawler-access
description
Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.
compatibility
Claude Code, Cursor, Codex, Gemini CLI
role
agentic
toolkit
shopify-admin, shopify-admin-execution
api_version
2025-01
graphql_operations
themes:query, themeFilesUpsert:mutation
status
stable
audit_signals
robots-ai-rules

Purpose

If your robots.txt doesn't address the AI crawlers by name — or worse, disallows them — assistants like ChatGPT, Claude, and Perplexity may never read your store, so you simply don't exist to AI shoppers. Shopify generates robots.txt from a theme template (templates/robots.txt.liquid); this skill reads that template and appends explicit allow rules for the major AI user-agents (with a clear, idempotent managed block) so foundational models and AI search are permitted to crawl your pages. Fixes the agentiq.report finding robots-ai-rules.

Prerequisites

  • Authenticated Shopify CLI session (shopify auth login --store <domain>)
  • Required API scopes: read_themes, write_themes

Parameters

All skills accept these universal parameters:

ParameterTypeRequiredDefaultDescription
storestringyes—Store domain (e.g., mystore.myshopify.com)
formatstringnohumanOutput format: human (default) or json
dry_runboolnotruePreview the new robots.txt.liquid without writing (defaults ON — this edits the live theme)

Skill-specific parameters:

ParameterTypeRequiredDefaultDescription
theme_idstringno—Theme GID to edit (defaults to the published MAIN theme)
agentsstringnoGPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,Claude-Web,PerplexityBot,Perplexity-User,Google-Extended,Amazonbot,Applebot-ExtendedComma list of AI user-agents to allow
modestringnoallowallow (add Allow rules) or audit (report current state only, no write)

Safety

⚠️ Step 2 (themeFilesUpsert) modifies a file in the LIVE published theme. A malformed robots.txt.liquid can de-index the store from ALL search engines. This skill writes only inside a clearly-delimited # BEGIN agentic-crawler-access … # END managed block (re-running replaces just that block, never your other rules), defaults dry_run: true, and recommends duplicating the theme first. Review the full proposed file before setting dry_run: false.

Workflow Steps

  1. OPERATION: themes — query Inputs: roles: [MAIN] (unless theme_id given), then theme.files(filenames: ["templates/robots.txt.liquid"]) Expected output: Current robots.txt.liquid content (or absence — Shopify uses a default if the template doesn't exist).

  2. COMPUTE (no API): construct the new template. If no custom template exists, start from the Shopify default ({{ robots.default_groups }}) and append the managed block with User-agent:/Allow: / stanzas for each agent in agents. If a managed block already exists, replace only it. Emit the full proposed file diff.

  3. OPERATION: themeFilesUpsert — mutation Inputs: themeId, files: [{ filename: "templates/robots.txt.liquid", body: { type: TEXT, value: <new content> } }] (skipped when dry_run or mode: audit). Expected output: Upserted theme file; collect userErrors.

Show full SKILL.md (198 more words)Show less

GraphQL Operations

graphql
# themes:query — validated against api_version 2025-01
query CrawlerThemeFile {
  themes(first: 1, roles: [MAIN]) {
    nodes {
      id
      name
      files(filenames: ["templates/robots.txt.liquid"]) {
        nodes {
          filename
          body { ... on OnlineStoreThemeFileBodyText { content } }
        }
      }
    }
  }
}
graphql
# themeFilesUpsert:mutation — validated against api_version 2025-01
mutation CrawlerAllowUpsert($themeId: ID!, $files: [OnlineStoreThemeFilesUpsertFileInput!]!) {
  themeFilesUpsert(themeId: $themeId, files: $files) {
    upsertedThemeFiles { filename }
    userErrors { filename code message }
  }
}

Managed block written into templates/robots.txt.liquid (after {{ robots.default_groups }}):

liquid
# BEGIN agentic-crawler-access (managed by shopify-admin-agentic-crawler-access)
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# ... one stanza per agent in `agents`
# END agentic-crawler-access

Session Tracking

Claude MUST emit the following output at each stage. This is mandatory.

On start, emit:

╔══════════════════════════════════════════════╗
║  SKILL: <skill name>                         ║
║  Store: <store domain>                       ║
║  Started: <YYYY-MM-DD HH:MM UTC>             ║
╚══════════════════════════════════════════════╝

After each step, emit:

[N/TOTAL] <QUERY|MUTATION>  <OperationName>
          → Params: <brief summary of key inputs>
          → Result: <count or outcome>

If dry_run: true, prefix every mutation step with [DRY RUN] and do not execute it.

On completion, emit:

For format: human (default):

══════════════════════════════════════════════
OUTCOME SUMMARY
  <Metric label>:   <value>
  Errors:           0
  Output:           <filename or "none">
══════════════════════════════════════════════

For format: json, emit:

json
{
  "skill": "<skill-slug>",
  "store": "<domain>",
  "started_at": "<ISO8601>",
  "completed_at": "<ISO8601>",
  "dry_run": false,
  "steps": [
    {
      "step": 1,
      "operation": "<OperationName>",
      "type": "query",
      "params_summary": "<string>",
      "result_summary": "<string>",
      "skipped": false
    }
  ],
  "outcome": {
    "metric_key": 0,
    "errors": 0,
    "output_file": null
  }
}

Output Format

human: the before/after robots.txt.liquid and the list of agents now allowed. json: { theme_id, agents_allowed:[...], wrote: true|false, errors }.

Error Handling

ErrorCauseRecovery
THROTTLEDAPI rate limitWait 2s, retry up to 3 times
ACCESS_DENIEDMissing write_themesAbort with a clear scope message; do not partially write
userErrors on upsertInvalid template bodyDo not write; surface the Liquid error for review

Best Practices

  • Duplicate the published theme before writing (Online Store → Themes → Duplicate) so you can roll back instantly.
  • Keep dry_run: true until you've read the entire proposed file — a broken robots.txt.liquid can de-index the whole store.
  • Use the managed-block markers; never hand-merge — re-running the skill then cleanly updates only its own block.
  • Allowing AI crawlers ≠ giving away content; it's permission to read public pages the same way Googlebot does. Pair with shopify-admin-agentic-llms-txt so crawlers also get a guided summary.

© 40RTY-ai, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/agentic/shopify-admin-agentic-crawler-access of 40RTY-ai/shopify-admin-skills.

Open the folder on GitHubat commit 6765cb4

Compare with similar skills

Shopify Admin Agentic Crawler Access next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Shopify Admin Agentic Crawler Access compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Shopify Admin Agentic Crawler Access this skill40RTY-ai/shopify-admin-skills194—~1.8kAutomated safety check: PassMIT
Docs Managementxiaoyuge886/aigc198—~7.8kAutomated safety check: NotesMIT
Apify Lead Scoring Enrichmentapify/awesome-skills265—~4.4kAutomated safety check: NotesApache-2.0
Agent Readiness Auditindranilbanerjee/digital-marketing-pro8591 repos~3.9kAutomated safety check: PassMIT
SEO Site Health Auditrampstackco/claude-skills941—~2.5kAutomated safety check: PassMIT
Robots Txtkostja94/marketing-skills1k—~1.9kAutomated safety check: PassMIT

Similar skills

  • Docs Management

    xiaoyuge886/aigc

    Single source of truth and librarian for ALL Claude official documentation.

    198 GitHub stars~7.8k tokensUpdated 2 mo ago
    Data & AnalyticsAuto-check: notes
  • Apify Lead Scoring Enrichment

    apify/awesome-skills

    Official

    Score and enrich a CSV of B2B leads using Apify Actors. An agent skill from apify/awesome-skills.

    265 GitHub stars~4.4k tokensUpdated 17 days ago
    Data & AnalyticsAuto-check: notes
  • Agent Readiness Audit

    indranilbanerjee/digital-marketing-pro

    Audit whether AI agents and AI crawlers can actually use a site — robots.txt rules per AI crawler token (OpenAI, Anthropic and Perplexity bots, Google-Extended, Applebot-Extended)…

    859 GitHub starsUsed in 1 repo~3.9k tokens
    Data & AnalyticsAuto-check passed
  • SEO Site Health Audit

    rampstackco/claude-skills

    Triage technical SEO findings from Ahrefs Site Audit (and similar crawlers) by SEO impact, not just severity.

    941 GitHub stars~2.5k tokensUpdated 2 days ago
    Data & AnalyticsAuto-check passed
  • Robots Txt

    kostja94/marketing-skills

    When the user wants to configure, audit, or optimize robots.txt.

    1k GitHub stars~1.9k tokensUpdated 3 days ago
    Data & AnalyticsAuto-check passed
  • LLM Crawler Access Check

    davepoon/buildwithclaude

    Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot.

    3.6k GitHub stars~1.5k tokensUpdated today
    Data & AnalyticsAuto-check passed

More from 40RTY-ai/shopify-admin-skills

All 116 skills in this repo
  • Shopify Admin Abandoned Cart Recovery

    40RTY-ai/shopify-admin-skills

    Query checkouts abandoned in the last N days, generate unique discount codes per customer, and tag them for re-engagement.

    194 GitHub stars~1.9k tokensUpdated 1 mo ago
    Auto-check passed
  • Shopify Admin Address Correction

    40RTY-ai/shopify-admin-skills

    Update the shipping address on an unfulfilled order before it ships.

    194 GitHub stars~1.5k tokensUpdated 1 mo ago
    Auto-check passed
  • Rewrite thin product descriptions into structured, fact-rich copy (materials, fit, use-cases, the words shoppers actually type) so AI agents have something concrete to quote and match.

    194 GitHub stars~1.6k tokensUpdated 1 mo ago
    Auto-check passed
  • Shopify Admin Agentic Image Alt Text

    40RTY-ai/shopify-admin-skills

    Generate and set descriptive alt text on product images so AI agents (which can't 'see' pixels) can understand and recommend what each product looks like.

    194 GitHub stars~1.6k tokensUpdated 1 mo ago
    Auto-check passed
  • Shopify Admin Agentic LLMs Txt

    40RTY-ai/shopify-admin-skills

    Generate and publish an /llms.txt guide (brand summary, flagship products, key policies, contact) via a theme template so AI assistants get a curated, machine-readable map of the store.

    194 GitHub stars~1.8k tokensUpdated 1 mo ago
    Auto-check passed
  • Shopify Admin Agentic Metafields Setup

    40RTY-ai/shopify-admin-skills

    Define and populate agentic-commerce metafields (material, attributes, key features, specs, sizing) so AI agents can filter and match products to specific shopper requirements.

    194 GitHub stars~1.9k tokensUpdated 1 mo ago
    Auto-check passed

Works with

Questions about Shopify Admin Agentic Crawler Access

What does Shopify Admin Agentic Crawler Access do?

Edit the theme's robots.txt.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog. Shopify Admin Agentic Crawler Access is an agent skill from 40RTY-ai/shopify-admin-skills.liquid to explicitly allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, Amazonbot) so AI assistants are permitted to read the catalog.

When should I use Shopify Admin Agentic Crawler Access?

Shopify Admin Agentic Crawler Access fits situations like: tasks that involve Web scraping; tasks that involve Technical SEO.

How do I install Shopify Admin Agentic Crawler Access in Claude Code?

Run `npx skills add 40RTY-ai/shopify-admin-skills --skill shopify-admin-agentic-crawler-access -a claude-code`. Or copy the skill folder (skills/agentic/shopify-admin-agentic-crawler-access in 40RTY-ai/shopify-admin-skills) into .claude/skills/shopify-admin-agentic-crawler-access in your project. Claude Code loads it when a task matches its description.

How do I install Shopify Admin Agentic Crawler Access in Codex?

Run `npx skills add 40RTY-ai/shopify-admin-skills --skill shopify-admin-agentic-crawler-access -a codex`. Or copy the skill folder (skills/agentic/shopify-admin-agentic-crawler-access in 40RTY-ai/shopify-admin-skills) into .agents/skills/shopify-admin-agentic-crawler-access in your project. Codex loads it when a task matches its description.

Can I use Shopify Admin Agentic Crawler Access in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add 40RTY-ai/shopify-admin-skills --skill shopify-admin-agentic-crawler-access -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/shopify-admin-agentic-crawler-access, .gemini/skills/shopify-admin-agentic-crawler-access, .github/skills/shopify-admin-agentic-crawler-access and .opencode/skills/shopify-admin-agentic-crawler-access in your project.

What does Shopify Admin Agentic Crawler Access need to run?

Going by SKILL.md and its folder, Shopify Admin Agentic Crawler Access needs the command-line tools its instructions call (shopify). Compatibility (from SKILL.md): Claude Code, Cursor, Codex, Gemini CLI.

Does Shopify Admin Agentic Crawler Access access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Shopify Admin Agentic Crawler Access safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Shopify Admin Agentic Crawler Access use?

Shopify Admin Agentic Crawler Access is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Shopify Admin Agentic Crawler Access use?

About 1.8k tokens (SKILL.md is roughly 7.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Shopify Admin Agentic Crawler Access?

Skills that share tags, products or a category with Shopify Admin Agentic Crawler Access: Docs Management (xiaoyuge886/aigc, 198 stars), Apify Lead Scoring Enrichment (apify/awesome-skills, 265 stars), Agent Readiness Audit (indranilbanerjee/digital-marketing-pro, 859 stars) and SEO Site Health Audit (rampstackco/claude-skills, 941 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Shopify Admin Agentic Crawler Access?

40RTY-ai (a GitHub organization) maintains it in 40RTY-ai/shopify-admin-skills, which has 194 GitHub stars. The repository holds 116 skills in this directory. The repository was last updated on August 14, 2026.

Source: 40RTY-ai/shopify-admin-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.