Specification Discovery Crawler

What it does

When your agent uses it

• Finding the whitepaper and docs for a project starting from its website
• Collecting specification URLs before an audit
• Mapping which technical documents a project publishes

Example prompts

• “Find all specification and whitepaper links starting from https://example.com/project.”
• “Crawl this protocol's docs site and list its architecture documents.”
• “Collect technical spec URLs for this project and write them to a JSON file.”

Requirements

• The MCP fetch tool
• Browser automation tools as a fallback
• An OUTPUT_FILE environment variable for the result path
• Pre-approved tools (allowed-tools): mcp__fetch__fetch, browser_navigate, browser_scroll, browser_click, browser_view, write

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

1. 1Initial Fetch: Use mcpfetchfetch with the provided url to retrieve the page content as Markdown.
2. 2Link Extraction: Parse the returned Markdown to identify links that likely lead to technical documentation. Keywords to look for include…
3. 3Recursive Fetch: For each discovered link, use mcpfetchfetch to retrieve content and extract further specification links. Limit depth to…
4. 4Fallback to Browser: If mcpfetchfetch fails (403, empty response, JavaScript-required content), fall back to browser tools…
5. 5Collect the URLs of any pages or PDF documents that appear to be technical specifications.
6. 6Consolidate all found specification URLs into a final list, deduplicating entries.

What it can do on your machine

Read from SKILL.md and the folder at commit d173893. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • mcp__fetch__fetch
  • browser_navigate
  • browser_scroll
  • browser_click
  • browser_view
  • write

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are json).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Specification Discovery Crawler loads about 669 tokens when it runs. Until then it costs about 19 tokens; SKILL.md has 255 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~19

When it runs · the whole SKILL.md, loaded when a task matches

~669

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NyxFoundation/speca at commit d173893, republished under its MIT licence (© NyxFoundation). 255 words, ~669 tokens.

name

spec-discovery

description

Crawl and discover specification documents from a given URL.

allowed-tools

mcp__fetch__fetch, browser_navigate, browser_scroll, browser_click, browser_view, write

context

fork

{
  "url": "https://example.com/project"
}

{
  "start_url": "https://example.com/project",
  "found_specs": [
    {
      "url": "https://example.com/project/docs/specification.md",
      "title": "Project Specification"
    },
    {
      "url": "https://example.com/project/whitepaper.pdf",
      "title": "Project Whitepaper"
    }
  ],
  "metadata": {
    "timestamp": "...",
    "urls_visited": []
  }
}

© NyxFoundation, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in .claude/skills/spec-discovery of NyxFoundation/speca.

Open the folder on GitHubat commit d173893