Agent skill

Extract Webpage Data

by gooseworks-ai in gooseworks-ai/goose-skills

Extract structured data from web pages using AI. An agent skill from gooseworks-ai/goose-skills.

MITAuto-check passedData & Analytics

Install Extract Webpage Data

skills CLI
$ npx skills add gooseworks-ai/goose-skills --skill extract-webpage-data -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install gooseworks-ai/goose-skills extract-webpage-data --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/gooseworks-ai/goose-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/research-tools/capabilities/extract-webpage-data .claude/skills/extract-webpage-data && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
extract-webpage-data
GitHub stars
1.2k
Used in
1 other repo
Token cost
~1.9k tokens
SKILL.md length
515 words
Files
2
Skills in repo
273
Repo updated
First seen
Licence
MIT

At a glance

Extract structured data from web pages using AI. An agent skill from gooseworks-ai/goose-skills.

  • Tasks that involve Web scraping
  • SKILL.md covers Setup, When to Use, How It Works and Usage, plus 5 more sections
  • Calls curl, python3 and npx; reaches api.gooseworks.ai; needs GOOSEWORKS_API_KEY

What it does

Extract Webpage Data is an agent skill from gooseworks-ai/goose-skills. Extract structured data from web pages using AI

Its SKILL.md is about 1.9k tokens, which your agent loads only when the skill is triggered. The skill folder holds 1 other file (for example `skill.meta.json`).

It sits in Data & Analytics, covering Web scraping. The repository describes itself as: Library of Growth & GTM skills + data APIs for Claude Code, Codex, Cursor to run ads, social, content, lead gen, seo and data scraping. The licence is MIT.

When your agent uses it

  • Tasks that involve Web scraping

Example prompts

  • “/extract-webpage-data”

Requirements

  • Python 3
  • Node.js
  • A credential in GOOSEWORKS_API_KEY

What it can do on your machine

Read from SKILL.md and the folder at commit c650c6d. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • curl
    • python3
    • npx

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • api.gooseworks.ai

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names these keys or tokens, usually read from environment variables:

    • GOOSEWORKS_API_KEY

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Extract Webpage Data loads about 1.9k tokens when it runs. Until then it costs about 17 tokens; SKILL.md has 515 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~17
When it runs · the whole SKILL.md, loaded when a task matches
~1.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from gooseworks-ai/goose-skills at commit c650c6d, republished under its MIT licence (© gooseworks-ai). 515 words, ~1,857 tokens.

Download SKILL.mdSave it as .claude/skills/extract-webpage-data/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
extract-webpage-data
description
Extract structured data from web pages using AI
source
orthogonal

Extract Webpage Data

Setup

Read your credentials from ~/.gooseworks/credentials.json:

bash
export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")

If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login

All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"

Extract structured data from any web page using AI. Turn messy HTML into clean, organized data.

When to Use

  • User wants to extract specific data from a website
  • User asks to scrape information from a page
  • User needs structured data from unstructured content
  • User wants to pull product info, contact details, etc.
  • Converting web content to usable data

How It Works

Uses Olostep, Scrapegraph, or Riveter APIs for AI-powered data extraction.

Usage

Simple Scrape with Olostep
bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/scrapes","body":{"url_to_scrape":"https://example.com/products"}}'
AI-Powered Extraction with Scrapegraph
bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"scrapegraph","path":"/v1/smartscraper","body":{"website_url":"https://example.com/team","user_prompt":"Extract all team members with their names, titles, and LinkedIn URLs"}}'
Schema-Based Extraction with Riveter
bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"riveter","path":"/v1/scrape","body":{"url":"https://example.com","schema":{"name":"string","price":"number","description":"string"}}}'
Get AI Answer from Web
bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/answers","body":{"task":"Find the pricing for Notion Teams plan from their website"}}'
Crawl Multiple Pages
bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/crawls","body":{"start_url":"https://example.com","max_pages":10}}'

Parameters

Olostep Scrape
  • url_to_scrape (required) - URL to scrape
  • formats - Output formats (markdown, html, text)
Scrapegraph
  • website_url (required) - URL to scrape
  • user_prompt (required) - Natural language description of what to extract
Riveter
  • url (required) - URL to scrape
  • schema - JSON schema defining the data structure to extract
Olostep Answer
  • task (required) - Natural language task/question

Response

Olostep Response

Returns a scrape object:

  • id (string) - Scrape ID (e.g., scrape_z926lxxon3)
  • result.markdown_content (string|null) - Page content as markdown
  • result.html_content (string|null) - Raw HTML (if requested via formats)
  • result.text_content (string|null) - Plain text (if requested)
  • result.markdown_hosted_url (string|null) - S3 URL for large content
  • result.links_on_page (array) - Links found on the page
  • result.screenshot_hosted_url (string|null) - Screenshot URL (if requested)
  • result.page_metadata (object) - status_code of the page
  • credits_consumed (integer) - Credits used for this scrape

Async crawls: POST /v1/crawls returns an id. Poll with GET /v1/crawls/{id} until complete.

Scrapegraph Response

Returns structured extraction result:

  • request_id (string) - Unique request identifier
  • status (string) - completed or pending
  • result (object) - AI-extracted data matching your prompt (dynamic keys)
  • error (string) - Empty on success, error message on failure

Note: For large pages, the POST may return status: "pending". Poll with GET /v1/smartscraper/{request_id} until status is completed.

Show full SKILL.md (210 more words)Show less
Riveter Response

Returns scrape result:

  • request_status (string) - success or error
  • message (string) - Human-readable status
  • text (string) - Extracted page text content
  • url (string) - URL that was scraped
  • status_code (integer) - HTTP status of the page
  • run_key (string) - Unique run identifier
  • base_url_for_links (string) - Base URL for resolving relative links
  • riveter_app_link (string) - Link to view run in Riveter dashboard
  • credit_used (integer) - Credits consumed

Examples

User: "Get all the product names and prices from this page"

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"scrapegraph","path":"/v1/smartscraper","body":{"website_url":"https://example.com/products","user_prompt":"Extract all products with name, price, and description"}}'

User: "Scrape the team page and get everyone's info"

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"scrapegraph","path":"/v1/smartscraper","body":{"website_url":"https://example.com/about/team","user_prompt":"Extract team members: name, role, bio, photo URL, LinkedIn"}}'

User: "What are Stripe's API pricing details?"

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"olostep","path":"/v1/answers","body":{"task":"Find Stripe API pricing breakdown from stripe.com/pricing"}}'

User: "Get all blog post titles and dates from this blog"

bash
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
  -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"api":"riveter","path":"/v1/scrape","body":{"url":"https://blog.example.com","schema":{"posts":[{"title":"string","date":"string","url":"string"}]}}}'

Error Handling

  • 504 - Olostep timeout on slow pages — retry or try a simpler URL
  • 400 - Missing required parameters (url_to_scrape for Olostep, website_url + user_prompt for Scrapegraph, url for Riveter)
  • Scrapegraph returns error field in response body — check it even on 200 status
  • Riveter returns request_status: "error" with details in message
  • Some sites block automated scraping — try a different API if one fails

Tips

  • Scrapegraph is best for natural language extraction
  • Riveter is best when you know the exact schema you want
  • Olostep is great for general scraping and AI answers
  • For dynamic sites (JavaScript-heavy), these tools handle rendering
  • Be specific in your prompts for better extraction results
  • Some sites may block automated access

© gooseworks-ai, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in skills/research-tools/capabilities/extract-webpage-data of gooseworks-ai/goose-skills.

  • SKILL.md
  • skill.meta.json

Open the folder on GitHubat commit c650c6d

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in gooseworks-ai/goose-skills, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Extract Webpage Data next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Extract Webpage Data compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Extract Webpage Data this skillgooseworks-ai/goose-skills1.2k1 repos~1.9kAutomated safety check: PassMIT
Tmuxtrpc-group/trpc-agent-go1.9k23 repos~868Automated safety check: PassApache-2.0
Ketch1broseidon/ketch7021 repos~3.9kAutomated safety check: PassMIT
Boss Zhipin Scrapereatmoreduck/boss-zhipin-scraper1.5k—~2.6kAutomated safety check: PassMIT
Crawl4AI Web Scrapingsmallnest/goclaw5991 repos~2.5kAutomated safety check: PassMIT
Axyusukebe/ax7191 repos~918Automated safety check: PassMIT

Similar skills

  • Tmux

    trpc-group/trpc-agent-go

    Remote-control tmux sessions for interactive CLIs by sending keystrokes and scraping pane output.

    1.9k GitHub starsUsed in 23 repos~868 tokens
    Data & AnalyticsAuto-check passed
  • Ketch

    1broseidon/ketch

    Research skill for ketch — a fast stateless CLI for web search, OSS code search, curated library docs, page scraping, and site crawling; an optional MCP server exists for operators who want it, but…

    702 GitHub starsUsed in 1 repo~3.9k tokens
    Data & AnalyticsAuto-check passed
  • Boss Zhipin Scraper

    eatmoreduck/boss-zhipin-scraper

    Scrape BOSS直聘 (job listing site) via Chrome CDP. An agent skill from eatmoreduck/boss-zhipin-scraper.

    1.5k GitHub stars~2.6k tokensUpdated today
    Data & AnalyticsAuto-check passed
  • Crawl4AI Web Scraping

    smallnest/goclaw

    Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.

    599 GitHub starsUsed in 1 repo~2.5k tokens
    Data & AnalyticsAuto-check passed
  • Ax

    yusukebe/ax

    Use the ax CLI instead of curl + throwaway parsing scripts whenever you fetch a URL, explore an unknown web page, or extract structured data from HTML.

    719 GitHub starsUsed in 1 repo~918 tokens
    Data & AnalyticsAuto-check passed
  • Anakinscraper

    Anakin-Inc/anakin

    Scrape any website into clean markdown or structured JSON. An agent skill from Anakin-Inc/anakin.

    4.5k GitHub stars~859 tokensUpdated 1 mo ago
    Data & AnalyticsAuto-check passed

More from gooseworks-ai/goose-skills

All 273 skills in this repo
  • Reddit Post Finder

    gooseworks-ai/goose-skills

    Scrape and search Reddit posts using Apify. An agent skill from gooseworks-ai/goose-skills.

    1.2k GitHub starsUsed in 1 repo~1.2k tokens
    Auto-check passed
  • Create Image Fal

    gooseworks-ai/goose-skills

    Generate or edit an image via any FAL image model (nano-banana edit, gpt-image, flux, ...), ROUTED THROUGH THE fal-proxy so it bills the Ads agent.

    1.2k GitHub stars~1.3k tokensUpdated today
    Auto-check passed
  • Render Hook Replacement

    gooseworks-ai/goose-skills

    Replace an existing video's opening with a supplied clip or free kinetic text hook while retaining and verifying every original body frame, audio, captions and ending.

    1.2k GitHub stars~2.3k tokensUpdated today
    Auto-check passed
  • Blog Feed Monitor

    gooseworks-ai/goose-skills

    Scrape blog posts via RSS feeds (free, no API key) with Apify fallback for JS-heavy sites.

    1.2k GitHub starsUsed in 1 repo~578 tokens
    Auto-check passed
  • Competitor Post Engagers

    gooseworks-ai/goose-skills

    Find leads by scraping engagers from a competitor's top LinkedIn posts.

    1.2k GitHub starsUsed in 1 repo~1.8k tokens
    Auto-check: notes
  • Render Chatgpt Chat

    gooseworks-ai/goose-skills

    Assemble a ChatGPT chat-reveal video ad from a thread + timeline JSON — one continuous Playwright recording of a ChatGPT mobile chat (user types with the iOS keyboard up → taps send → keyboard…

    1.2k GitHub stars~2.3k tokensUpdated today
    Auto-check passed

Questions about Extract Webpage Data

What does Extract Webpage Data do?

Extract structured data from web pages using AI. An agent skill from gooseworks-ai/goose-skills. Extract Webpage Data is an agent skill from gooseworks-ai/goose-skills.

When should I use Extract Webpage Data?

Extract Webpage Data fits situations like: tasks that involve Web scraping.

How do I install Extract Webpage Data in Claude Code?

Run `npx skills add gooseworks-ai/goose-skills --skill extract-webpage-data -a claude-code`. Or copy the skill folder (skills/research-tools/capabilities/extract-webpage-data in gooseworks-ai/goose-skills) into .claude/skills/extract-webpage-data in your project. Claude Code loads it when a task matches its description.

How do I install Extract Webpage Data in Codex?

Run `npx skills add gooseworks-ai/goose-skills --skill extract-webpage-data -a codex`. Or copy the skill folder (skills/research-tools/capabilities/extract-webpage-data in gooseworks-ai/goose-skills) into .agents/skills/extract-webpage-data in your project. Codex loads it when a task matches its description.

Can I use Extract Webpage Data in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add gooseworks-ai/goose-skills --skill extract-webpage-data -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/extract-webpage-data, .gemini/skills/extract-webpage-data, .github/skills/extract-webpage-data and .opencode/skills/extract-webpage-data in your project.

What does Extract Webpage Data need to run?

Going by SKILL.md and its folder, Extract Webpage Data needs the command-line tools its instructions call (curl, python3 and npx) and credentials named GOOSEWORKS_API_KEY. Our summary lists: Python 3; Node.js; A credential in GOOSEWORKS_API_KEY.

Does Extract Webpage Data access the network?

SKILL.md names 1 domain. In commands or code: api.gooseworks.ai; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is Extract Webpage Data safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Extract Webpage Data use?

Extract Webpage Data is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Extract Webpage Data use?

About 1.9k tokens (SKILL.md is roughly 7.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Extract Webpage Data?

Skills that share tags, products or a category with Extract Webpage Data: Tmux (trpc-group/trpc-agent-go, 1.9k stars), Ketch (1broseidon/ketch, 702 stars), Boss Zhipin Scraper (eatmoreduck/boss-zhipin-scraper, 1.5k stars) and Crawl4AI Web Scraping (smallnest/goclaw, 599 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Extract Webpage Data?

gooseworks-ai (a GitHub organization) maintains it in gooseworks-ai/goose-skills, which has 1,240 GitHub stars. The repository holds 273 skills in this directory. The repository was last updated on October 8, 2026.

Source: gooseworks-ai/goose-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.