New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Minerals Web Ingest

by lamm-mit in lamm-mit/scienceclaw

Ingest and normalize web pages for critical-minerals intelligence, with optional Firecrawl fetching, deduplication manifest, and JSONL export

Apache-2.0Auto-check passedData & Analytics

Install Minerals Web Ingest

skills CLI
$ npx skills add lamm-mit/scienceclaw --skill minerals-web-ingest -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install lamm-mit/scienceclaw minerals-web-ingest --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/lamm-mit/scienceclaw.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/minerals-web-ingest .claude/skills/minerals-web-ingest && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
minerals-web-ingest
GitHub stars
246
Token cost
~512 tokens
SKILL.md length
125 words
Files
2 (incl. scripts)
Skills in repo
86
Repo updated
First seen
Licence
Apache-2.0

At a glance

Ingest and normalize web pages for critical-minerals intelligence, with optional Firecrawl fetching, deduplication manifest, and JSONL export

  • Tasks that involve Web scraping
  • SKILL.md covers Usage, Parameters, Output Schema and Notes
  • Runs Python scripts from its folder; calls python3; reaches energy.gov and usgs.gov; needs FIRECRAWL_API_KEY
  • Tasks that involve Data cleaning

What it does

Minerals Web Ingest is an agent skill from lamm-mit/scienceclaw. Ingest and normalize web pages for critical-minerals intelligence, with optional Firecrawl fetching, deduplication manifest, and JSONL export

Its SKILL.md is about 510 tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files, including scripts (for example `scripts/web_ingest.py`).

It sits in Data & Analytics, covering Web scraping and Data cleaning. It works with Firecrawl. The licence is Apache-2.0.

When your agent uses it

  • Tasks that involve Web scraping
  • Tasks that involve Data cleaning

Example prompts

  • “/minerals-web-ingest”

Requirements

  • Python 3
  • A credential in FIRECRAWL_API_KEY

What it can do on your machine

Read from SKILL.md and the folder at commit ab9aba1. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 1 file in scripts/ (Python), which the agent can run.

    Shell commands in SKILL.md call:

    • python3

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • energy.gov
    • usgs.gov

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names these keys or tokens, usually read from environment variables:

    • FIRECRAWL_API_KEY

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Minerals Web Ingest loads about 512 tokens when it runs. Until then it costs about 40 tokens; SKILL.md has 125 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~40
When it runs · the whole SKILL.md, loaded when a task matches
~512

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from lamm-mit/scienceclaw at commit ab9aba1, republished under its Apache-2.0 licence (© lamm-mit). 125 words, ~512 tokens.

Download SKILL.mdSave it as .claude/skills/minerals-web-ingest/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
minerals-web-ingest
description
Ingest and normalize web pages for critical-minerals intelligence, with optional Firecrawl fetching, deduplication manifest, and JSONL export

Minerals Web Ingest

Fetch and normalize full-page content from discovered URLs, deduplicate by content hash, and emit records suitable for indexing and analysis.

Usage

bash
# Ingest URLs from monitor output
python3 {baseDir}/scripts/web_ingest.py \
  --input-json monitor_records.json \
  --output-jsonl ingested_records.jsonl \
  --format summary

# Direct URL ingest
python3 {baseDir}/scripts/web_ingest.py \
  --url https://www.energy.gov/articles/example \
  --url https://www.usgs.gov/news/example \
  --format json

# Prefer Firecrawl (if FIRECRAWL_API_KEY is set)
python3 {baseDir}/scripts/web_ingest.py --input-json gov_records.json --prefer-firecrawl

Parameters

ParameterDescriptionDefault
--input-jsonJSON input file with URL records-
--urlDirect URL input (repeatable)-
--output-jsonlOptional JSONL output path-
--manifest-pathURL->content hash dedupe manifest~/.scienceclaw/minerals_web_ingest_manifest.json
--timeoutHTTP timeout seconds30
--max-charsMax chars stored per page12000
--prefer-firecrawlUse Firecrawl first, fallback to requestsfalse
--formatsummary, detailed, jsonsummary

Output Schema

Ingested records include: url, source, published_at, title, summary, commodity_tags, country_tags, policy_signal, confidence, retrieved_at, source_type, content, content_hash.

Notes

  • If Firecrawl is unavailable or fails, script falls back to requests + BeautifulSoup extraction.
  • Manifest-based dedupe prevents reprocessing unchanged pages.
  • Use this output as input to corpus indexing or profile-generation pipelines.

© lamm-mit, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (scripts) in skills/minerals-web-ingest of lamm-mit/scienceclaw.

  • SKILL.md
  • scripts/web_ingest.py

Open the folder on GitHubat commit ab9aba1

Compare with similar skills

Minerals Web Ingest next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Minerals Web Ingest compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Minerals Web Ingest this skilllamm-mit/scienceclaw246—~512Automated safety check: PassApache-2.0
Authoritative Data Harvesteryushui2022/MathModel-Skill4541 repos~1.1kAutomated safety check: PassMIT
Firecrawl Scrapefirecrawl/skills117—~1.8kAutomated safety check: PassISC
Firecrawl Agentfirecrawl/skills117—~1.2kAutomated safety check: PassISC
Firecrawl Company Directoriesfirecrawl/skills117—~557Automated safety check: PassISC
Firecrawl Competitive Intelfirecrawl/skills117—~604Automated safety check: PassISC

Similar skills

  • Authoritative Data Harvester

    yushui2022/MathModel-Skill

    Finds authoritative public data sources for modeling tasks, prefers official APIs and bulk downloads, and outputs a reproducible fetch and cleaning plan with citations.

    454 GitHub starsUsed in 1 repo~1.1k tokens
    Data & AnalyticsAuto-check passed

  • Firecrawl Scrape

    firecrawl/skills

    Read a known webpage or execute a discovered workflow or data-provider capability.

    117 GitHub stars~1.8k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Firecrawl Agent

    firecrawl/skills

    Autonomously navigate websites and extract structured data across pages.

    117 GitHub stars~1.2k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Firecrawl Company Directories

    firecrawl/skills

    Extract structured company lists from directories with Firecrawl.

    117 GitHub stars~557 tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Firecrawl Competitive Intel

    firecrawl/skills

    Monitor competitor pricing, features, changelogs, dashboards, and product changes with Firecrawl.

    117 GitHub stars~604 tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Firecrawl

    zapier/connectors

    Official

    Agent-callable Firecrawl tools — scrape a URL to clean Markdown, crawl a site, search the web, map site URLs, and extract structured data.

    177 GitHub stars~4.2k tokensUpdated 1 mo ago
    Data & AnalyticsAuto-check passed

More from lamm-mit/scienceclaw

All 86 skills in this repo

  • Fred Economic Data

    lamm-mit/scienceclaw

    Query FRED (Federal Reserve Economic Data) API for 800,000+ economic time series from 100+ sources.

    246 GitHub starsUsed in 4 repos~3k tokens
    Auto-check passed

  • Drug Research

    lamm-mit/scienceclaw

    Generates comprehensive drug research reports with compound disambiguation, evidence grading, and mandatory completeness sections.

    246 GitHub starsUsed in 3 repos~1.7k tokens
    Auto-check passed

  • Imaging Data Commons

    lamm-mit/scienceclaw

    Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index.

    246 GitHub starsUsed in 5 repos~11k tokens
    Auto-check passed

  • Rowan

    lamm-mit/scienceclaw

    Cloud-based quantum chemistry platform with Python API. An agent skill from lamm-mit/scienceclaw.

    246 GitHub starsUsed in 4 repos~3.1k tokens
    Auto-check: warnings

  • Infographics

    lamm-mit/scienceclaw

    Create professional infographics using Nano Banana Pro AI with smart iterative refinement.

    246 GitHub starsUsed in 6 repos~4.4k tokens
    Auto-check: notes

  • Disease Research

    lamm-mit/scienceclaw

    Generate comprehensive disease research reports using 100+ ToolUniverse tools.

    246 GitHub stars~946 tokensUpdated 1 mo ago
    Auto-check passed

Works with

Categories

Questions about Minerals Web Ingest

What does Minerals Web Ingest do?

Ingest and normalize web pages for critical-minerals intelligence, with optional Firecrawl fetching, deduplication manifest, and JSONL export. Minerals Web Ingest is an agent skill from lamm-mit/scienceclaw.

When should I use Minerals Web Ingest?

Minerals Web Ingest fits situations like: tasks that involve Web scraping; tasks that involve Data cleaning.

How do I install Minerals Web Ingest in Claude Code?

Run `npx skills add lamm-mit/scienceclaw --skill minerals-web-ingest -a claude-code`. Or copy the skill folder (skills/minerals-web-ingest in lamm-mit/scienceclaw) into .claude/skills/minerals-web-ingest in your project. Claude Code loads it when a task matches its description.

How do I install Minerals Web Ingest in Codex?

Run `npx skills add lamm-mit/scienceclaw --skill minerals-web-ingest -a codex`. Or copy the skill folder (skills/minerals-web-ingest in lamm-mit/scienceclaw) into .agents/skills/minerals-web-ingest in your project. Codex loads it when a task matches its description.

Can I use Minerals Web Ingest in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add lamm-mit/scienceclaw --skill minerals-web-ingest -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/minerals-web-ingest, .gemini/skills/minerals-web-ingest, .github/skills/minerals-web-ingest and .opencode/skills/minerals-web-ingest in your project.

What does Minerals Web Ingest need to run?

Going by SKILL.md and its folder, Minerals Web Ingest needs Python for the scripts in its folder, the command-line tools its instructions call (python3) and credentials named FIRECRAWL_API_KEY. Our summary lists: Python 3; A credential in FIRECRAWL_API_KEY.

Does Minerals Web Ingest access the network?

SKILL.md names 2 domains. In commands or code: energy.gov and usgs.gov; the agent is likely to contact these when it follows the instructions. This is read from the text; nothing was executed.

Is Minerals Web Ingest safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Minerals Web Ingest use?

Minerals Web Ingest is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Minerals Web Ingest use?

About 512 tokens (SKILL.md is roughly 2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Minerals Web Ingest?

Skills that share tags, products or a category with Minerals Web Ingest: Authoritative Data Harvester (yushui2022/MathModel-Skill, 454 stars), Firecrawl Scrape (firecrawl/skills, 117 stars), Firecrawl Agent (firecrawl/skills, 117 stars) and Firecrawl Company Directories (firecrawl/skills, 117 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Minerals Web Ingest?

lamm-mit (a GitHub user) maintains it in lamm-mit/scienceclaw, which has 246 GitHub stars. The repository holds 86 skills in this directory. The repository was last updated on August 21, 2026.

Source: lamm-mit/scienceclaw on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.