Crawl4AI Web Scraping
smallnest/goclaw
Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.
Handle training data and model artifacts on CoreWeave persistent storage.
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handling --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .claude/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .claude/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handlingType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handling --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .agents/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .agents/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handling --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .cursor/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .cursor/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/jeremylongshore/tons-of-skills-marketplace.git --path skills/.curated/coreweave-data-handling--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handling --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .gemini/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .gemini/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handlingInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .github/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .github/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install jeremylongshore/tons-of-skills-marketplace coreweave-data-handling --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/.curated/coreweave-data-handling .opencode/skills/coreweave-data-handling && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "coreweave-data-handling" agent skill from https://github.com/jeremylongshore/tons-of-skills-marketplace/tree/main/skills/.curated/coreweave-data-handling into .opencode/skills/coreweave-data-handling/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "coreweave-data-handling", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
coreweave-data-handlingHandle training data and model artifacts on CoreWeave persistent storage.
Coreweave Data Handling is an agent skill from jeremylongshore/tons-of-skills-marketplace. Handle training data and model artifacts on CoreWeave persistent storage. Use when managing large datasets, configuring storage classes, or implementing data pipelines for GPU workloads. Trigger with phrases like "coreweave data", "coreweave storage", "coreweave pvc", "coreweave dataset management".
Its SKILL.md is about 1.8k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts. Compatibility notes: Designed for Claude Code
It sits in Data & Analytics, covering Data pipelines and ETL. The repository describes itself as: Model-agnostic agent-skills platform with a harness-free canonical layer, verified adapters, and the ccpi package manager. Explore at tonsofskills.com. The licence is MIT.
4 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit cfae287. It shows what the files ask for, not the result of running them.
Pre-approves these tools, so the agent can use them without asking each time:
ReadWriteEditBash(kubectl:*)GrepFrom allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
kubectlFrom the folder's file list and the shell code blocks in SKILL.md.
Links to these hosts (documentation or services it may open):
kubernetes.ioFrom URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Designed for Claude Code
From compatibility in the SKILL.md frontmatter.
Coreweave Data Handling loads about 1.8k tokens when it runs. Until then it costs about 81 tokens; SKILL.md has 514 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from jeremylongshore/tons-of-skills-marketplace at commit cfae287, republished under its MIT licence (© jeremylongshore). 514 words, ~1,799 tokens.
.claude/skills/coreweave-data-handling/SKILL.md (or your agent's skills folder).Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.
CoreWeave GPU cloud workloads involve large-scale data artifacts: model weights (multi-GB safetensors/GGUF), training datasets (parquet, TFRecord, WebDataset), checkpoint snapshots, and inference cache volumes. Data flows through Kubernetes PersistentVolumeClaims backed by region-specific storage classes. Compliance requires encryption at rest via the storage driver, namespace-scoped RBAC for volume access, and audit logging for any data egress from GPU nodes.
| Data Type | Sensitivity | Retention | Encryption |
|---|---|---|---|
| Model weights | Medium | Until deprecated | AES-256 at rest |
| Training datasets | High (may contain PII) | Per data license | AES-256 + TLS in transit |
| Checkpoint snapshots | Medium | 30 days post-training | AES-256 at rest |
| Inference cache | Low | Session/TTL | Volume-level encryption |
| HuggingFace tokens | Critical | Rotate quarterly | K8s Secret + KMS |
import { KubeConfig, BatchV1Api } from '@kubernetes/client-node';
async function importDataset(pvcName: string, sourceUrl: string, namespace: string) {
const kc = new KubeConfig();
kc.loadFromDefault();
const batch = kc.makeApiClient(BatchV1Api);
const job = {
metadata: { name: `import-${Date.now()}`, namespace },
spec: { template: { spec: {
restartPolicy: 'Never',
containers: [{ name: 'loader', image: 'python:3.11-slim',
command: ['python3', '-c', `
import urllib.request, hashlib
dest = '/data/dataset.tar.gz'
urllib.request.urlretrieve('${sourceUrl}', dest)
print(f"SHA256: {hashlib.sha256(open(dest,'rb').read()).hexdigest()}")`],
volumeMounts: [{ name: 'storage', mountPath: '/data' }],
}],
volumes: [{ name: 'storage', persistentVolumeClaim: { claimName: pvcName } }],
}}}
};
await batch.createNamespacedJob(namespace, { body: job });
}async function exportCheckpoint(pvcName: string, destBucket: string, ns: string) {
// Validate export destination is in approved region list
const APPROVED_REGIONS = ['us-east-1', 'us-central-1', 'eu-west-1'];
const region = destBucket.split('-').slice(0, 3).join('-');
if (!APPROVED_REGIONS.some(r => destBucket.includes(r))) {
throw new Error(`Export blocked: ${region} not in approved regions`);
}
// Stream from PVC → object storage with integrity check
const exportCmd = `tar czf - /models | gsutil cp - gs://${destBucket}/export.tar.gz`;
console.log(`Exporting from PVC ${pvcName} to ${destBucket}`);
return exportCmd;
}interface ModelArtifact {
name: string; format: 'safetensors' | 'gguf' | 'bin' | 'pt';
sizeBytes: number; sha256: string;
}
function validateArtifact(artifact: ModelArtifact): string[] {
const errors: string[] = [];
if (!artifact.name || artifact.name.length > 255) errors.push('Invalid artifact name');
if (artifact.sizeBytes <= 0) errors.push('Size must be positive');
if (!/^[a-f0-9]{64}$/.test(artifact.sha256)) errors.push('Invalid SHA-256 hash');
if (!['safetensors', 'gguf', 'bin', 'pt'].includes(artifact.format)) errors.push(`Unsupported format`);
return errors;
}| Issue | Cause | Fix |
|---|---|---|
| PVC pending indefinitely | Storage class unavailable in region | Check kubectl get sc and switch to available class |
| Download job OOMKilled | Dataset exceeds container memory limit | Increase resource limits or use streaming download |
| Permission denied on volume | RBAC misconfigured for namespace | Verify ServiceAccount has PVC access via RoleBinding |
| Checksum mismatch after import | Partial transfer or corruption | Re-run import job; enable retry with backoff |
| Secret not found | KMS key rotation or namespace mismatch | Verify secret exists in target namespace with kubectl get secret |
Import only a manifest-approved artifact and verify its checksum inside the isolated job before promoting it to a serving or training workload:
kubectl -n research apply -f dataset-import-job.yaml
kubectl -n research wait --for=condition=complete job/dataset-import --timeout=30m
kubectl -n research logs job/dataset-import | grep SHA256If the checksum differs, quarantine the PVC content, retain the redacted job receipt, and reacquire the artifact from the approved source. Do not retry into a production volume or disable integrity verification.
See coreweave-security-basics.
© jeremylongshore, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in skills/.curated/coreweave-data-handling of jeremylongshore/tons-of-skills-marketplace.
Open the folder on GitHubat commit cfae287
Coreweave Data Handling next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Coreweave Data Handling this skilljeremylongshore/tons-of-skills-marketplace | 2.8k | — | ~1.8k | Automated safety check: Pass | MIT | |
| Crawl4AI Web Scrapingsmallnest/goclaw | 599 | 1 repos | ~2.5k | Automated safety check: Pass | MIT | |
| Glue 09 10 Migrationaws-samples/aws-glue-samples | 1.5k | — | ~2.4k | Automated safety check: Pass | MIT-0 | |
| Migrate Glue Devendpoint To Interactive Sessionsaws-samples/aws-glue-samples | 1.5k | — | ~3.6k | Automated safety check: Pass | MIT-0 | |
| Dbt Databricks PR Readydatabricks/dbt-databricks | 380 | — | ~2.8k | Automated safety check: Pass | Apache-2.0 | |
| Apache Spark EngineerJeffallan/claude-skills | 12k | 1 repos | ~1.7k | Automated safety check: Pass | MIT |
smallnest/goclaw
Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.
aws-samples/aws-glue-samples
Upgrade an AWS Glue ETL job from Glue version 0.9 or 1.0 to Glue 4.0.
aws-samples/aws-glue-samples
Migrate a legacy AWS Glue development endpoint to a Glue interactive session, following the official AWS migration checklist.
databricks/dbt-databricks
A skill your agent uses for an open dbt-databricks pull request, including your own PR or a fork PR, to assess merge readiness and optionally repair selected gaps on the PR head branch.
Jeffallan/claude-skills
Guides writing and tuning Apache Spark jobs: DataFrame and RDD code, Spark SQL, partitioning, caching, shuffle tuning and structured streaming.
MaterializeInc/materialize
Cut a dbt-materialize PyPI release: bump the version in version.py and setup.py, date the Unreleased CHANGELOG entry, and open the release PR with a Ship: <url body.
jeremylongshore/tons-of-skills-marketplace
Execute this skill enables AI assistant to conduct a security-focused code review using the security-agent plugin.
jeremylongshore/tons-of-skills-marketplace
Build this skill automates the adaptation of pre-trained machine learning models using transfer learning techniques.
jeremylongshore/tons-of-skills-marketplace
Execute proactive auto-loading: automatically detects and loads agents.md files.
jeremylongshore/tons-of-skills-marketplace
Aggregate and centralize performance metrics from applications, systems, databases, caches, and services.
jeremylongshore/tons-of-skills-marketplace
Execute this skill enables AI assistant to analyze capacity requirements and plan for future growth.
jeremylongshore/tons-of-skills-marketplace
Process use when you need to work with database indexing. An agent skill from jeremylongshore/tons-of-skills-marketplace.
Categories
Handle training data and model artifacts on CoreWeave persistent storage. Coreweave Data Handling is an agent skill from jeremylongshore/tons-of-skills-marketplace. Handle training data and model artifacts on CoreWeave persistent storage.
Coreweave Data Handling fits situations like: managing large datasets; configuring storage classes; implementing data pipelines for GPU workloads; with phrases like coreweave data.
Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a claude-code`. Or copy the skill folder (skills/.curated/coreweave-data-handling in jeremylongshore/tons-of-skills-marketplace) into .claude/skills/coreweave-data-handling in your project. Claude Code loads it when a task matches its description.
Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a codex`. Or copy the skill folder (skills/.curated/coreweave-data-handling in jeremylongshore/tons-of-skills-marketplace) into .agents/skills/coreweave-data-handling in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill coreweave-data-handling -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/coreweave-data-handling, .gemini/skills/coreweave-data-handling, .github/skills/coreweave-data-handling and .opencode/skills/coreweave-data-handling in your project.
Going by SKILL.md and its folder, Coreweave Data Handling needs the command-line tools its instructions call (kubectl). Its frontmatter pre-approves these tools: Read, Write, Edit, Bash(kubectl:*), Grep. Compatibility (from SKILL.md): Designed for Claude Code.
SKILL.md names 1 domain. As links in the text: kubernetes.io. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Coreweave Data Handling is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.8k tokens (SKILL.md is roughly 7.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Coreweave Data Handling: Crawl4AI Web Scraping (smallnest/goclaw, 599 stars), Glue 09 10 Migration (aws-samples/aws-glue-samples, 1.5k stars), Migrate Glue Devendpoint To Interactive Sessions (aws-samples/aws-glue-samples, 1.5k stars) and Dbt Databricks PR Ready (databricks/dbt-databricks, 380 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
jeremylongshore (a GitHub user) maintains it in jeremylongshore/tons-of-skills-marketplace, which has 2,827 GitHub stars. The repository holds 3,342 skills in this directory. The repository was last updated on October 10, 2026.
Source: jeremylongshore/tons-of-skills-marketplace on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.