Agent skill

Data Pipeline

Design, build, or debug data processing pipelines. An agent skill from agulli/atlas-agents.

Install Data Pipeline

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .claude/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .claude/skills/data-pipeline && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .claude/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .agents/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .agents/skills/data-pipeline && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .agents/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .cursor/skills/data-pipeline && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .cursor/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/agulli/atlas-agents.git --path ch09_agent_skills/skills/data-pipeline

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .gemini/skills/data-pipeline && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .gemini/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .github/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .github/skills/data-pipeline && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .github/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add agulli/atlas-agents --skill data-pipeline -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install agulli/atlas-agents data-pipeline --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/agulli/atlas-agents.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/ch09_agent_skills/skills/data-pipeline .opencode/skills/data-pipeline && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "data-pipeline" agent skill from https://github.com/agulli/atlas-agents/tree/main/ch09_agent_skills/skills/data-pipeline into .opencode/skills/data-pipeline/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "data-pipeline", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: data-pipeline
GitHub stars: 579
Token cost: ~714 tokens
SKILL.md length: 275 words
Files: 1
Skills in repo: 12
Repo updated: Jul 17, 2026
First seen: Oct 7, 2026
Licence: MIT

At a glance

Design, build, or debug data processing pipelines. An agent skill from agulli/atlas-agents.

Works in 6 steps: Define the contract. Before writing any… → Validate at the boundary. The first… → Make it idempotent. Running the pipeline… → …
Asked to process a dataset
SKILL.md covers Overview, Process, Rationalizations and Verification
Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Data Pipeline is an agent skill from agulli/atlas-agents. Design, build, or debug data processing pipelines. Use when asked to process a dataset, transform data, build an ETL pipeline, schedule batch jobs, or fix data quality issues.

Its SKILL.md is about 710 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts. Compatibility notes: Requires python 3.10+

It sits in Data & Analytics, covering Data pipelines and ETL, Background jobs and Data cleaning. The licence is MIT.

When your agent uses it

Asked to process a dataset
Build an ETL pipeline
Schedule batch jobs
Fix data quality issues

Example prompts

“/data-pipeline”

Requirements

Python 3
Compatibility (from SKILL.md): Requires python 3.10+

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

Define the contract. Before writing any transformation code, specify
Validate at the boundary. The first thing any pipeline stage does is validate its input
Make it idempotent. Running the pipeline twice on the same input must produce the same output. Use upserts, not inserts. Use deterministic…
Log progress at meaningful checkpoints. After every major stage (extract, validate, transform, load), log the record count and any failures.
Test with a sample. Before running on the full dataset, run on 100 records. Confirm the output schema, record count, and that no records…
Run on the full dataset. Monitor progress. On completion, report: records in, records out, records failed, and time elapsed.

What it can do on your machine

Read from SKILL.md and the folder at commit 2b21998. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md (its code samples are python).
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Compatibility
Requires python 3.10+
From compatibility in the SKILL.md frontmatter.

Context cost

Data Pipeline loads about 714 tokens when it runs. Until then it costs about 47 tokens; SKILL.md has 275 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~47
When it runs · the whole SKILL.md, loaded when a task matches: ~714

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from agulli/atlas-agents at commit 2b21998, republished under its MIT licence (© agulli). 275 words, ~714 tokens.

Download SKILL.mdSave it as .claude/skills/data-pipeline/SKILL.md (or your agent's skills folder).

name: data-pipeline
description: Design, build, or debug data processing pipelines. Use when asked to process a dataset, transform data, build an ETL pipeline, schedule batch jobs, or fix data quality issues.
compatibility: Requires python 3.10+
license: MIT

Overview

Data pipelines fail silently and corrupt downstream systems. Every pipeline must be observable, idempotent, and validated at the boundary.

Process

Define the contract. Before writing any transformation code, specify:
- Input schema: What fields, types, and constraints does the data arrive with?
- Output schema: What fields, types, and constraints must the output satisfy?
- Volume: How many records? Per-run? Per-day?
- Frequency: One-time, scheduled, or event-driven?

Validate at the boundary. The first thing any pipeline stage does is validate its input:

python

from pydantic import BaseModel, ValidationError

class InputRecord(BaseModel):
    user_id: int
    event_type: str
    timestamp: str  # ISO 8601
    value: float | None = None

def process(raw_records: list[dict]) -> list[dict]:
    valid, invalid = [], []
    for r in raw_records:
        try:
            valid.append(InputRecord(**r).model_dump())
        except ValidationError as e:
            invalid.append({"record": r, "error": str(e)})
    if invalid:
        log_invalid_records(invalid)  # Never silently drop
    return transform(valid)

Make it idempotent. Running the pipeline twice on the same input must produce the same output. Use upserts, not inserts. Use deterministic IDs based on input content, not auto-increment.
Log progress at meaningful checkpoints. After every major stage (extract, validate, transform, load), log the record count and any failures.
Test with a sample. Before running on the full dataset, run on 100 records. Confirm the output schema, record count, and that no records were silently dropped.
Run on the full dataset. Monitor progress. On completion, report: records in, records out, records failed, and time elapsed.

Rationalizations

Excuse	Rebuttal
"I'll add validation later"	Invalid data corrupts your database. Validate at the boundary now.
"Logging slows the pipeline down"	A pipeline that fails without logs requires a full rerun to debug. Log it.
"It worked on the sample"	Test samples are not representative. Always run a full-dataset dry run before writing to the destination.

Verification

Input and output schemas are defined before any code is written
Invalid records are logged, not silently dropped
Pipeline was tested on a 100-record sample before full run
Final report includes: records in, records out, records failed

Files

Just SKILL.md in ch09_agent_skills/skills/data-pipeline of agulli/atlas-agents.

Open the folder on GitHubat commit 2b21998

Compare with similar skills

Data Pipeline next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Data Pipeline compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Data Pipeline this skillagulli/atlas-agents	579	—	~714	Automated safety check: Pass	MIT	2 mo ago
Monitor With HaolemeHaolemeApp/Haoleme	157	—	~1.3k	Automated safety check: Pass	AGPL-3.0	1 mo ago
Credit Risk Data Cleaninggithub/awesome-copilot	40k	1 repos	~1.5k	Automated safety check: Pass	MIT	today
Authoritative Data Harvesteryushui2022/MathModel-Skill	452	1 repos	~1.1k	Automated safety check: Pass	MIT	yesterday
Data Quality Frameworkswshobson/agents	40k	11 repos	~1.1k	Automated safety check: Pass	MIT	3 days ago
Dbt Transformation Patternswshobson/agents	40k	9 repos	~781	Automated safety check: Pass	MIT	3 days ago

Similar skills

Monitor With Haoleme
HaolemeApp/Haoleme
Selectively monitor important long-running or resource-intensive commands with Haoleme by prefixing them with hao, so status, output, and completion notifications sync to the mobile app.
157 GitHub stars~1.3k tokensUpdated 1 mo ago
Data & AnalyticsAuto-check passed
Credit Risk Data Cleaning
github/awesome-copilot
Official
Cleans raw credit data and screens variables before loan modeling, dropping unstable, noisy or redundant features and writing an Excel report of every step.
40k GitHub starsUsed in 1 repo~1.5k tokens
Data & AnalyticsAuto-check passed
Authoritative Data Harvester
yushui2022/MathModel-Skill
Finds authoritative public data sources for modeling tasks, prefers official APIs and bulk downloads, and outputs a reproducible fetch and cleaning plan with citations.
452 GitHub starsUsed in 1 repo~1.1k tokens
Data & AnalyticsAuto-check passed
Data Quality Frameworks
wshobson/agents
Sets up data quality checks with Great Expectations, dbt tests and data contracts, with checkpoints and pass-fail reports for pipelines.
40k GitHub starsUsed in 11 repos~1.1k tokens
Data & AnalyticsAuto-check passed
Dbt Transformation Patterns
wshobson/agents
Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies.
40k GitHub starsUsed in 9 repos~781 tokens
Data & AnalyticsAuto-check passed
Data Engineering
rohitg00/awesome-claude-code-toolkit
Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation
2.7k GitHub stars~1.7k tokensUpdated 4 mo ago
Data & AnalyticsAuto-check passed

Questions about Data Pipeline

What does Data Pipeline do?

Design, build, or debug data processing pipelines. An agent skill from agulli/atlas-agents. Data Pipeline is an agent skill from agulli/atlas-agents. Design, build, or debug data processing pipelines.

When should I use Data Pipeline?

Data Pipeline fits situations like: asked to process a dataset; build an ETL pipeline; schedule batch jobs; fix data quality issues.

How do I install Data Pipeline in Claude Code?

Run `npx skills add agulli/atlas-agents --skill data-pipeline -a claude-code`. Or copy the skill folder (ch09_agent_skills/skills/data-pipeline in agulli/atlas-agents) into .claude/skills/data-pipeline in your project. Claude Code loads it when a task matches its description.

How do I install Data Pipeline in Codex?

Run `npx skills add agulli/atlas-agents --skill data-pipeline -a codex`. Or copy the skill folder (ch09_agent_skills/skills/data-pipeline in agulli/atlas-agents) into .agents/skills/data-pipeline in your project. Codex loads it when a task matches its description.

Can I use Data Pipeline in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add agulli/atlas-agents --skill data-pipeline -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/data-pipeline, .gemini/skills/data-pipeline, .github/skills/data-pipeline and .opencode/skills/data-pipeline in your project.

What does Data Pipeline need to run?

SKILL.md names no scripts, command-line tools or credentials: Data Pipeline is instructions for the agent only. Our summary lists: Python 3. Compatibility (from SKILL.md): Requires python 3.10+.

Does Data Pipeline access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Data Pipeline safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Data Pipeline use?

Data Pipeline is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Data Pipeline use?

About 714 tokens (SKILL.md is roughly 2.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Data Pipeline?

Skills that share tags, products or a category with Data Pipeline: Monitor With Haoleme (HaolemeApp/Haoleme, 157 stars), Credit Risk Data Cleaning (github/awesome-copilot, 40k stars), Authoritative Data Harvester (yushui2022/MathModel-Skill, 452 stars) and Data Quality Frameworks (wshobson/agents, 40k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Data Pipeline?

agulli (a GitHub user) maintains it in agulli/atlas-agents, which has 579 GitHub stars. The repository holds 12 skills in this directory. The repository was last updated on July 17, 2026.

Source: agulli/atlas-agents on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Data Pipeline

Install Data Pipeline

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Compatibility

Context cost

Safety

SKILL.md

Overview

Process

Rationalizations

Verification

Files

Compare with similar skills

Similar skills

Monitor With Haoleme

Credit Risk Data Cleaning

Authoritative Data Harvester

Data Quality Frameworks

Dbt Transformation Patterns

Data Engineering

More from agulli/atlas-agents

API Design

Database Migration

Deploy Checklist

Documentation Writer

Git Commit

Performance Profiling

Questions about Data Pipeline

Install Data Pipeline

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Compatibility

Context cost

Safety

SKILL.md

Overview

Process

Rationalizations

Verification

Files

Compare with similar skills

Similar skills

More from agulli/atlas-agents

Related

Questions about Data Pipeline