New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Data Pipeline

by RightNow-AI in RightNow-AI/openfang

Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

Apache-2.0Auto-check passedData & Analytics

Install Data Pipeline

skills CLI
$ npx skills add RightNow-AI/openfang --skill data-pipeline -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install RightNow-AI/openfang data-pipeline --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/RightNow-AI/openfang.git skills-src && mkdir -p .claude/skills && cp -r skills-src/crates/openfang-skills/bundled/data-pipeline .claude/skills/data-pipeline && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
data-pipeline
GitHub stars
18k
Token cost
~847 tokens
SKILL.md length
438 words
Files
1
Skills in repo
68
Repo updated
First seen
Licence
Apache-2.0

At a glance

Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

  • Tasks that involve Data pipelines and ETL
  • SKILL.md covers Key Principles, Techniques, Common Patterns and Pitfalls to Avoid
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Data Pipeline is an agent skill from RightNow-AI/openfang. Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

Its SKILL.md is about 850 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Data & Analytics, covering Data pipelines and ETL. It works with dbt, Apache Airflow and Apache Spark. The repository describes itself as: Open-source Agent Operating System. The licence is Apache-2.0.

When your agent uses it

  • Tasks that involve Data pipelines and ETL

Example prompts

  • “/data-pipeline”

What it can do on your machine

Read from SKILL.md and the folder at commit acf2587. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Data Pipeline loads about 847 tokens when it runs. Until then it costs about 22 tokens; SKILL.md has 438 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~22
When it runs · the whole SKILL.md, loaded when a task matches
~847

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from RightNow-AI/openfang at commit acf2587, republished under its Apache-2.0 licence (© RightNow-AI). 438 words, ~847 tokens.

Download SKILL.mdSave it as .claude/skills/data-pipeline/SKILL.md (or your agent's skills folder).
name
data-pipeline
description
Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality

Data Pipeline Expert

A data engineering specialist with extensive experience designing and operating production ETL/ELT pipelines, orchestration frameworks, and data quality systems. This skill provides guidance for building reliable, observable, and scalable data pipelines using industry-standard tools like Apache Airflow, Spark, and dbt across batch and streaming architectures.

Key Principles

  • Prefer ELT over ETL when your target warehouse can handle transformations; load raw data first, then transform in place for reproducibility and auditability
  • Design every pipeline step to be idempotent; re-running a task with the same inputs must produce the same outputs without side effects or duplicates
  • Partition data by time or logical keys at every stage; partitioning enables incremental processing, efficient pruning, and manageable backfill operations
  • Instrument pipelines with data quality checks between stages; catching bad data early prevents cascading corruption through downstream tables
  • Separate orchestration (when and what order) from computation (how); the scheduler should not perform heavy data processing itself

Techniques

  • Build Airflow DAGs with task-level retries, timeouts, and SLAs; use sensors for external dependencies and XCom for lightweight inter-task communication
  • Design Spark jobs with proper partitioning (repartition/coalesce), broadcast joins for small dimension tables, and caching for reused DataFrames
  • Structure dbt projects with staging models (source cleaning), intermediate models (business logic), and mart models (final consumption tables)
  • Write dbt tests at multiple levels: schema tests (not_null, unique, accepted_values), relationship tests, and custom data tests for business rules
  • Implement data quality gates using frameworks like Great Expectations: define expectations on row counts, column distributions, and referential integrity
  • Use Change Data Capture (CDC) patterns with tools like Debezium to stream database changes into event pipelines without polling
Show full SKILL.md (170 more words)Show less

Common Patterns

  • Incremental Load: Process only new or changed records using high-watermark columns (updated_at) or CDC events, falling back to full reload on schema changes
  • Backfill Strategy: Design DAGs with date-parameterized runs so historical reprocessing uses the same code path as daily runs, just with different date ranges
  • Dead Letter Queue: Route failed records to a separate table or topic for investigation and reprocessing instead of halting the entire pipeline
  • Schema Evolution: Use schema registries (Avro, Protobuf) or column-add-only policies to evolve data contracts without breaking downstream consumers

Pitfalls to Avoid

  • Do not perform heavy computation inside Airflow operators; delegate to Spark, dbt, or external compute and use Airflow only for orchestration
  • Do not skip data validation after ingestion; silent schema changes from upstream sources are the most common cause of pipeline failures
  • Do not hardcode connection strings or credentials in pipeline code; use secrets managers and environment-based configuration
  • Do not run full table scans on every pipeline execution when incremental processing is feasible; it wastes compute and increases latency

© RightNow-AI, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in crates/openfang-skills/bundled/data-pipeline of RightNow-AI/openfang.

Open the folder on GitHubat commit acf2587

Compare with similar skills

Data Pipeline next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Data Pipeline compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Data Pipeline this skillRightNow-AI/openfang18k—~847Automated safety check: PassApache-2.0
Data Engineerdavila7/claude-code-templates32k8 repos~2.8kAutomated safety check: PassMIT
Migrating Dagster To Airflowastronomer/agents451—~3.8kAutomated safety check: PassApache-2.0
Deploying Airflowastronomer/agents4511 repos~2.8kAutomated safety check: PassApache-2.0
Senior Data Engineerbenchflow-ai/skillsbench1.8k—~5.9kAutomated safety check: PassMIT
Senior Data Engineeralirezarezvani/claude-skills28k3 repos~1.4kAutomated safety check: PassMIT

Similar skills

  • Data Engineer

    davila7/claude-code-templates

    Build scalable data pipelines, modern data warehouses, and real-time streaming architectures.

    32k GitHub starsUsed in 8 repos~2.8k tokens
    Data & AnalyticsAuto-check passed

  • Migrating Dagster To Airflow

    astronomer/agents

    Guide for migrating Dagster projects to Apache Airflow 3 on Astro.

    451 GitHub stars~3.8k tokensUpdated 2 days ago
    Data & AnalyticsAuto-check passed

  • Deploying Airflow

    astronomer/agents

    Deploys Airflow DAGs and projects. An agent skill from astronomer/agents.

    451 GitHub starsUsed in 1 repo~2.8k tokens
    Data & AnalyticsAuto-check passed

  • Senior Data Engineer

    benchflow-ai/skillsbench

    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure.

    1.8k GitHub stars~5.9k tokensUpdated 2 mo ago
    Data & AnalyticsAuto-check passed

  • Senior Data Engineer

    alirezarezvani/claude-skills

    Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

    28k GitHub starsUsed in 3 repos~1.4k tokens
    Data & AnalyticsAuto-check passed

  • Senior Data Engineer

    davila7/claude-code-templates

    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

    32k GitHub starsUsed in 1 repo~1.4k tokens
    Data & AnalyticsAuto-check passed

More from RightNow-AI/openfang

All 68 skills in this repo

  • Playwright Browser Automation Patterns

    RightNow-AI/openfang

    Reference of CSS selectors, step-by-step web workflows and error recovery tactics for an agent that browses, fills forms and compares prices on live sites.

    18k GitHub stars~1k tokensUpdated 3 mo ago
    Auto-check passed

  • Intelligence Collection Methodology

    RightNow-AI/openfang

    Reference knowledge for open-source intelligence collection: the collection cycle, source reliability tiers, search query patterns and entity extraction.

    18k GitHub stars~2.1k tokensUpdated 3 mo ago
    Auto-check passed

  • Lead Generation Research Guide

    RightNow-AI/openfang

    Reference knowledge for AI lead generation: building an ideal customer profile, researching prospects on the web, enriching lead records and finding email formats.

    18k GitHub stars~1.8k tokensUpdated 3 mo ago
    Auto-check passed

  • Video Clipping Reference

    RightNow-AI/openfang

    Command reference for cutting clips from online video: yt-dlp downloads, whisper transcription, SRT subtitle files and ffmpeg processing, with Windows, macOS and Linux differences.

    18k GitHub stars~4.1k tokensUpdated 3 mo ago
    Auto-check: warnings

  • Forecasting Expert Knowledge

    RightNow-AI/openfang

    Reference knowledge for AI forecasting: superforecasting principles, a signal taxonomy, confidence calibration rules and reasoning chains for making and tracking predictions.

    18k GitHub stars~2.5k tokensUpdated 3 mo ago
    Auto-check passed

  • Deep Research Methodology

    RightNow-AI/openfang

    Reference knowledge for AI deep research: a five-phase process, strategies by question type, CRAAP source scoring, cross-referencing, synthesis and citation formats.

    18k GitHub stars~2.6k tokensUpdated 3 mo ago
    Auto-check passed

Works with

Categories

Questions about Data Pipeline

What does Data Pipeline do?

Data pipeline expert for ETL, Apache Spark, Airflow, dbt, and data quality. Data Pipeline is an agent skill from RightNow-AI/openfang.

When should I use Data Pipeline?

Data Pipeline fits situations like: tasks that involve Data pipelines and ETL.

How do I install Data Pipeline in Claude Code?

Run `npx skills add RightNow-AI/openfang --skill data-pipeline -a claude-code`. Or copy the skill folder (crates/openfang-skills/bundled/data-pipeline in RightNow-AI/openfang) into .claude/skills/data-pipeline in your project. Claude Code loads it when a task matches its description.

How do I install Data Pipeline in Codex?

Run `npx skills add RightNow-AI/openfang --skill data-pipeline -a codex`. Or copy the skill folder (crates/openfang-skills/bundled/data-pipeline in RightNow-AI/openfang) into .agents/skills/data-pipeline in your project. Codex loads it when a task matches its description.

Can I use Data Pipeline in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add RightNow-AI/openfang --skill data-pipeline -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/data-pipeline, .gemini/skills/data-pipeline, .github/skills/data-pipeline and .opencode/skills/data-pipeline in your project.

What does Data Pipeline need to run?

SKILL.md names no scripts, command-line tools or credentials: Data Pipeline is instructions for the agent only.

Does Data Pipeline access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Data Pipeline safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Data Pipeline use?

Data Pipeline is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Data Pipeline use?

About 847 tokens (SKILL.md is roughly 3.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Data Pipeline?

Skills that share tags, products or a category with Data Pipeline: Data Engineer (davila7/claude-code-templates, 32k stars), Migrating Dagster To Airflow (astronomer/agents, 451 stars), Deploying Airflow (astronomer/agents, 451 stars) and Senior Data Engineer (benchflow-ai/skillsbench, 1.8k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Data Pipeline?

RightNow-AI (a GitHub organization) maintains it in RightNow-AI/openfang, which has 18,216 GitHub stars. The repository holds 68 skills in this directory. The repository was last updated on July 2, 2026.

Source: RightNow-AI/openfang on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.