New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Airflow DAG Patterns

by wshobson in wshobson/agents

Patterns for writing production-ready Apache Airflow DAGs: task dependencies, custom operators and sensors, local testing, and rules for what to avoid.

MITAuto-check passedData & Analytics

Install Airflow DAG Patterns

skills CLI
$ npx skills add wshobson/agents --skill airflow-dag-patterns -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install wshobson/agents airflow-dag-patterns --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/wshobson/agents.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/data-engineering/skills/airflow-dag-patterns .claude/skills/airflow-dag-patterns && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
airflow-dag-patterns
GitHub stars
40k
Used in
9 other repos
Token cost
~784 tokens
SKILL.md length
180 words
Files
2 (incl. references)
Skills in repo
142
Repo updated
First seen
Licence
MIT

At a glance

Patterns for writing production-ready Apache Airflow DAGs: task dependencies, custom operators and sensors, local testing, and rules for what to avoid.

  • Works in 2 steps: DAG Design Principles → Task Dependencies
  • Designing a new Airflow DAG and its task dependencies
  • SKILL.md covers When to Use This Skill, Core Concepts, Quick Start and Detailed patterns and worked…, plus 1 more section
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

This skill collects patterns for writing Apache Airflow DAGs that are ready for production. It lays out four design principles, idempotent, atomic, incremental and observable, and shows how to wire task dependencies with the bitshift syntax, with a quick-start DAG file as a starting point. A reference file, `references/details.md`, holds longer pattern write-ups and worked examples for cases the main page does not cover.

A do and don't list sets the house rules: prefer the TaskFlow API, set timeouts, run sensors in reschedule mode, test DAGs and keep tasks safe to retry. It advises against `depends_on_past=True`, hardcoded dates, global state, blind catchup choices and heavy logic inside the DAG file. The topics it names include custom operators and sensors, local DAG testing, production setup and debugging failed runs.

When your agent uses it

  • Designing a new Airflow DAG and its task dependencies
  • Writing a custom operator or sensor
  • Testing DAGs locally before deploying them
  • Debugging a DAG run that failed

Example prompts

  • “Create an Airflow DAG that pulls yesterday's sales files and loads them into the warehouse each morning.”
  • “Convert this cron-driven script into a DAG with retries and timeouts.”
  • “This sensor holds a worker slot the whole time. Switch it to reschedule mode.”
  • “Write tests that check every DAG in dags/ imports without errors.”

Requirements

  • An Apache Airflow project or installation

Workflow steps

2 steps, taken from the step headings in SKILL.md.

  1. DAG Design Principles
  2. Task Dependencies

What it can do on your machine

Read from SKILL.md and the folder at commit 46891e7. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Airflow DAG Patterns loads about 784 tokens when it runs, and up to ~3.6k if it reads all its reference files. Until then it costs about 53 tokens; SKILL.md has 180 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~53
When it runs · the whole SKILL.md, loaded when a task matches
~784
With references · SKILL.md plus every file in references/, read only if the agent opens them
~3.6k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from wshobson/agents at commit 46891e7, republished under its MIT licence (© wshobson). 180 words, ~784 tokens.

Download SKILL.mdSave it as .claude/skills/airflow-dag-patterns/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
airflow-dag-patterns
description
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

  • Creating data pipeline orchestration with Airflow
  • Designing DAG structures and dependencies
  • Implementing custom operators and sensors
  • Testing Airflow DAGs locally
  • Setting up Airflow in production
  • Debugging failed DAG runs

Core Concepts

1. DAG Design Principles
PrincipleDescription
IdempotentRunning twice produces same result
AtomicTasks succeed or fail completely
IncrementalProcess only new/changed data
ObservableLogs, metrics, alerts at every step
2. Task Dependencies
python
# Linear
task1 >> task2 >> task3

# Fan-out
task1 >> [task2, task3, task4]

# Fan-in
[task1, task2, task3] >> task4

# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4

Quick Start

python
# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(hours=1),
}

with DAG(
    dag_id='example_etl',
    default_args=default_args,
    description='Example ETL pipeline',
    schedule='0 6 * * *',  # Daily at 6 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['etl', 'example'],
    max_active_runs=1,
) as dag:

    start = EmptyOperator(task_id='start')

    def extract_data(**context):
        execution_date = context['ds']
        # Extract logic here
        return {'records': 1000}

    extract = PythonOperator(
        task_id='extract',
        python_callable=extract_data,
    )

    end = EmptyOperator(task_id='end')

    start >> extract >> end

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's
  • Use TaskFlow API - Cleaner code, automatic XCom
  • Set timeouts - Prevent zombie tasks
  • Use mode='reschedule' - For sensors, free up workers
  • Test DAGs - Unit tests and integration tests
  • Idempotent tasks - Safe to retry
Don'ts
  • Don't use depends_on_past=True - Creates bottlenecks
  • Don't hardcode dates - Use {{ ds }} macros
  • Don't use global state - Tasks should be stateless
  • Don't skip catchup blindly - Understand implications
  • Don't put heavy logic in DAG file - Import from modules

© wshobson, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (references) in plugins/data-engineering/skills/airflow-dag-patterns of wshobson/agents.

  • SKILL.md
  • references/details.md

Open the folder on GitHubat commit 46891e7

Used in 9 other repositories

We found 25 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 9 other GitHub owners. This page covers the copy in wshobson/agents, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Airflow DAG Patterns next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Airflow DAG Patterns compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Airflow DAG Patterns this skillwshobson/agents40k9 repos~784Automated safety check: PassMIT
Senior Data Engineerbenchflow-ai/skillsbench1.8k—~5.9kAutomated safety check: PassMIT
Version Bumpergodatadriven/whirl205—~1.2kAutomated safety check: PassApache-2.0
Senior Data Engineeralirezarezvani/claude-skills28k3 repos~1.4kAutomated safety check: PassMIT
Authoring Go SDK Tasksastronomer/agents451—~2.1kAutomated safety check: PassApache-2.0
Senior Data Engineerdavila7/claude-code-templates32k1 repos~1.4kAutomated safety check: PassMIT

Similar skills

  • Senior Data Engineer

    benchflow-ai/skillsbench

    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure.

    1.8k GitHub stars~5.9k tokensUpdated 2 mo ago
    Data & AnalyticsAuto-check passed

  • Version Bumper

    godatadriven/whirl

    Bump the Airflow or Python version across all project files.

    205 GitHub stars~1.2k tokensUpdated 7 days ago
    Data & AnalyticsAuto-check passed

  • Senior Data Engineer

    alirezarezvani/claude-skills

    Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

    28k GitHub starsUsed in 3 repos~1.4k tokens
    Data & AnalyticsAuto-check passed

  • Authoring Go SDK Tasks

    astronomer/agents

    Writes Airflow task logic in Go using the Airflow Go SDK. An agent skill from astronomer/agents.

    451 GitHub stars~2.1k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Senior Data Engineer

    davila7/claude-code-templates

    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

    32k GitHub starsUsed in 1 repo~1.4k tokens
    Data & AnalyticsAuto-check passed

  • Authoring Java SDK Tasks

    astronomer/agents

    Writes Airflow task logic in Java, Kotlin, or any JVM language using the Airflow Java SDK.

    451 GitHub stars~2.6k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

More from wshobson/agents

All 142 skills in this repo

  • Cloud Cost Optimization

    wshobson/agents

    Cuts cloud spend across AWS, Azure, GCP and OCI with cost tagging, rightsizing, commitment and spot pricing models, and architecture changes.

    40k GitHub starsUsed in 14 repos~1.7k tokens
    Auto-check passed

  • Billing Automation

    wshobson/agents

    Covers building subscription billing: billing cycles, subscription states, invoice generation, proration, tax handling and dunning for failed payments.

    40k GitHub starsUsed in 13 repos~473 tokens
    Auto-check passed

  • Python Performance Optimization

    wshobson/agents

    Profiles slow Python code with cProfile and memory profilers, then applies targeted fixes for CPU, memory, I/O and query bottlenecks.

    40k GitHub starsUsed in 13 repos~814 tokens
    Auto-check passed

  • Bats Shell Testing Patterns

    wshobson/agents

    Writes unit tests for shell scripts with Bats: error-condition tests, fixtures and mocks, cross-shell checks, parallel runs, helper files and CI integration.

    40k GitHub starsUsed in 12 repos~1.3k tokens
    Auto-check passed

  • Distributed Tracing

    wshobson/agents

    Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks.

    40k GitHub starsUsed in 12 repos~527 tokens
    Auto-check passed

  • Prompt Engineering Patterns

    wshobson/agents

    Reference for designing and tuning production LLM prompts: few-shot examples, chain-of-thought, structured outputs, templates and system prompts.

    40k GitHub stars~1.3k tokensUpdated 4 days ago
    Auto-check passed

Works with

Categories

Questions about Airflow DAG Patterns

What does Airflow DAG Patterns do?

Patterns for writing production-ready Apache Airflow DAGs: task dependencies, custom operators and sensors, local testing, and rules for what to avoid. This skill collects patterns for writing Apache Airflow DAGs that are ready for production. It lays out four design principles, idempotent, atomic, incremental and observable, and shows how to wire task dependencies with the bitshift syntax, with a quick-start DAG file as a starting point.

When should I use Airflow DAG Patterns?

Airflow DAG Patterns fits situations like: designing a new Airflow DAG and its task dependencies; writing a custom operator or sensor; testing DAGs locally before deploying them; debugging a DAG run that failed.

How do I install Airflow DAG Patterns in Claude Code?

Run `npx skills add wshobson/agents --skill airflow-dag-patterns -a claude-code`. Or copy the skill folder (plugins/data-engineering/skills/airflow-dag-patterns in wshobson/agents) into .claude/skills/airflow-dag-patterns in your project. Claude Code loads it when a task matches its description.

How do I install Airflow DAG Patterns in Codex?

Run `npx skills add wshobson/agents --skill airflow-dag-patterns -a codex`. Or copy the skill folder (plugins/data-engineering/skills/airflow-dag-patterns in wshobson/agents) into .agents/skills/airflow-dag-patterns in your project. Codex loads it when a task matches its description.

Can I use Airflow DAG Patterns in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add wshobson/agents --skill airflow-dag-patterns -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/airflow-dag-patterns, .gemini/skills/airflow-dag-patterns, .github/skills/airflow-dag-patterns and .opencode/skills/airflow-dag-patterns in your project.

What does Airflow DAG Patterns need to run?

SKILL.md names no scripts, command-line tools or credentials: Airflow DAG Patterns is instructions for the agent only. Our summary lists: An Apache Airflow project or installation.

Does Airflow DAG Patterns access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Airflow DAG Patterns safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Airflow DAG Patterns use?

Airflow DAG Patterns is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Airflow DAG Patterns use?

About 784 tokens (SKILL.md is roughly 3.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 2.9k tokens, read only when the agent opens those files.

What are the alternatives to Airflow DAG Patterns?

Skills that share tags, products or a category with Airflow DAG Patterns: Senior Data Engineer (benchflow-ai/skillsbench, 1.8k stars), Version Bumper (godatadriven/whirl, 205 stars), Senior Data Engineer (alirezarezvani/claude-skills, 28k stars) and Authoring Go SDK Tasks (astronomer/agents, 451 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Airflow DAG Patterns?

wshobson (a GitHub user) maintains it in wshobson/agents, which has 40,305 GitHub stars. The repository holds 142 skills in this directory. The repository was last updated on October 5, 2026.

Source: wshobson/agents on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.