New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

ML Pipeline Workflow

by wshobson in wshobson/agents

Guides an agent through designing an MLOps pipeline that covers data preparation, training, validation and deployment, with DAG orchestration and reference guides.

MITAuto-check passedDevOps & Cloud

Install ML Pipeline Workflow

skills CLI
$ npx skills add wshobson/agents --skill ml-pipeline-workflow -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install wshobson/agents ml-pipeline-workflow --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/wshobson/agents.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/machine-learning-ops/skills/ml-pipeline-workflow .claude/skills/ml-pipeline-workflow && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
ml-pipeline-workflow
GitHub stars
40k
Used in
12 other repos
Token cost
~1.8k tokens
SKILL.md length
645 words
Files
1
Skills in repo
142
Repo updated
First seen
Licence
MIT

At a glance

Guides an agent through designing an MLOps pipeline that covers data preparation, training, validation and deployment, with DAG orchestration and reference guides.

  • Works in 5 steps: Pipeline Architecture → Data Preparation → Model Training → …
  • Building a new ML pipeline from data ingestion through deployment
  • SKILL.md covers Overview, When to Use This Skill, What This Skill Provides and Usage Patterns, plus 7 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

The skill lays out the stages of a production machine learning pipeline: data ingestion and preparation, model training, validation, deployment and monitoring. For each stage it names the topics to cover, such as data quality checks, feature engineering, hyperparameter management, experiment tracking, canary and blue-green rollouts, and rollback mechanisms.

It points to four reference guides on data preparation, training, validation and deployment, plus an assets folder with a DAG template, a training configuration template and a pre-deployment validation checklist. Airflow, Dagster and Kubeflow are named as orchestration options, and a short Python sketch shows how to define the pipeline stages.

When your agent uses it

  • Building a new ML pipeline from data ingestion through deployment
  • Designing DAG-based orchestration for training and validation jobs
  • Setting up reproducible training workflows with experiment tracking
  • Planning a canary or blue-green rollout for a trained model

Example prompts

  • “Design an end-to-end training and deployment pipeline for our churn model.”
  • “Set up an Airflow DAG that validates data, trains, and gates deployment on metrics.”
  • “Draft a pre-deployment validation checklist for the fraud detection model.”

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

  1. Pipeline Architecture
  2. Data Preparation
  3. Model Training
  4. Model Validation
  5. Deployment Automation

What it can do on your machine

Read from SKILL.md and the folder at commit 46891e7. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are python and yaml).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

ML Pipeline Workflow loads about 1.8k tokens when it runs. Until then it costs about 64 tokens; SKILL.md has 645 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~64
When it runs · the whole SKILL.md, loaded when a task matches
~1.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from wshobson/agents at commit 46891e7, republished under its MIT licence (© wshobson). 645 words, ~1,770 tokens.

Download SKILL.mdSave it as .claude/skills/ml-pipeline-workflow/SKILL.md (or your agent's skills folder).
name
ml-pipeline-workflow
description
Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.

ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill

  • Building new ML pipelines from scratch
  • Designing workflow orchestration for ML systems
  • Implementing data → model → deployment automation
  • Setting up reproducible training workflows
  • Creating DAG-based ML orchestration
  • Integrating ML components into production systems

What This Skill Provides

Core Capabilities

  1. Pipeline Architecture

    • End-to-end workflow design
    • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
    • Component dependencies and data flow
    • Error handling and retry strategies

  2. Data Preparation

    • Data validation and quality checks
    • Feature engineering pipelines
    • Data versioning and lineage
    • Train/validation/test splitting strategies

  3. Model Training

    • Training job orchestration
    • Hyperparameter management
    • Experiment tracking integration
    • Distributed training patterns

  4. Model Validation

    • Validation frameworks and metrics
    • A/B testing infrastructure
    • Performance regression detection
    • Model comparison workflows

  5. Deployment Automation

    • Model serving patterns
    • Canary deployments
    • Blue-green deployment strategies
    • Rollback mechanisms
Reference Documentation

See the references/ directory for detailed guides:

  • data-preparation.md - Data cleaning, validation, and feature engineering
  • model-training.md - Training workflows and best practices
  • model-validation.md - Validation strategies and metrics
  • model-deployment.md - Deployment patterns and serving architectures
Assets and Templates

The assets/ directory contains:

  • pipeline-dag.yaml.template - DAG template for workflow orchestration
  • training-config.yaml - Training configuration template
  • validation-checklist.md - Pre-deployment validation checklist

Usage Patterns

Basic Pipeline Setup
python
# 1. Define pipeline stages
stages = [
    "data_ingestion",
    "data_validation",
    "feature_engineering",
    "model_training",
    "model_validation",
    "model_deployment"
]

# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example
Production Workflow

  1. Data Preparation Phase

    • Ingest raw data from sources
    • Run data quality checks
    • Apply feature transformations
    • Version processed datasets

  2. Training Phase

    • Load versioned training data
    • Execute training jobs
    • Track experiments and metrics
    • Save trained models

  3. Validation Phase

    • Run validation test suite
    • Compare against baseline
    • Generate performance reports
    • Approve for deployment

  4. Deployment Phase

    • Package model artifacts
    • Deploy to serving infrastructure
    • Configure monitoring
    • Validate production traffic

Best Practices

Pipeline Design
  • Modularity: Each stage should be independently testable
  • Idempotency: Re-running stages should be safe
  • Observability: Log metrics at every stage
  • Versioning: Track data, code, and model versions
  • Failure Handling: Implement retry logic and alerting
Data Management
  • Use data validation libraries (Great Expectations, TFX)
  • Version datasets with DVC or similar tools
  • Document feature engineering transformations
  • Maintain data lineage tracking
Model Operations
  • Separate training and serving infrastructure
  • Use model registries (MLflow, Weights & Biases)
  • Implement gradual rollouts for new models
  • Monitor model performance drift
  • Maintain rollback capabilities
Show full SKILL.md (267 more words)Show less
Deployment Strategies
  • Start with shadow deployments
  • Use canary releases for validation
  • Implement A/B testing infrastructure
  • Set up automated rollback triggers
  • Monitor latency and throughput

Integration Points

Orchestration Tools
  • Apache Airflow: DAG-based workflow orchestration
  • Dagster: Asset-based pipeline orchestration
  • Kubeflow Pipelines: Kubernetes-native ML workflows
  • Prefect: Modern dataflow automation
Experiment Tracking
  • MLflow for experiment tracking and model registry
  • Weights & Biases for visualization and collaboration
  • TensorBoard for training metrics
Deployment Platforms
  • AWS SageMaker for managed ML infrastructure
  • Google Vertex AI for GCP deployments
  • Azure ML for Azure cloud
  • OCI Data Science for Oracle Cloud Infrastructure deployments
  • Kubernetes + KServe for cloud-agnostic serving

Progressive Disclosure

Start with the basics and gradually add complexity:

  1. Level 1: Simple linear pipeline (data → train → deploy)
  2. Level 2: Add validation and monitoring stages
  3. Level 3: Implement hyperparameter tuning
  4. Level 4: Add A/B testing and gradual rollouts
  5. Level 5: Multi-model pipelines with ensemble strategies

Common Patterns

Batch Training Pipeline
yaml
# See assets/pipeline-dag.yaml.template
stages:
  - name: data_preparation
    dependencies: []
  - name: model_training
    dependencies: [data_preparation]
  - name: model_evaluation
    dependencies: [model_training]
  - name: model_deployment
    dependencies: [model_evaluation]
Real-time Feature Pipeline
python
# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.md
Continuous Training
python
# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.md

Troubleshooting

Common Issues
  • Pipeline failures: Check dependencies and data availability
  • Training instability: Review hyperparameters and data quality
  • Deployment issues: Validate model artifacts and serving config
  • Performance degradation: Monitor data drift and model metrics
Debugging Steps
  1. Check pipeline logs for each stage
  2. Validate input/output data at boundaries
  3. Test components in isolation
  4. Review experiment tracking metrics
  5. Inspect model artifacts and metadata

Next Steps

After setting up your pipeline:

  1. Explore hyperparameter-tuning skill for optimization
  2. Learn experiment-tracking-setup for MLflow/W&B
  3. Review model-deployment-patterns for serving strategies
  4. Implement monitoring with observability tools
  • experiment-tracking-setup: MLflow and Weights & Biases integration
  • hyperparameter-tuning: Automated hyperparameter optimization
  • model-deployment-patterns: Advanced deployment strategies

© wshobson, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in plugins/machine-learning-ops/skills/ml-pipeline-workflow of wshobson/agents.

Open the folder on GitHubat commit 46891e7

Used in 12 other repositories

We found 26 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 12 other GitHub owners. This page covers the copy in wshobson/agents, which our catalogue first saw on October 7, 2026.

Compare with similar skills

ML Pipeline Workflow next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

ML Pipeline Workflow compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
ML Pipeline Workflow this skillwshobson/agents40k12 repos~1.8kAutomated safety check: PassMIT
ML Pipeline ExpertJeffallan/claude-skills12k—~1.9kAutomated safety check: PassMIT
AI Data Engineeringancoleman/ai-design-components526—~3.5kAutomated safety check: PassMIT
ML Pipeline Automationsecondsky/claude-skills227—~3.2kAutomated safety check: PassMIT
Upgrading Mwaa Environmentsaws/agent-toolkit-for-aws2.8k—~7.3kAutomated safety check: PassApache-2.0
Senior Data Scientistborghei/Claude-Skills886—~1.7kAutomated safety check: PassMIT

Similar skills

  • ML Pipeline Expert

    Jeffallan/claude-skills

    Designs ML pipeline infrastructure: experiment tracking with MLflow or Weights & Biases, Kubeflow and Airflow orchestration, Feast feature stores and model validation gates.

    12k GitHub stars~1.9k tokensUpdated 6 days ago
    DevOps & CloudAuto-check passed

  • AI Data Engineering

    ancoleman/ai-design-components

    Data pipelines, feature stores, and embedding generation for AI/ML systems.

    526 GitHub stars~3.5k tokensUpdated 10 mo ago
    Data & AnalyticsAuto-check passed

  • ML Pipeline Automation

    secondsky/claude-skills

    Automate ML workflows with Airflow, Kubeflow, MLflow. An agent skill from secondsky/claude-skills.

    227 GitHub stars~3.2k tokensUpdated 11 days ago
    DevOps & CloudAuto-check passed

  • Upgrading Mwaa Environments

    aws/agent-toolkit-for-aws

    Official

    Upgrades an MWAA environment to a newer Airflow version — within 2.x, within 3.x, or across the 2.x-to-3.x boundary.

    2.8k GitHub stars~7.3k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Senior Data Scientist

    borghei/Claude-Skills

    A skill your agent uses when the user asks to "design an experiment", "build a predictive model", "run A/B test analysis", "perform causal inference", "engineer features", "evaluate model…

    886 GitHub stars~1.7k tokensUpdated 2 days ago
    Data & AnalyticsAuto-check passed

  • Model Deployment

    secondsky/claude-skills

    Deploy ML models with FastAPI, Docker, Kubernetes. An agent skill from secondsky/claude-skills.

    227 GitHub stars~2.4k tokensUpdated 11 days ago
    DevOps & CloudAuto-check passed

More from wshobson/agents

All 142 skills in this repo

  • Cloud Cost Optimization

    wshobson/agents

    Cuts cloud spend across AWS, Azure, GCP and OCI with cost tagging, rightsizing, commitment and spot pricing models, and architecture changes.

    40k GitHub starsUsed in 14 repos~1.7k tokens
    Auto-check passed

  • Billing Automation

    wshobson/agents

    Covers building subscription billing: billing cycles, subscription states, invoice generation, proration, tax handling and dunning for failed payments.

    40k GitHub starsUsed in 13 repos~473 tokens
    Auto-check passed

  • Python Performance Optimization

    wshobson/agents

    Profiles slow Python code with cProfile and memory profilers, then applies targeted fixes for CPU, memory, I/O and query bottlenecks.

    40k GitHub starsUsed in 13 repos~814 tokens
    Auto-check passed

  • Bats Shell Testing Patterns

    wshobson/agents

    Writes unit tests for shell scripts with Bats: error-condition tests, fixtures and mocks, cross-shell checks, parallel runs, helper files and CI integration.

    40k GitHub starsUsed in 12 repos~1.3k tokens
    Auto-check passed

  • Distributed Tracing

    wshobson/agents

    Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks.

    40k GitHub starsUsed in 12 repos~527 tokens
    Auto-check passed

  • Prompt Engineering Patterns

    wshobson/agents

    Reference for designing and tuning production LLM prompts: few-shot examples, chain-of-thought, structured outputs, templates and system prompts.

    40k GitHub stars~1.3k tokensUpdated 5 days ago
    Auto-check passed

Works with

Categories

Questions about ML Pipeline Workflow

What does ML Pipeline Workflow do?

Guides an agent through designing an MLOps pipeline that covers data preparation, training, validation and deployment, with DAG orchestration and reference guides. The skill lays out the stages of a production machine learning pipeline: data ingestion and preparation, model training, validation, deployment and monitoring. For each stage it names the topics to cover, such as data quality checks, feature engineering, hyperparameter management, experiment tracking, canary and blue-green rollouts, and rollback mechanisms.

When should I use ML Pipeline Workflow?

ML Pipeline Workflow fits situations like: building a new ML pipeline from data ingestion through deployment; designing DAG-based orchestration for training and validation jobs; setting up reproducible training workflows with experiment tracking; planning a canary or blue-green rollout for a trained model.

How do I install ML Pipeline Workflow in Claude Code?

Run `npx skills add wshobson/agents --skill ml-pipeline-workflow -a claude-code`. Or copy the skill folder (plugins/machine-learning-ops/skills/ml-pipeline-workflow in wshobson/agents) into .claude/skills/ml-pipeline-workflow in your project. Claude Code loads it when a task matches its description.

How do I install ML Pipeline Workflow in Codex?

Run `npx skills add wshobson/agents --skill ml-pipeline-workflow -a codex`. Or copy the skill folder (plugins/machine-learning-ops/skills/ml-pipeline-workflow in wshobson/agents) into .agents/skills/ml-pipeline-workflow in your project. Codex loads it when a task matches its description.

Can I use ML Pipeline Workflow in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add wshobson/agents --skill ml-pipeline-workflow -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ml-pipeline-workflow, .gemini/skills/ml-pipeline-workflow, .github/skills/ml-pipeline-workflow and .opencode/skills/ml-pipeline-workflow in your project.

What does ML Pipeline Workflow need to run?

SKILL.md names no scripts, command-line tools or credentials: ML Pipeline Workflow is instructions for the agent only.

Does ML Pipeline Workflow access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is ML Pipeline Workflow safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does ML Pipeline Workflow use?

ML Pipeline Workflow is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does ML Pipeline Workflow use?

About 1.8k tokens (SKILL.md is roughly 7.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to ML Pipeline Workflow?

Skills that share tags, products or a category with ML Pipeline Workflow: ML Pipeline Expert (Jeffallan/claude-skills, 12k stars), AI Data Engineering (ancoleman/ai-design-components, 526 stars), ML Pipeline Automation (secondsky/claude-skills, 227 stars) and Upgrading Mwaa Environments (aws/agent-toolkit-for-aws, 2.8k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains ML Pipeline Workflow?

wshobson (a GitHub user) maintains it in wshobson/agents, which has 40,305 GitHub stars. The repository holds 142 skills in this directory. The repository was last updated on October 5, 2026.

Source: wshobson/agents on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.