Agent skill

Preprocessing Data With Automated Pipelines

by jeremylongshore in jeremylongshore/tons-of-skills-marketplace

Process automate data cleaning, transformation, and validation for ML tasks.

MITAuto-check passedData & Analytics

Install Preprocessing Data With Automated Pipelines

skills CLI
$ npx skills add jeremylongshore/tons-of-skills-marketplace --skill preprocessing-data-with-automated-pipelines -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install jeremylongshore/tons-of-skills-marketplace preprocessing-data-with-automated-pipelines --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/jeremylongshore/tons-of-skills-marketplace.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/.curated/preprocessing-data-with-automated-pipelines .claude/skills/preprocessing-data-with-automated-pipelines && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
preprocessing-data-with-automated-pipelines
GitHub stars
2.8k
Token cost
~1k tokens
SKILL.md length
490 words
Files
9 (incl. scripts, references, assets)
Skills in repo
3,342
Repo updated
First seen
Licence
MIT

At a glance

Process automate data cleaning, transformation, and validation for ML tasks.

  • Works in 4 steps: Analyze Requirements: Claude analyzes… → Generate Pipeline Code: Based on the… → Execute Pipeline: The generated code is… → …
  • Requesting preprocess data
  • SKILL.md covers Overview, How It Works, When to Use This Skill and Examples, plus 7 more sections
  • Runs Python scripts from its folder

What it does

Preprocessing Data With Automated Pipelines is an agent skill from jeremylongshore/tons-of-skills-marketplace. Process automate data cleaning, transformation, and validation for ML tasks. Use when requesting "preprocess data", "clean data", "ETL pipeline", or "data transformation". Trigger with relevant phrases based on skill purpose.

Its SKILL.md is about 1k tokens, which your agent loads only when the skill is triggered. The skill folder holds 11 other files, including scripts, reference files and assets (for example `assets/README.md`, `references/README.md` and `scripts/README.md`). Compatibility notes: Designed for Claude Code

It sits in Data & Analytics, covering Data cleaning and Data pipelines and ETL. It works with Python. The repository describes itself as: Model-agnostic agent-skills platform with a harness-free canonical layer, verified adapters, and the ccpi package manager. Explore at tonsofskills.com. The licence is MIT.

When your agent uses it

  • Requesting preprocess data
  • Data transformation
  • With relevant phrases based on skill purpose

Example prompts

  • “preprocess data”
  • “clean data”
  • “ETL pipeline”
  • “/preprocessing-data-with-automated-pipelines”

Requirements

  • Python 3
  • Compatibility (from SKILL.md): Designed for Claude Code
  • Pre-approved tools (allowed-tools): Read, Write, Edit, Grep, Glob, Bash(cmd:*)

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

  1. Analyze Requirements: Claude analyzes the user's request to understand the specific data preprocessing needs, including data sources…
  2. Generate Pipeline Code: Based on the requirements, Claude generates Python code for an automated data preprocessing pipeline using…
  3. Execute Pipeline: The generated code is executed, performing the data preprocessing steps.
  4. Provide Metrics and Insights: Claude provides performance metrics and insights about the pipeline's execution, including data quality…

What it can do on your machine

Read from SKILL.md and the folder at commit cfae287. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves these tools, so the agent can use them without asking each time:

    • Read
    • Write
    • Edit
    • Grep
    • Glob
    • Bash(cmd:*)

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 5 files in scripts/ (Python), which the agent can run.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Designed for Claude Code

    From compatibility in the SKILL.md frontmatter.

Context cost

Preprocessing Data With Automated Pipelines loads about 1k tokens when it runs, and up to ~1k if it reads all its reference files. Until then it costs about 67 tokens; SKILL.md has 490 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~67
When it runs · the whole SKILL.md, loaded when a task matches
~1k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from jeremylongshore/tons-of-skills-marketplace at commit cfae287, republished under its MIT licence (© jeremylongshore). 490 words, ~1,022 tokens.

Download SKILL.mdSave it as .claude/skills/preprocessing-data-with-automated-pipelines/SKILL.md (or your agent's skills folder). This skill also uses 8 other files; get the full folder from GitHub.
name
preprocessing-data-with-automated-pipelines
description
Process automate data cleaning, transformation, and validation for ML tasks. Use when requesting "preprocess data", "clean data", "ETL pipeline", or "data transformation". Trigger with relevant phrases based on skill purpose.
allowed-tools
Read, Write, Edit, Grep, Glob, Bash(cmd:*)
compatibility
Designed for Claude Code
version
1.23.0
author
Jeremy Longshore <jeremy@intentsolutions.io>
license
MIT
tags
ai, ml, etl

Data Preprocessing Pipeline

Construct and execute automated data preprocessing pipelines for cleaning, transforming, and validating ML-ready datasets.

Overview

construct and execute automated data preprocessing pipelines, ensuring data quality and readiness for machine learning. It streamlines the data preparation process by automating common tasks such as data cleaning, transformation, and validation.

How It Works

  1. Analyze Requirements: Claude analyzes the user's request to understand the specific data preprocessing needs, including data sources, target format, and desired transformations.
  2. Generate Pipeline Code: Based on the requirements, Claude generates Python code for an automated data preprocessing pipeline using relevant libraries and best practices. This includes data validation and error handling.
  3. Execute Pipeline: The generated code is executed, performing the data preprocessing steps.
  4. Provide Metrics and Insights: Claude provides performance metrics and insights about the pipeline's execution, including data quality reports and potential issues encountered.

When to Use This Skill

This skill activates when you need to:

  • Prepare raw data for machine learning models.
  • Automate data cleaning and transformation processes.
  • Implement a robust ETL (Extract, Transform, Load) pipeline.

Examples

Example 1: Cleaning Customer Data

User request: "Preprocess the customer data from the CSV file to remove duplicates and handle missing values."

The skill will:

  1. Generate a Python script to read the CSV file, remove duplicate entries, and impute missing values using appropriate techniques (e.g., mean imputation).
  2. Execute the script and provide a summary of the changes made, including the number of duplicates removed and the number of missing values imputed.
Show full SKILL.md (240 more words)Show less
Example 2: Transforming Sensor Data

User request: "Create an ETL pipeline to transform the sensor data from the database into a format suitable for time series analysis."

The skill will:

  1. Generate a Python script to extract sensor data from the database, transform it into a time series format (e.g., resampling to a fixed frequency), and load it into a suitable storage location.
  2. Execute the script and provide performance metrics, such as the time taken for each step of the pipeline and the size of the transformed data.

Best Practices

  • Data Validation: Always include data validation steps to ensure data quality and catch potential errors early in the pipeline.
  • Error Handling: Implement robust error handling to gracefully handle unexpected issues during pipeline execution.
  • Performance Optimization: Optimize the pipeline for performance by using efficient algorithms and data structures.

Integration

This skill can be integrated with other Claude Code skills for data analysis, model training, and deployment. It provides a standardized way to prepare data for these tasks, ensuring consistency and reliability.

Prerequisites

  • Appropriate file access permissions
  • Required dependencies installed

Instructions

  1. Invoke this skill when the trigger conditions are met
  2. Provide necessary context and parameters
  3. Review the generated output
  4. Apply modifications as needed

Output

The skill produces structured output relevant to the task.

Error Handling

  • Invalid input: Prompts for correction
  • Missing dependencies: Lists required components
  • Permission errors: Suggests remediation steps

Resources

  • Project documentation
  • Related skills and commands

© jeremylongshore, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 8 other files (scripts, references, assets) in skills/.curated/preprocessing-data-with-automated-pipelines of jeremylongshore/tons-of-skills-marketplace.

  • SKILL.md
  • assets/README.md
  • assets/example_data.csv
  • references/README.md
  • scripts/README.md
  • scripts/handle_errors.py
  • scripts/pipeline.py
  • scripts/transform_data.py
  • scripts/validate_data.py

Open the folder on GitHubat commit cfae287

Compare with similar skills

Preprocessing Data With Automated Pipelines next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Preprocessing Data With Automated Pipelines compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Preprocessing Data With Automated Pipelines this skilljeremylongshore/tons-of-skills-marketplace2.8k—~1kAutomated safety check: PassMIT
Credit Risk Data Cleaninggithub/awesome-copilot40k1 repos~1.5kAutomated safety check: PassMIT
Authoritative Data Harvesteryushui2022/MathModel-Skill4541 repos~1.1kAutomated safety check: PassMIT
Data Quality Frameworkswshobson/agents40k11 repos~1.1kAutomated safety check: PassMIT
Bio Batch ProcessingGPTomics/bioSkills1.2k1 repos~3kAutomated safety check: PassMIT
Data Cleaningmagnus919/agent-skills115—~2.1kAutomated safety check: PassMIT

Similar skills

  • Credit Risk Data Cleaning

    github/awesome-copilot

    Official

    Cleans raw credit data and screens variables before loan modeling, dropping unstable, noisy or redundant features and writing an Excel report of every step.

    40k GitHub starsUsed in 1 repo~1.5k tokens
    Data & AnalyticsAuto-check passed
  • Authoritative Data Harvester

    yushui2022/MathModel-Skill

    Finds authoritative public data sources for modeling tasks, prefers official APIs and bulk downloads, and outputs a reproducible fetch and cleaning plan with citations.

    454 GitHub starsUsed in 1 repo~1.1k tokens
    Data & AnalyticsAuto-check passed
  • Sets up data quality checks with Great Expectations, dbt tests and data contracts, with checkpoints and pass-fail reports for pipelines.

    40k GitHub starsUsed in 11 repos~1.1k tokens
    Data & AnalyticsAuto-check passed
  • Bio Batch Processing

    GPTomics/bioSkills

    Process many sequence files in batch (count, merge, split, convert, summarize) with memory-safe streaming and on-disk indexing using Biopython, pysam, or pyfastx.

    1.2k GitHub starsUsed in 1 repo~3k tokens
    Data & AnalyticsAuto-check passed
  • Data Cleaning

    magnus919/agent-skills

    Clean, profile, validate, reshape, and document messy tabular, text, JSON, and relational data through an evidence-first, reproducible workflow.

    115 GitHub stars~2.1k tokensUpdated today
    Data & AnalyticsAuto-check passed
  • Crawl4AI Web Scraping

    smallnest/goclaw

    Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.

    599 GitHub starsUsed in 1 repo~2.5k tokens
    Data & AnalyticsAuto-check passed

More from jeremylongshore/tons-of-skills-marketplace

All 3,342 skills in this repo
  • Performing Security Code Review

    jeremylongshore/tons-of-skills-marketplace

    Execute this skill enables AI assistant to conduct a security-focused code review using the security-agent plugin.

    2.8k GitHub starsUsed in 2 repos~1.3k tokens
    Auto-check: notes
  • Adapting Transfer Learning Models

    jeremylongshore/tons-of-skills-marketplace

    Build this skill automates the adaptation of pre-trained machine learning models using transfer learning techniques.

    2.8k GitHub stars~1.1k tokensUpdated today
    Auto-check passed
  • Agent Context Loader

    jeremylongshore/tons-of-skills-marketplace

    Execute proactive auto-loading: automatically detects and loads agents.md files.

    2.8k GitHub stars~1.1k tokensUpdated today
    Auto-check passed
  • Aggregating Performance Metrics

    jeremylongshore/tons-of-skills-marketplace

    Aggregate and centralize performance metrics from applications, systems, databases, caches, and services.

    2.8k GitHub stars~1.2k tokensUpdated today
    Auto-check passed
  • Analyzing Capacity Planning

    jeremylongshore/tons-of-skills-marketplace

    Execute this skill enables AI assistant to analyze capacity requirements and plan for future growth.

    2.8k GitHub stars~947 tokensUpdated today
    Auto-check passed
  • Analyzing Database Indexes

    jeremylongshore/tons-of-skills-marketplace

    Process use when you need to work with database indexing. An agent skill from jeremylongshore/tons-of-skills-marketplace.

    2.8k GitHub stars~2k tokensUpdated today
    Auto-check passed

Works with

Questions about Preprocessing Data With Automated Pipelines

What does Preprocessing Data With Automated Pipelines do?

Process automate data cleaning, transformation, and validation for ML tasks. Preprocessing Data With Automated Pipelines is an agent skill from jeremylongshore/tons-of-skills-marketplace. Process automate data cleaning, transformation, and validation for ML tasks.

When should I use Preprocessing Data With Automated Pipelines?

Preprocessing Data With Automated Pipelines fits situations like: requesting preprocess data; data transformation; with relevant phrases based on skill purpose.

How do I install Preprocessing Data With Automated Pipelines in Claude Code?

Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill preprocessing-data-with-automated-pipelines -a claude-code`. Or copy the skill folder (skills/.curated/preprocessing-data-with-automated-pipelines in jeremylongshore/tons-of-skills-marketplace) into .claude/skills/preprocessing-data-with-automated-pipelines in your project. Claude Code loads it when a task matches its description.

How do I install Preprocessing Data With Automated Pipelines in Codex?

Run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill preprocessing-data-with-automated-pipelines -a codex`. Or copy the skill folder (skills/.curated/preprocessing-data-with-automated-pipelines in jeremylongshore/tons-of-skills-marketplace) into .agents/skills/preprocessing-data-with-automated-pipelines in your project. Codex loads it when a task matches its description.

Can I use Preprocessing Data With Automated Pipelines in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add jeremylongshore/tons-of-skills-marketplace --skill preprocessing-data-with-automated-pipelines -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/preprocessing-data-with-automated-pipelines, .gemini/skills/preprocessing-data-with-automated-pipelines, .github/skills/preprocessing-data-with-automated-pipelines and .opencode/skills/preprocessing-data-with-automated-pipelines in your project.

What does Preprocessing Data With Automated Pipelines need to run?

Going by SKILL.md and its folder, Preprocessing Data With Automated Pipelines needs Python for the scripts in its folder. Our summary lists: Python 3. Its frontmatter pre-approves these tools: Read, Write, Edit, Grep, Glob, Bash(cmd:*). Compatibility (from SKILL.md): Designed for Claude Code.

Does Preprocessing Data With Automated Pipelines access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Preprocessing Data With Automated Pipelines safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Preprocessing Data With Automated Pipelines use?

Preprocessing Data With Automated Pipelines is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Preprocessing Data With Automated Pipelines use?

About 1k tokens (SKILL.md is roughly 4.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 18 tokens, read only when the agent opens those files.

What are the alternatives to Preprocessing Data With Automated Pipelines?

Skills that share tags, products or a category with Preprocessing Data With Automated Pipelines: Credit Risk Data Cleaning (github/awesome-copilot, 40k stars), Authoritative Data Harvester (yushui2022/MathModel-Skill, 454 stars), Data Quality Frameworks (wshobson/agents, 40k stars) and Bio Batch Processing (GPTomics/bioSkills, 1.2k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Preprocessing Data With Automated Pipelines?

jeremylongshore (a GitHub user) maintains it in jeremylongshore/tons-of-skills-marketplace, which has 2,827 GitHub stars. The repository holds 3,342 skills in this directory. The repository was last updated on October 10, 2026.

Source: jeremylongshore/tons-of-skills-marketplace on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.