Python · Data pipelines and ETL

Scrapes sites, handles JavaScript-heavy pages and extracts structured data with Crawl4AI, through its crwl CLI or Python SDK, including schema-based extraction without an LLM.

smallnest/goclaw

Shows how to write custom cognee tasks, chain them into pipelines, store custom DataPoints and run enrichment over the existing graph.

topoteretes/cognee

Selectively monitor important long-running or resource-intensive commands with Haoleme by prefixing them with hao, so status, output, and completion notifications sync to the mobile app.

HaolemeApp/Haoleme

An AI-Native skill for parsing PDF / Office / image files into Markdown with MinerU — a fast, zero-config document parser for AI agents.

Nebutra/MinerU-Skill

Turns a Tushare API doc URL into a full data plugin for the stock_datasource repo: extractor, ClickHouse schema, query service, config and curl examples.

Yourdaylight/stock_datasource

Explains how to load text, files, folders, URLs, repositories and databases into cognee memory with remember(), including datasets, extractors and tags.

topoteretes/cognee

Cleans raw credit data and screens variables before loan modeling, dropping unstable, noisy or redundant features and writing an Excel report of every step.

github/awesome-copilot

Refreshes dbt artifact schemas from dbt-labs/dbt-core and regenerates Pydantic parser classes.

yu-iskw/dbt-artifacts-parser

Writes a new Dinobase YAML connector for a REST API that has no verified dlt source, covering auth, pagination, read and write endpoints and incremental loading.

kappa90/dinobase

Build, validate, and manage semantic models using Sidemantic.

sidequery/sidemantic

End-to-end AI-assisted meta-analysis pipeline orchestration from TOPIC.txt to final manuscript and reviewer responses.

htlin222/meta-pipe

Analyzes a dbt project and suggests dbt-bouncer checks that already pass (for existing projects) or a sensible starter config (for greenfield projects).

godatadriven/dbt-bouncer

Finds authoritative public data sources for modeling tasks, prefers official APIs and bulk downloads, and outputs a reproducible fetch and cleaning plan with citations.

yushui2022/MathModel-Skill

Bumps the dbt-artifacts-parser package semver, prepares a PyPI release, and aligns GitHub Releases with version.

yu-iskw/dbt-artifacts-parser

Uses Ray Data to read, transform and write large datasets across a cluster for ML training and batch inference, with streaming execution and optional GPU steps.

Orchestra-Research/AI-Research-SKILLs

Sets up data quality checks with Great Expectations, dbt tests and data contracts, with checkpoints and pass-fail reports for pipelines.

wshobson/agents

Extracts pixel, video-frame, speech, music and visual-semantic features from image, video and audio files for research datasets, using local tools or the Gemini API.

TyrealQ/q-skills

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure.

benchflow-ai/skillsbench

Pre-PR code review gate. An agent skill from ByteDance-Seed/VeOmni.

ByteDance-Seed/VeOmni

Scaffold a new dbt-bouncer check class with tests. An agent skill from godatadriven/dbt-bouncer.

godatadriven/dbt-bouncer

Patterns for writing production-ready Apache Airflow DAGs: task dependencies, custom operators and sensors, local testing, and rules for what to avoid.

wshobson/agents

Speed up slow Apache Spark jobs by tuning partitions, shuffles, data skew, caching and executor memory, with do and don't rules for PySpark code.

wshobson/agents

Bump the Airflow or Python version across all project files.

godatadriven/whirl

Extracts research papers from .bib and .html venue files, filters them by relevance, labels them with a two-level taxonomy through Claude and rebuilds a website.

PurCL/ASE

Microscopy data management platform. An agent skill from davila7/claude-code-templates.

davila7/claude-code-templates

Run Python code in the cloud with serverless containers, GPUs, and autoscaling.

davila7/claude-code-templates

Regenerate dbt test fixtures after dbtproject changes. An agent skill from godatadriven/dbt-bouncer.

godatadriven/dbt-bouncer

Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

alirezarezvani/claude-skills

Build automated AI workflows combining multiple models and services.

NeverSight/learn-skills.dev

Guides technology selection and implementation of AI and ML features in .NET 8+ applications using ML.NET, Microsoft.Extensions.AI (MEAI), Microsoft Agent Framework (MAF), GitHub Copilot SDK, ONNX…

dotnet/skills

Writes Airflow task logic in Go using the Airflow Go SDK. An agent skill from astronomer/agents.

astronomer/agents

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

davila7/claude-code-templates

Queries 20+ bioinformatics resources through CLI/Python. An agent skill from K-Dense-AI/scientific-agent-skills.

K-Dense-AI/scientific-agent-skills

Develop and deploy Lakeflow Jobs on Databricks via DABs, Python SDK, or the CLI.

Kilo-Org/kilo-marketplace

Writes Airflow task logic in Java, Kotlin, or any JVM language using the Airflow Java SDK.

astronomer/agents

A skill your agent uses when: comparing SQL Server tables across instances, data migration validation, ETL verification, row mismatch detection, schema drift, reconciliation report, production vs…

github/awesome-copilot

The language-neutral foundation for Airflow language SDKs — implement task logic in a non-Python language while the DAG stays in Python.

astronomer/agents

Comprehensive Snowflake development assistant covering SQL best practices, data pipeline design (Dynamic Tables, Streams, Tasks, Snowpipe), Cortex AI functions, Cortex Agents, Snowpark Python, dbt…

sickn33/agentic-awesome-skills

Python error handling patterns including input validation, exception hierarchies, and partial failure handling.

wshobson/agents

A skill your agent uses when writing Snowflake SQL, building data pipelines with Dynamic Tables or Streams/Tasks, using Cortex AI functions, creating Cortex Agents, writing Snowpark Python…

alirezarezvani/claude-skills

Data ingestion patterns for loading data from cloud storage, APIs, files, and streaming sources into databases.

ancoleman/ai-design-components

Transform raw data into analytical assets using ETL/ELT patterns, SQL (dbt), Python (pandas/polars/PySpark), and orchestration (Airflow).

ancoleman/ai-design-components

Generate complete solutions for specific Dataverse SDK use cases with architecture recommendations

github/awesome-copilot

Covers scripting Autodesk 3ds Max, the 3D modeling and rendering application, with MAXScript and Python (pymxs): scene manipulation, object creation, material assignment, camera and light setup…

TerminalSkills/skills

A skill your agent uses when automating ZBrush 2026: batch processing a folder of OBJ or ZTL files, a ZBrush Python script or ZScript, recording a macro to learn a palette path, the ZBrush -script…

scenario-labs/skills

Running experiments out of the data warehouse instead of via dedicated experiment platforms.

rampstackco/claude-skills

Provides guidance for authoring Apache Airflow DAGs in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer).

google/skills

Authors Apache Airflow DAGs declaratively from dag-factory YAML configs.

astronomer/agents