New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Data Engineering

by cbrock84 in cbrock84/headcount

Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery.

MITAuto-check passedData & Analytics

Install Data Engineering

skills CLI
$ npx skills add cbrock84/headcount --skill data-engineering -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install cbrock84/headcount data-engineering --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/cbrock84/headcount.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/data-analytics/skills/data-engineering .claude/skills/data-engineering && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
data-engineering
GitHub stars
2k
Token cost
~937 tokens
SKILL.md length
490 words
Files
2 (incl. references)
Skills in repo
177
Repo updated
First seen
Licence
MIT

At a glance

Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery.

  • Tasks that involve Data pipelines and ETL
  • SKILL.md covers Land raw, transform downstream, Idempotence is the property…, Late, duplicate and… and Test data, not just code, plus 3 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
  • Tasks that involve Data cleaning

What it does

Data Engineering is an agent skill from cbrock84/headcount. Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.

Its SKILL.md is about 940 tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files, including reference files (for example `references/sources.md`).

It sits in Data & Analytics, covering Data pipelines and ETL and Data cleaning. The repository describes itself as: An agent organization structured as a company — 15+ departments, 125+ skills, each independently installable, citing the standards and regulators that settle the question. Runs… The licence is MIT.

When your agent uses it

  • Tasks that involve Data pipelines and ETL
  • Tasks that involve Data cleaning

Example prompts

  • “Use the data-engineering skill to build and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of…”
  • “/data-engineering”

What it can do on your machine

Read from SKILL.md and the folder at commit 98d1c17. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Data Engineering loads about 937 tokens when it runs, and up to ~1.5k if it reads all its reference files. Until then it costs about 90 tokens; SKILL.md has 490 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~90
When it runs · the whole SKILL.md, loaded when a task matches
~937
With references · SKILL.md plus every file in references/, read only if the agent opens them
~1.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from cbrock84/headcount at commit 98d1c17, republished under its MIT licence (© cbrock84). 490 words, ~937 tokens.

Download SKILL.mdSave it as .claude/skills/data-engineering/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
data-engineering
description
Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.

Data engineering

Pipelines are production systems whose failures are quiet. A broken service pages someone; a broken pipeline produces plausible numbers that people act on for a week.

This is movement and transformation. Schema and semantics belong to data-analytics:data-modeling, policy and stewardship to data-analytics:data-governance.

Land raw, transform downstream

Keep an immutable copy of source data exactly as received. Transformation logic will be wrong at some point, and raw data is what lets you reprocess rather than re-request from a source that may no longer have it.

Business logic belongs downstream where it is visible and testable, not buried in ingestion. The exception is transformation required for privacy — minimization, pseudonymization, dropping fields you have no basis to hold — which belongs at ingest precisely because raw storage is what the obligation attaches to. See legal-risk:privacy-and-data-protection.

Idempotence is the property that matters

Every pipeline will be re-run: after a failure, after a fix, after a late-arriving correction. A re-run that double-counts is worse than a failure, because it produces a wrong answer silently.

Design for exactly-once effect at the destination — deterministic keys, merges rather than blind appends, partitioned overwrites. Then re-running is safe and recovery stops being frightening.

Late, duplicate and out-of-order data

Real sources deliver all three. Decide explicitly, per pipeline: how late is an event still accepted, what happens to one arriving after its window closed, and how duplicates are identified.

Distinguish event time from processing time and partition on event time. Aggregations built on arrival time silently reassign yesterday's activity to today whenever a delivery is delayed.

Show full SKILL.md (233 more words)Show less

Test data, not just code

Unit tests on transformation logic catch the wrong class of failure. Most damage comes from data that is valid but wrong. Assert on the data itself, in the pipeline, and fail loudly:

  • Row counts within an expected range, not merely non-zero.
  • Uniqueness of keys, and referential integrity across joins.
  • Freshness — the newest record is recent enough to be meaningful.
  • Distribution shifts in important columns.

A silent failure is worse than a loud one. Prefer stopping the pipeline to publishing data you do not trust.

Sources

references/sources.md in this skill lists the outside authorities that settle the questions here — what each one is authoritative for, and what you may do with it. Check them before answering on anything they cover, and cite what you used. Most are free to read and not free to reproduce; the use note on each is binding.

Tooling

Warehouses and lakehouses: Snowflake, BigQuery, Databricks, Redshift, and Postgres or DuckDB at small scale, and similar.

Ingestion: Fivetran, Airbyte, Stitch, and similar. Transformation: dbt, SQLMesh. Orchestration: Airflow, Dagster, Prefect, and similar.

Buy ingestion and build transformation. Connector maintenance returns nothing for the time your team puts into it.

Never

  • Transform on ingest for business reasons and discard the raw copy.
  • Build a pipeline whose re-run double-counts.
  • Aggregate on processing time when event time is available.
  • Let a pipeline fail silently and publish stale data as current.

© cbrock84, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (references) in plugins/data-analytics/skills/data-engineering of cbrock84/headcount.

  • SKILL.md
  • references/sources.md

Open the folder on GitHubat commit 98d1c17

Compare with similar skills

Data Engineering next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Data Engineering compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Data Engineering this skillcbrock84/headcount2k—~937Automated safety check: PassMIT
Credit Risk Data Cleaninggithub/awesome-copilot40k1 repos~1.5kAutomated safety check: PassMIT
Data Pipelineagulli/atlas-agents578—~714Automated safety check: PassMIT
Authoritative Data Harvesteryushui2022/MathModel-Skill4521 repos~1.1kAutomated safety check: PassMIT
Data Quality Frameworkswshobson/agents40k10 repos~1.1kAutomated safety check: PassMIT
Dbt Transformation Patternswshobson/agents40k8 repos~781Automated safety check: PassMIT

Similar skills

  • Credit Risk Data Cleaning

    github/awesome-copilot

    Official

    Cleans raw credit data and screens variables before loan modeling, dropping unstable, noisy or redundant features and writing an Excel report of every step.

    40k GitHub starsUsed in 1 repo~1.5k tokens
    Data & AnalyticsAuto-check passed

  • Data Pipeline

    agulli/atlas-agents

    Design, build, or debug data processing pipelines. An agent skill from agulli/atlas-agents.

    578 GitHub stars~714 tokensUpdated 2 mo ago
    Data & AnalyticsAuto-check passed

  • Authoritative Data Harvester

    yushui2022/MathModel-Skill

    Finds authoritative public data sources for modeling tasks, prefers official APIs and bulk downloads, and outputs a reproducible fetch and cleaning plan with citations.

    452 GitHub starsUsed in 1 repo~1.1k tokens
    Data & AnalyticsAuto-check passed

  • Data Quality Frameworks

    wshobson/agents

    Sets up data quality checks with Great Expectations, dbt tests and data contracts, with checkpoints and pass-fail reports for pipelines.

    40k GitHub starsUsed in 10 repos~1.1k tokens
    Data & AnalyticsAuto-check passed

  • Dbt Transformation Patterns

    wshobson/agents

    Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies.

    40k GitHub starsUsed in 8 repos~781 tokens
    Data & AnalyticsAuto-check passed

  • Data Engineering

    rohitg00/awesome-claude-code-toolkit

    Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation

    2.7k GitHub stars~1.7k tokensUpdated 4 mo ago
    Data & AnalyticsAuto-check passed

More from cbrock84/headcount

All 177 skills in this repo

  • Agent Hierarchy

    cbrock84/headcount

    Designs orchestrator-and-subagent hierarchies for a repository — splitting agents by exclusive write surface, pairing every producer with an independent auditor, and enforcing the split with a…

    2k GitHub stars~1.2k tokensUpdated 19 days ago
    Auto-check passed

  • Access And Identity

    cbrock84/headcount

    Designs and audits who can reach what — authentication, authorization models, privileged access, service credentials, and joiner-mover-leaver process.

    2k GitHub stars~1.1k tokensUpdated 19 days ago
    Auto-check passed

  • Account Based Marketing

    cbrock84/headcount

    Concentrates marketing and sales effort on a named set of accounts rather than on volume — qualifying whether the model fits your economics at all, building the account list and the buying group…

    2k GitHub stars~1.2k tokensUpdated 19 days ago
    Auto-check passed

  • Activation

    cbrock84/headcount

    Gets new users from signup to first real value — signup flow, onboarding, time-to-value, and the early experience that determines whether someone becomes a user or a lapsed account.

    2k GitHub stars~865 tokensUpdated 19 days ago
    Auto-check passed

  • AI Research Analyst

    cbrock84/headcount

    Produces executive-level research — market sizing, competitor mapping, trend analysis, and strategic intelligence — grounded in cited sources with the confidence in each claim made explicit.

    2k GitHub stars~916 tokensUpdated 19 days ago
    Auto-check passed

  • AI Search Optimization

    cbrock84/headcount

    Optimizes for AI assistants and AI-generated answers — being retrievable, being cited, and being represented accurately when a model answers on your behalf.

    2k GitHub stars~829 tokensUpdated 19 days ago
    Auto-check passed

Categories

Questions about Data Engineering

What does Data Engineering do?

Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Data Engineering is an agent skill from cbrock84/headcount. Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery.

When should I use Data Engineering?

Data Engineering fits situations like: tasks that involve Data pipelines and ETL; tasks that involve Data cleaning.

How do I install Data Engineering in Claude Code?

Run `npx skills add cbrock84/headcount --skill data-engineering -a claude-code`. Or copy the skill folder (plugins/data-analytics/skills/data-engineering in cbrock84/headcount) into .claude/skills/data-engineering in your project. Claude Code loads it when a task matches its description.

How do I install Data Engineering in Codex?

Run `npx skills add cbrock84/headcount --skill data-engineering -a codex`. Or copy the skill folder (plugins/data-analytics/skills/data-engineering in cbrock84/headcount) into .agents/skills/data-engineering in your project. Codex loads it when a task matches its description.

Can I use Data Engineering in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add cbrock84/headcount --skill data-engineering -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/data-engineering, .gemini/skills/data-engineering, .github/skills/data-engineering and .opencode/skills/data-engineering in your project.

What does Data Engineering need to run?

SKILL.md names no scripts, command-line tools or credentials: Data Engineering is instructions for the agent only.

Does Data Engineering access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Data Engineering safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Data Engineering use?

Data Engineering is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Data Engineering use?

About 937 tokens (SKILL.md is roughly 3.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 577 tokens, read only when the agent opens those files.

What are the alternatives to Data Engineering?

Skills that share tags, products or a category with Data Engineering: Credit Risk Data Cleaning (github/awesome-copilot, 40k stars), Data Pipeline (agulli/atlas-agents, 578 stars), Authoritative Data Harvester (yushui2022/MathModel-Skill, 452 stars) and Data Quality Frameworks (wshobson/agents, 40k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Data Engineering?

cbrock84 (a GitHub user) maintains it in cbrock84/headcount, which has 2,001 GitHub stars. The repository holds 177 skills in this directory. The repository was last updated on September 17, 2026.

Source: cbrock84/headcount on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.