New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

dbt Incremental Models

by AltimateAI in AltimateAI/data-engineering-skills

Helps choose an incremental strategy, design a reliable unique_key and debug failing dbt incremental models, and says when a plain table is the better choice.

MITAuto-check passedData & Analytics

Install dbt Incremental Models

skills CLI
$ npx skills add AltimateAI/data-engineering-skills --skill developing-incremental-models -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install AltimateAI/data-engineering-skills developing-incremental-models --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/AltimateAI/data-engineering-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/dbt/developing-incremental-models .claude/skills/developing-incremental-models && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
developing-incremental-models
GitHub stars
127
Token cost
~2.3k tokens
SKILL.md length
618 words
Files
1
Skills in repo
12
Repo updated
First seen
Licence
MIT

At a glance

Helps choose an incremental strategy, design a reliable unique_key and debug failing dbt incremental models, and says when a plain table is the better choice.

  • Works in 8 steps: Confirm Incremental is Needed → Understand the Source Data Pattern → Choose the Right Strategy → …
  • Creating a new incremental dbt model and picking its strategy and unique_key
  • SKILL.md covers When to Use Incremental, Critical Rules, Workflow and Common Incremental Problems, plus 3 more sections
  • Calls dbt

What it does

The skill's default advice is to keep a model as a `table` unless there is a clear performance reason: under 10 million source rows a full refresh is simpler and fast enough, while larger sources, rows updated in place or append-only logs justify incremental. Before choosing, the agent checks the source size with `dbt show --inline` and answers four questions: is the data append-only, are rows updated, is there a trustworthy timestamp, and what identifies a row.

Strategy choices are `append`, `merge` (named as the safest default), `delete+insert` and `insert_overwrite` for partitioned warehouse tables, with a reminder that adapter support varies. Four rules apply: test with `--full-refresh` first, confirm the unique_key is truly unique in source and target, inspect it for duplicates if a merge keeps failing, and run a full refresh now and then to avoid drift. The description also mentions partition pruning, schema drift and late-arriving data; the excerpt stops after the unique key section.

When your agent uses it

  • Creating a new incremental dbt model and picking its strategy and unique_key
  • Debugging merge errors, partition pruning problems or schema drift in an incremental model
  • Handling late-arriving data in an incremental load
  • Deciding whether a large model should be a table or incremental

Example prompts

  • “Turn the events model into an incremental model. The source is append-only.”
  • “My incremental merge keeps failing on duplicate rows. Check the unique_key.”
  • “Is it worth making stg_orders incremental, or should it stay a table?”
  • “Handle late-arriving records in the sessions model without a full refresh.”

Requirements

  • A dbt project connected to a warehouse adapter

Workflow steps

8 steps, taken from the step headings in SKILL.md.

  1. Confirm Incremental is Needed
  2. Understand the Source Data Pattern
  3. Choose the Right Strategy
  4. Design the Unique Key
  5. Write the Incremental Model
  6. Build with Full Refresh First
  7. Test Incremental Logic
  8. Handle Schema Changes

What it can do on your machine

Read from SKILL.md and the folder at commit 705c68b. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • dbt

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • docs.getdbt.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

dbt Incremental Models loads about 2.3k tokens when it runs. Until then it costs about 133 tokens; SKILL.md has 618 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~133
When it runs · the whole SKILL.md, loaded when a task matches
~2.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from AltimateAI/data-engineering-skills at commit 705c68b, republished under its MIT licence (© AltimateAI). 618 words, ~2,294 tokens.

Download SKILL.mdSave it as .claude/skills/developing-incremental-models/SKILL.md (or your agent's skills folder).
name
developing-incremental-models
description
Develops and troubleshoots dbt incremental models. Use when working with incremental materialization for: (1) Creating new incremental models (choosing strategy, unique_key, partition) (2) Task mentions "incremental", "append", "merge", "upsert", or "late arriving data" (3) Troubleshooting incremental failures (merge errors, partition pruning, schema drift) (4) Optimizing incremental performance or deciding table vs incremental Guides through strategy selection, handles common incremental gotchas.

dbt Incremental Model Development

Choose the right strategy. Design the unique_key carefully. Handle edge cases.

When to Use Incremental

ScenarioRecommendation
Source data < 10M rowsUse table (simpler, full refresh is fast)
Source data > 10M rowsConsider incremental
Source data updated in placeUse incremental with merge strategy
Append-only source (logs, events)Use incremental with append strategy
Partitioned warehouse dataUse insert_overwrite if supported

Default to table unless you have a clear performance reason for incremental.

Critical Rules

  1. ALWAYS test with --full-refresh first before relying on incremental logic
  2. ALWAYS verify unique_key is truly unique in both source and target
  3. If merge fails 3+ times, check unique_key for duplicates
  4. Run full refresh periodically to prevent data drift

Workflow

1. Confirm Incremental is Needed
bash
# Check source table size
dbt show --inline "select count(*) from {{ source('schema', 'table') }}"

If count < 10 million, consider using table instead. Incremental adds complexity.

2. Understand the Source Data Pattern

Before choosing a strategy, answer:

  • Is data append-only? (new rows added, never updated)
  • Are existing rows updated? (need merge/upsert)
  • Is there a reliable timestamp? (for filtering new data)
  • What's the unique identifier? (for merge matching)
bash
# Check for timestamp column
dbt show --inline "
  select
    min(updated_at) as earliest,
    max(updated_at) as latest,
    count(distinct date(updated_at)) as days_of_data
  from {{ source('schema', 'table') }}
"
3. Choose the Right Strategy
StrategyUse WhenHow It Works
appendData is append-only, no updatesINSERT only, no deduplication
mergeData can be updatedMERGE/UPSERT by unique_key
delete+insertData updated in batchesDELETE matching rows, then INSERT
insert_overwritePartitioned tables (BigQuery, Spark)Replace entire partitions

Default: merge is safest for most use cases.

Note: Strategy availability varies by adapter. Check the dbt incremental strategy docs for your specific warehouse.

4. Design the Unique Key

CRITICAL: unique_key must be truly unique in your data.

bash
# Verify uniqueness BEFORE creating model
dbt show --inline "
  select {{ unique_key_column }}, count(*)
  from {{ source('schema', 'table') }}
  group by 1
  having count(*) > 1
  limit 10
"

If duplicates exist:

  • Add more columns to make composite key
  • Add deduplication logic in model
  • Use delete+insert instead of merge
5. Write the Incremental Model
sql
{{
    config(
        materialized='incremental',
        incremental_strategy='merge',  -- or append, delete+insert
        unique_key='id',               -- MUST be unique
        on_schema_change='append_new_columns'  -- handle new columns
    )
}}

select
    id,
    column_a,
    column_b,
    updated_at
from {{ source('schema', 'table') }}

{% if is_incremental() %}
where updated_at > (select max(updated_at) from {{ this }})
{% endif %}
6. Build with Full Refresh First

ALWAYS verify with full refresh before trusting incremental logic.

bash
# First run: full refresh to establish baseline
dbt build --select <model_name> --full-refresh

# Verify output
dbt show --select <model_name> --limit 10
dbt show --inline "select count(*) from {{ ref('model_name') }}"
7. Test Incremental Logic
bash
# Run incrementally (no --full-refresh)
dbt build --select <model_name>

# Verify row count changed appropriately
dbt show --inline "select count(*) from {{ ref('model_name') }}"
8. Handle Schema Changes

Set on_schema_change based on your needs:

SettingBehavior
ignore (default)New columns in source are ignored
append_new_columnsNew columns added to target
sync_all_columnsTarget schema matches source exactly
failError if schema changes

Common Incremental Problems

Problem: Merge Fails with Duplicate Key

Symptom: "Cannot MERGE with duplicate values"

Cause: Multiple rows with same unique_key in source or target.

Fix:

sql
-- Add deduplication using a CTE (cross-database compatible)
with deduplicated as (
    select *,
        row_number() over (partition by id order by updated_at desc) as rn
    from {{ source('schema', 'table') }}
    {% if is_incremental() %}
    where updated_at > (select max(updated_at) from {{ this }})
    {% endif %}
)
select * from deduplicated where rn = 1
Show full SKILL.md (248 more words)Show less
Problem: No Partition Pruning (Full Table Scan)

Symptom: Incremental runs take as long as full refresh.

Cause: Dynamic date filter prevents partition pruning.

Fix:

sql
{% if is_incremental() %}
-- Use static date instead of subquery for partition pruning
where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }}
  and updated_at > (select max(updated_at) from {{ this }})
{% endif %}
Problem: Late-Arriving Data is Missed

Symptom: Some records never appear in incremental model.

Cause: Filtering by max(updated_at) misses late arrivals.

Fix: Use a lookback window with a fixed offset from current date:

sql
{% if is_incremental() %}
-- Lookback 3 days to catch late-arriving data
where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }}
{% endif %}

Alternatively, use a variable for the lookback period:

sql
{% set lookback_days = 3 %}

{% if is_incremental() %}
where updated_at >= {{ dbt.dateadd('day', -lookback_days, dbt.current_timestamp()) }}
{% endif %}
Problem: Schema Drift Causes Errors

Symptom: "Column X not found" after source adds column.

Fix: Set on_schema_change='append_new_columns' in config.

Problem: Data Drift Over Time

Symptom: Counts diverge between incremental and full refresh.

Fix: Schedule periodic full refresh:

bash
# Weekly full refresh
dbt build --select <model_name> --full-refresh

Incremental Strategy Reference

Append (Simplest)
sql
{{ config(materialized='incremental', incremental_strategy='append') }}

select * from {{ source('events', 'raw') }}
{% if is_incremental() %}
where event_timestamp > (select max(event_timestamp) from {{ this }})
{% endif %}
  • No unique_key needed
  • Fastest performance
  • Only use for append-only data (logs, events, immutable records)
Merge (Default)
sql
{{ config(
    materialized='incremental',
    incremental_strategy='merge',
    unique_key='id'
) }}

select * from {{ source('crm', 'contacts') }}
{% if is_incremental() %}
where updated_at > (select max(updated_at) from {{ this }})
{% endif %}
  • Requires unique_key
  • Handles updates and inserts
  • Most common strategy
Delete+Insert (Batch Updates)
sql
{{ config(
    materialized='incremental',
    incremental_strategy='delete+insert',
    unique_key='id'
) }}

select * from {{ source('orders', 'raw') }}
{% if is_incremental() %}
where order_date >= {{ dbt.dateadd('day', -7, dbt.current_timestamp()) }}
{% endif %}
  • Deletes all matching rows first
  • Good for reprocessing batches
  • Use when merge has duplicate key issues
Insert Overwrite (Partitioned)
sql
{{ config(
    materialized='incremental',
    incremental_strategy='insert_overwrite',
    partition_by={'field': 'event_date', 'data_type': 'date'}
) }}

select * from {{ source('events', 'raw') }}
{% if is_incremental() %}
where event_date >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }}
{% endif %}
  • Replaces entire partitions
  • Best for partitioned tables in BigQuery/Spark
  • No unique_key needed (operates on partitions)

Anti-Patterns

  • Using incremental for small tables (< 10M rows)
  • Not testing with full-refresh first
  • Using append strategy when data can be updated
  • Not verifying unique_key uniqueness
  • Relying on exact timestamp match without lookback
  • Never running full refresh (causes data drift)
  • Using merge with non-unique keys

Testing Checklist

  • Model runs with --full-refresh
  • Model runs incrementally (without flag)
  • unique_key verified as truly unique
  • Row counts reasonable after incremental run
  • Late-arriving data handled (lookback window)
  • Schema changes handled (on_schema_change set)
  • Periodic full refresh scheduled

© AltimateAI, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/dbt/developing-incremental-models of AltimateAI/data-engineering-skills.

Open the folder on GitHubat commit 705c68b

Compare with similar skills

dbt Incremental Models next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

dbt Incremental Models compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
dbt Incremental Models this skillAltimateAI/data-engineering-skills127—~2.3kAutomated safety check: PassMIT
Erd Studio Setupliam-machine/erd-studio165—~8.5kAutomated safety check: PassCustom licence
Analytics Engineerborghei/Claude-Skills874—~3.4kAutomated safety check: PassMIT
Airflow State Storeastronomer/agents450—~6.1kAutomated safety check: PassApache-2.0
Modeling Revenue MetricsPostHog/posthog-foss721—~1.9kAutomated safety check: PassMIT
Migrating Dbt Project Across PlatformsKilo-Org/kilo-marketplace189—~3.9kAutomated safety check: PassApache-2.0

Similar skills

  • Erd Studio Setup

    liam-machine/erd-studio

    Friendly, step-by-step setup for ERD Studio in an existing dbt project, for people who may be new to dbt or data modelling.

    165 GitHub stars~8.5k tokensUpdated 3 days ago
    Data & AnalyticsAuto-check passed

  • Analytics Engineer

    borghei/Claude-Skills

    Analytics engineering across data modeling, dbt, transformation, and semantic layers.

    874 GitHub stars~3.4k tokensUpdated today
    Data & AnalyticsAuto-check passed

  • Airflow State Store

    astronomer/agents

    Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (taskstatestore, assetstatestore) and the crash-safe ResumableJobMixin.

    450 GitHub stars~6.1k tokensUpdated yesterday
    Data & AnalyticsAuto-check passed

  • Modeling Revenue Metrics

    PostHog/posthog-foss

    Official

    Build reusable revenue models — MRR, ARR, gross revenue, new/expansion/contraction/churn, ARPU, LTV, and per-customer/per-account revenue — on either PostHog data-warehouse views (HogQL) or an…

    721 GitHub stars~1.9k tokensUpdated today
    Data & AnalyticsAuto-check passed

  • Migrating Dbt Project Across Platforms

    Kilo-Org/kilo-marketplace

    A skill your agent uses when migrating a dbt project from one data platform or data warehouse to another (e.g., Snowflake to Databricks, Databricks to Snowflake) using dbt Fusion's real-time…

    189 GitHub stars~3.9k tokensUpdated 8 days ago
    Data & AnalyticsAuto-check passed

  • Modeling Activation Metrics

    PostHog/posthog-foss

    Official

    Build reusable activation models — an activation-rate metric and a per-user/per-account activated flag — on either PostHog data-warehouse views (HogQL) or an external dbt project.

    721 GitHub stars~1.4k tokensUpdated today
    Data & AnalyticsAuto-check passed

More from AltimateAI/data-engineering-skills

All 12 skills in this repo

  • Altimate Data Warehouse Delegate

    AltimateAI/data-engineering-skills

    Delegates dbt and warehouse tasks such as lineage, migrations and cost attribution to the altimate-code CLI agent and relays its answer back.

    127 GitHub stars~1.4k tokensUpdated 5 days ago
    Auto-check passed

  • dbt Model Builder

    AltimateAI/data-engineering-skills

    Creates or modifies dbt models in line with a project's own conventions, then runs dbt build and dbt show to check the output instead of stopping at compile.

    127 GitHub stars~890 tokensUpdated 5 days ago
    Auto-check passed

  • dbt Error Debugging

    AltimateAI/data-engineering-skills

    Walks through fixing dbt compilation, database and test errors: read the full error, check upstream models, apply a fix, then verify with dbt build and a data preview.

    127 GitHub stars~1.1k tokensUpdated 5 days ago
    Auto-check passed

  • dbt Model Documentation

    AltimateAI/data-engineering-skills

    Writes model and column descriptions in dbt schema.yml files, matching the project's existing documentation style and recording grain, business rules and caveats.

    127 GitHub stars~1.2k tokensUpdated 5 days ago
    Auto-check passed

  • Expensive Snowflake Query Finder

    AltimateAI/data-engineering-skills

    Ranks the costliest, slowest or heaviest-scanning Snowflake queries from query history and suggests how to optimize them.

    127 GitHub stars~662 tokensUpdated 5 days ago
    Auto-check passed

  • Migrating SQL To Dbt

    AltimateAI/data-engineering-skills

    Converts legacy SQL to modular dbt models. An agent skill from AltimateAI/data-engineering-skills.

    127 GitHub stars~762 tokensUpdated 5 days ago
    Auto-check passed

Works with

Categories

Questions about dbt Incremental Models

What does dbt Incremental Models do?

Helps choose an incremental strategy, design a reliable unique_key and debug failing dbt incremental models, and says when a plain table is the better choice. The skill's default advice is to keep a model as a `table` unless there is a clear performance reason: under 10 million source rows a full refresh is simpler and fast enough, while larger sources, rows updated in place or append-only logs justify incremental. Before choosing, the agent checks the source size with `dbt show --inline` and answers four questions: is the data append-only, are rows updated, is there a trustworthy timestamp, and what identifies a row.

When should I use dbt Incremental Models?

dbt Incremental Models fits situations like: creating a new incremental dbt model and picking its strategy and unique_key; debugging merge errors, partition pruning problems or schema drift in an incremental model; handling late-arriving data in an incremental load; deciding whether a large model should be a table or incremental.

How do I install dbt Incremental Models in Claude Code?

Run `npx skills add AltimateAI/data-engineering-skills --skill developing-incremental-models -a claude-code`. Or copy the skill folder (skills/dbt/developing-incremental-models in AltimateAI/data-engineering-skills) into .claude/skills/developing-incremental-models in your project. Claude Code loads it when a task matches its description.

How do I install dbt Incremental Models in Codex?

Run `npx skills add AltimateAI/data-engineering-skills --skill developing-incremental-models -a codex`. Or copy the skill folder (skills/dbt/developing-incremental-models in AltimateAI/data-engineering-skills) into .agents/skills/developing-incremental-models in your project. Codex loads it when a task matches its description.

Can I use dbt Incremental Models in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add AltimateAI/data-engineering-skills --skill developing-incremental-models -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/developing-incremental-models, .gemini/skills/developing-incremental-models, .github/skills/developing-incremental-models and .opencode/skills/developing-incremental-models in your project.

What does dbt Incremental Models need to run?

Going by SKILL.md and its folder, dbt Incremental Models needs the command-line tools its instructions call (dbt). Our summary lists: A dbt project connected to a warehouse adapter.

Does dbt Incremental Models access the network?

SKILL.md names 1 domain. As links in the text: docs.getdbt.com. This is read from the text; nothing was executed.

Is dbt Incremental Models safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does dbt Incremental Models use?

dbt Incremental Models is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does dbt Incremental Models use?

About 2.3k tokens (SKILL.md is roughly 9.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to dbt Incremental Models?

Skills that share tags, products or a category with dbt Incremental Models: Erd Studio Setup (liam-machine/erd-studio, 165 stars), Analytics Engineer (borghei/Claude-Skills, 874 stars), Airflow State Store (astronomer/agents, 450 stars) and Modeling Revenue Metrics (PostHog/posthog-foss, 721 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains dbt Incremental Models?

AltimateAI (a GitHub organization) maintains it in AltimateAI/data-engineering-skills, which has 127 GitHub stars. The repository holds 12 skills in this directory. The repository was last updated on October 1, 2026.

Source: AltimateAI/data-engineering-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.