Migrating Dagster To Airflow

Dagster → Airflow 3 (Astro) migration

Migrate a Dagster project to Airflow 3 on Astro Runtime, honestly. The migration is asset-first (Dagster asset graphs translate to Airflow assets and asset-aware schedules, not flattened DAGs), incremental (domain by domain, Dagster stays authoritative until parity), and honest (every definition gets an explicit disposition; semantic deltas are documented, never papered over).

First time driving this? Read reference/quickstart.md first: hour-one commands, the glossary, and what can and cannot break.

Migration at a glance

1. Baseline the source project's tests, then inventory it read-only (scripts/inventory.py → manifest).
2. Review classifications (MECH/JUDG/REDESIGN/NONE per reference/mapping.md); make the go/no-go call (three outcomes; migrate-with-conditions is the common case, stay is the narrow one); plan DAG boundaries, per-edge IO decisions, and Gate 3 expectations into the manifest.
3. Trial-migrate 2-3 representative units end-to-end through every validation gate.
4. Migrate domain by domain through the six-gate ladder (reference/validation.md), tracking per-unit state (scripts/status.py); fix failure classes via reference/troubleshooting.md, never stub.
5. Map the platform layer (secrets, alerts, CI/CD, Deployments) per reference/astro-deployment.md.
6. Run side by side, then cut over per domain (consumers unpause first; see the checklist), keeping rollback one step away.
7. Deliver the migration report: every definition dispositioned, an equivalence row per trigger, losses stated plainly.

Version drift

Verified against Airflow 3.3.0 / Astro Runtime 3.3-2 / astronomer-cosmos 1.15 / Dagster 1.13 (2026-07). Version-sensitive rows in the references carry their floor (notably the 3.2-vs-3.3 partition surface). Before relying on a version-gated claim: check the target (airflow version, astro deployment inspect), probe imports for sdk surface (python3 -c "from airflow.sdk import X"), and prefer --help / API spec discovery over assuming verbatim CLI/REST contracts on newer versions. Playbook entries are version-scoped per entry.

Requirements

• Target Astro Runtime 3.3+ (Airflow 3.3+); the native asset-partition surface requires it. Below 3.2 the mapping degrades badly; say so and recommend upgrading before migrating.
• The Dagster repo, and ideally a running Dagster instance (its materialization metadata provides parity-test fixtures).
• astro CLI for the target project.

Hard rules

1. Never stub. A translated unit either works through its validation gate or is deferred with a written reason. Fake-success bodies and workaround code with long justifying comments are failures.
2. No silent omissions. Every record in the inventory manifest ends complete or deferred (reason). scripts/status.py summary exits nonzero otherwise; run it before claiming done.
3. Equivalence rows for every trigger. Each schedule/sensor/automation condition gets a report row: source spelling, target spelling, delta in one sentence. Semantic deltas exist (catchup, on_cron inversion, eager guarantees); the sin is not the delta, it is the undocumented delta.
4. Fix classes, not instances. When a translation pattern fails validation, fix the pattern (and record it in reference/troubleshooting.md), then re-apply; do not hand-patch one unit.
5. Do not invent APIs. The references contain verified names only. Anything not covered there gets verified against official docs before use.

Workflow

Phase 0: Preflight

Confirm target Runtime version, astro CLI presence, and repo access. Detect the project layout: classic (@repository/workspace.yaml), modern (Definitions), or Components (pyproject.toml [tool.dg], defs.yaml files); all three occur, sometimes together. Baseline the source project's test suite now: pre-existing failures are recorded and excluded from migration blame.

Phase 1: Inventory (read-only)

python3 scripts/inventory.py <dagster_repo> --out manifest.json          # static scan
python3 scripts/inventory.py <dagster_repo> --runtime --out manifest.json # + runtime introspection when the project imports

The manifest lists every definition with file:line, captured params, current-vs-deprecated spelling, and dependency edges with their IO manager; every record starts classification: "pending". Classifying is YOUR first judgment task: assign each record MECH / JUDG / REDESIGN / NONE from its row in reference/mapping.md and write it into the manifest. The scanner enumerates (deterministic completeness); the agent classifies (judgment). A record you cannot map to a mapping.md row is itself a finding: record it, do not guess. Also grep for DAGSTER_CLOUD_ and EnvVar( (platform layer, Phase 5).

Manifest conventions: the canonical manifest lives in the migration run directory. Once the Astro project exists (Phase 2 scaffold), copy the manifest to its include/inventory/manifest.json so the Gate 3 pytest and status.py defaults find it; until then it just stays in the run dir (keep the two in sync afterward, the run-dir copy wins). Static records are the canonical migration units; runtime-mode enrichment merges into them, and only genuinely runtime-only definitions (factory-generated) become new units.

Emit the migration report skeleton now: one section per manifest record, plus the secrets/env naming map from reference/astro-deployment.md. Scale the skeleton to the project: a secretless local project gets a one-line "no secrets/platform layer" note, not empty boilerplate sections.

Phase 1.5: Go/no-go (the honest gate)

Before translating anything, answer the project-level question the inventory makes answerable: what does this team give up by migrating, and does each loss have an acceptable answer? Assess the NONE and REDESIGN rows against what is load-bearing for THIS team, evaluating the mitigation, not just the loss:

One rule the table implies, stated plainly: no dbt-only condition reaches "stay." Between Cosmos, state-aware dbt builds, and CI-triggered builds, every dbt-workflow loss has an accepted-practice mitigation (execution-proven in this skill's eval program, including on a real warehouse); dbt items are conditions to record, never blockers. The observability rows (per-asset cost, column-level lineage) are separate conditions and are evaluated on their own, even for dbt-heavy teams.

The gate's outcome is three-valued, and the middle one is the common case:

• Migrate: no stay-signals; proceed to Phase 2.
• Migrate with conditions (most real projects): losses exist, mitigations are named and accepted in writing in the report's first section, specific domains may carry REDESIGN work; proceed to Phase 2 with those conditions recorded.
• Stay on Dagster, today: reserved for the case where MULTIPLE stay-signal conditions in the right column genuinely hold at once and the migration is not externally mandated. Then the honest deliverable is that recommendation, in writing, with the specific unmitigated losses named, and the run stops there. A migration guide that cannot say "don't" cannot be trusted when it says "do", but "don't" is earned by unmitigatable losses, not by the mere existence of deltas.