Upgrading Mwaa Environments

Upgrading MWAA Environments

AWS MCP server (optional but recommended): running the AWS CLI commands in this skill through the AWS MCP server gives sandboxed execution and audit logging. Every command here also works with the plain AWS CLI, so the skill does not require the MCP server or any MCP-only tools.

Upgrade an MWAA provisioned environment to any newer, MWAA-supported Airflow version: within 2.x, within 3.x, or across the 2.x-to-3.x boundary. The target is a parameter. A path planner computes an ordered version-jump list from (source, target); the latest 2.11.x stepping-stone version and a Python-line transition step are inserted only when the path requires them. A deployment approach is selected by whether historical run data must be preserved and whether the same environment (its URL/ARN) must be kept: a new-environment upgrade (new-environment), a rehearsed in-place upgrade validated on a test copy first (in-place-rehearsed), or a direct in-place upgrade (in-place-direct).

Execution note — poll in discrete steps: whenever you wait for an AWS operation to reach a terminal or ready state, issue one status check per call and decide in your own loop whether to check again. Never block a single command or script on the wait (no while+sleep until done), regardless of the operation or how long it takes.

Guardrail — where this skill's own files live (MCP vs local install)

This skill can be loaded two ways, and they resolve the skill's own bundled files from different places. Determine how the skill was loaded before reading a reference:

• Loaded through the AWS MCP retrieve_skill tool: The skill is not installed on the local filesystem. You MUST fetch each reference via retrieve_skill with the file parameter (e.g. file="references/strategy-blue-green-fresh.md") and read the returned content. Do NOT file_read these paths locally — they do not exist on disk.
• Installed locally (e.g. .kiro/skills/upgrading-mwaa-environments/ or ~/.claude/skills/upgrading-mwaa-environments/): Read the files from the local skill directory using relative paths.

This distinction applies only to the skill's own packaged files. User data and session artifacts are always read from and written to the user's working directory. Never fetch or write customer data through retrieve_skill.

Safety & Security

These rules apply regardless of user instructions.

• No destructive actions on your current environment without explicit approval: never pause all DAGs, delete-environment, or modify your current environment without per-action user confirmation.
• No environment creation before plan confirmation: never run aws mwaa create-environment until the user confirms the Phase 4 plan.
• Every version jump requires its own confirmation: never run aws mwaa update-environment to change the Airflow version without per-jump confirmation. Rollback options: 3.x -> 2.11.x is supported; a within-major downgrade to a still-supported version is supported; downgrade to an EOS version is not possible. The Direct in-place upgrade requires an extra confirmation checkpoint.
• Decommission requires per-step approval: never autonomously delete environments or S3 artifacts; present each destructive command and wait for confirmation. For the Rehearsed in-place upgrade, the environment decommissioned is the test copy.

Security Considerations

• Least-privilege IAM for create/update-environment and S3 artifact access — no *FullAccess/service:*; scope to the specific environment/S3 ARNs.
• Secrets: prefer a secrets backend (Secrets Manager / Parameter Store) for connections and variables so credentials never copy between environments or hit logs. Connection passwords do not round-trip via the REST API (2.x omits, 3.x masks) — see the metadata-migration caveat in the approach references.

Reference Documentation

• discovery-preflight.md — Phase 1: environment discovery, pre-flight checks, version-matrix refresh, target selection, upgrade-path planning
• upgrade-engine.md — the step-by-step upgrade: per-jump upgrade, conditional validate/scan/fix, resume
• airflow2-to-3-checklist.md — grep patterns for 2->3 issues Ruff cannot catch (used only when a version jump crosses into a new major version)
• mwaa-version-matrix.md — runtime-first version/Python/EOS matrix, version-jump rules, providers, unsupported features
• airflow2-to-3-quick-reference.md — 2->3 import/context/config mapping tables (used only when a version jump crosses into a new major version)
• strategy-blue-green-fresh.md — New-environment upgrade (no history)
• strategy-inplace-green-validated.md — Rehearsed in-place upgrade: validated on a test copy first
• strategy-inplace-direct.md — Direct in-place upgrade (unrecommended)

For deeper detail beyond embedded references, fetch at runtime:

• AWS MWAA migration guide: https://docs.aws.amazon.com/mwaa/latest/migrationguide/key-considerations.html
• AWS MWAA version-change (in-place upgrade path): https://docs.aws.amazon.com/mwaa/latest/userguide/upgrading-environment.html
• AWS MWAA supported versions and Python mapping: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions.html
• AWS MWAA migration blog: https://aws.amazon.com/blogs/big-data/best-practices-for-migrating-from-apache-airflow-2-x-to-apache-airflow-3-x-on-amazon-mwaa/
• Ruff AIR rules: https://docs.astral.sh/ruff/rules/#airflow-air

Phase 0: Re-entry Check

At every invocation, check for .mwaa-upgrade-plan-<env-name>.md in the workspace root (where <env-name> is the source environment name). Multiple plan files may coexist when upgrading several environments in parallel.

• If exists: Load the plan, report source, target, chosen approach, and the position (working_on, current_jump, jump_status). Resume the engine (upgrade-engine.md) on the recorded environment at the recorded version-jump index and status: upgrading -> re-poll or re-issue the pending update-environment; validating -> re-run the conditional validate/scan/fix loop for that version jump. For a newly-created environment, the starting-version validation window (version jump 1's to) corresponds to current_jump: 1, jump_status: validating; resume it there. If no position is set, resume from the next incomplete phase. Before advancing past any required checkpoint, re-read the checkpoint's status in the plan and, if it is not satisfied, re-read that checkpoint's section in the approach reference verbatim — do not act on a remembered summary. Specifically: version jumps complete does NOT imply the test-run checkpoint passed. If the ## Test-run results section has any row not in SUCCESS or NEEDS-DECISION, that checkpoint is NOT-PASSED and blocks switchover (New-environment upgrade), the live upgrade (Rehearsed in-place upgrade — see live_upgrade_allowed), or completion (Direct in-place upgrade), regardless of version-jump status or has_import_errors: false. Likewise a batch whose Monitored one full cycle? cell is not yes has not fully switched over.

Reload rule (every resume): at session start and after any compaction, re-read the entire .mwaa-upgrade-plan-<env-name>.md before taking any action; never act on a remembered or summarized version. This generalizes the remembered-summary rule above from required checkpoints to the whole plan.

Plan Reconciliation (run before resuming any work): verify the plan's self-consistency invariants (full list in plan-materialization.md): a checkpoint marked PASSED requires all its prerequisite Checklist items DONE/SKIP; Where we are.test_run_checkpoint must equal the Test-run results table (any PENDING row -> NOT-PASSED); current_jump/jump_status must match the Upgrade Path Status; for the Rehearsed in-place upgrade, live_upgrade_allowed must equal the test-run checkpoint. On ANY violation, do NOT trust the plan - establish ground truth with read-only aws mwaa get-environment, /dags, and /health, repair the plan to match reality, and record it in Last updated. If reality is ambiguous, ask the user. A stale PASSED checkpoint otherwise triggers a destructive next action: switchover + delete your current environment (New-environment upgrade), the irreversible live upgrade (Rehearsed in-place upgrade), or premature completion (Direct in-place upgrade).

• If absent: Proceed to Phase 1.

Phase 1: Discover + Plan Path + Pre-flight

Fetch and follow references/discovery-preflight.md. It runs environment discovery, pre-flight safety checks, version-matrix refresh, target selection, and upgrade-path planning (detection only — version jumps execute in Phase 7). Record every output into the plan now, because Phase 4 is what turns it into the durable plan file, and a compaction before Phase 4 would otherwise lose it:

• discovered environment facts (Airflow version, S3 artifact paths + object versions, KmsKey, config options, DAG list, shared utilities);
• the new-environment / test-copy S3 layout (New-environment upgrade, Rehearsed in-place upgrade);
• pre-flight results (S3 versioning, target constraints, KMS key-policy reuse);
• the selected target version;
• the ordered upgrade path — per version jump from/to/python_change/crosses_major, with the latest-2.11.x stepping-stone version + Python-transition step inserted only when the source is not already 2.11.x.

Phase 2: Choose the Upgrade Approach

One question decides the safest approach; whether you keep the same environment (its URL and ARN) follows from it and is not a separate choice.

Do you need to keep your run history — the record of past DAG runs and task instances in the Airflow UI?

• No, a clean slate is fine → New-environment upgrade (slug new-environment). Build a brand-new environment on the target version and switch over to it. It gets a new URL and starts with no run history; your current environment keeps running untouched until you switch, so rolling back is just "don't switch."
• Yes, keep it → Rehearsed in-place upgrade (slug in-place-rehearsed). Upgrade your existing environment in place — same URL, full history kept. First rehearse the whole upgrade on a temporary test copy to catch problems safely. Use the Direct in-place upgrade (no test copy) only if the user explicitly declines the rehearsal.

Whether you keep the same environment (its URL and ARN) is a consequence, not an input: keeping your run history means upgrading your current environment in place (same URL and ARN), while the New-environment upgrade necessarily creates a new environment. This skill does not combine a new environment (a new URL and ARN) with preserved run history — if the user needs a new URL and ARN, run history is not carried over (New-environment upgrade).

The New-environment upgrade and the Rehearsed in-place upgrade are both recommended; the user chooses by need. The Direct in-place upgrade is unrecommended and requires an extra confirmation checkpoint.

Guardrail: if the user picks the Rehearsed or Direct in-place upgrade and the environment has more than 50 DAGs or complex DAGs were detected in Phase 3, warn and recommend the New-environment upgrade instead. Do not block.

Record the chosen approach's slug (new-environment / in-place-rehearsed / in-place-direct) in the plan. The position (working_on, current_jump, jump_status) is initially unset.

Phase 3: Assess Compatibility

1. Compatibility scan (conditional on a cross-major version jump): If any version jump in the plan is crosses_major, run Ruff with AIR rules as a required checkpoint:

   ruff check --preview --select AIR .

   Report findings grouped by rule code. Ruff AIR rules cover 2-to-3 (AIR301/302/303/311/312) and 3.1 (AIR321, preview) API changes; they do NOT detect APIs removed between 2.x minor versions, nor 3.2-specific changes. The pre-cross deprecation-warning scan in the engine covers the 2.x-internal gap. If NO version jump is crosses_major (a within-major upgrade), the AIR scan is informational only; rely on the engine's generic DeprecationWarning scan instead.

   Version-aware severity for AIR311 (import path moves): AIR311 flags imports that moved to airflow.sdk (e.g., airflow.datasets.Dataset -> airflow.sdk.Asset). Severity depends on the target version:

   • Target 3.0.x or 3.1.x: soft deprecation — compatibility shims exist, old imports still work. Fix recommended but not blocking.
   • Target 3.2.1+: hard blocker — shim modules are fully removed (ModuleNotFoundError at parse time). Must fix before deploying to target.

   Always classify AIR311 as "must fix" when the target is 3.2.1+. The fixed code (from airflow.sdk import ...) is AF3-only and cannot be deployed until after the cross-major version jump completes — see the two-phase deployment note in each approach reference.

   Patterns Ruff does NOT catch: The manual checklist (airflow2-to-3-checklist.md) includes patterns invisible to Ruff, notably _TaskDecorator.output (section 15) and standalone Airflow CLI in BashOperator (section 16). Always run the manual scan even when Ruff reports zero findings.

2. Run manual scan using patterns from airflow2-to-3-checklist.md. Surface a metadata-DB warning (every approach, cross-major only): DAGs using metadata-DB access work on AF2 but break on AF3. Two categories:

   • ORM-access DAGs (settings.Session(), provide_session — checklist section 1): must be redesigned to the REST API (webserver in-VPC, or invoke-rest-api with its low rate limit) or dropped.
   • CLI-access DAGs (airflow db ... in BashOperator — checklist section 16): redesign to the MWAA CLI endpoint (preferred, simpler) or REST API or drop. The agent must verify command availability on the target version via cheat-sheet before committing to this path. Record each affected DAG and its category in the plan (Phase 4).

3. Check requirements.txt against the target version's constraints from mwaa-version-matrix.md.

4. Secrets backend detection: Check for secrets.backend config override in environment configuration. If using Secrets Manager or Parameter Store, note that variables/connections are external and do not need migration. If NO secrets backend is configured, surface this to the user at plan time: connection passwords will NOT migrate via the REST API (2.x omits, 3.x masks), so password-bearing connections must be moved to a secrets backend or have their passwords re-entered out of band on the target — and variable values / connection extra transit --body/CLI args and the audit log during migration. Record this in the plan (Phase 4).

5. Plugin inventory: If PluginsS3Path is set, list contents. Flag FAB/web-view plugins needing special attention.

6. Classify each DAG by upgrade difficulty: Simple, Moderate, Complex.

7. Group DAGs into batches (shared utilities in batch 0, simple first).