Databricks Uc Migration Pilot

Databricks Unity Catalog Migration Pilot

Pilots a Hive Metastore → Unity Catalog migration end to end: it audits which tables can migrate (and why the rest cannot), turns that audit into a dependency-ordered execution plan, traces UC's two-level permission model when access breaks, and picks an environment-isolation pattern for the one-metastore-per-region constraint.

Overview

The September 30, 2026 forcing function. Databricks is deprecating the legacy Hive Metastore. On the cutover date the hive_metastore catalog goes read-only — no new writes, no schema changes — and stays available only for read-through while you migrate. Everything that writes to HMS (jobs, DLT pipelines, dashboards, external BI) must be pointed at a Unity Catalog table before then, or it breaks. This is not optional and it is not fast: the blockers below (legacy storage schemes, DBFS-root managed data, DENY-based ACLs) each require a physical data move or a grant rewrite, so the work is measured in weeks, not an afternoon.

⚠️ Verify the exact date and scope against your workspace's current deprecation notice (Databricks account console → Previews / Announcements). Deprecation dates move; this skill's job is to get you ready well ahead of whatever the live date is — treat 2026-09-30 as the planning deadline, not an excuse to wait for it.

This skill does four things, in order, and each is deterministic where it can be:

1. Detect — scripts/audit-hms-readiness.py classifies every HMS table by its storage URI into READY (cloud-native, UC-governable), BLOCKED (a named un-migratable condition), or ORPHAN (a dangling HMS row pointing at deleted storage). The LLM never eyeballs a URI; the script owns the verdict.
2. Plan — the migration-planner subagent turns the readiness CSV into a per-table plan (SYNC / DEEP CLONE / rewrite / skip) ordered by dependencies so a view never migrates before its base tables.
3. Trace — the uc-permission-tracer subagent walks UC's two-level access model to answer "why can't user X read this?" in one pass — replacing ~90 minutes of doc-spelunking per ticket.
4. Isolate — /uc-env-pattern-picker recommends one of four dev/test/prod isolation patterns under the single-metastore-per-region limit.

It is architecturally distinct from the v1 databricks-upgrade-migration and databricks-migration-deep-dive skills: those narrate migration in prose. This one runs a real readiness audit, emits a dependency-ordered plan, and traces live permissions. Deep, load-on-demand knowledge lives in references/; the arithmetic-and-classification lives in scripts/.

Prerequisites

The three role grants below are the hard dependencies and the most common reason the pilot stalls mid-flow. The skill checks the role chain first (Step 1) and reports exactly what is missing before doing any work.

• Account-admin — required to enable the UC system schemas (system.information_schema, system.access) the audit reads, and to create the metastore/assign it to workspaces. Enable at the account level, not per workspace.
• Metastore-admin — required to CREATE CATALOG, run GRANT on UC objects, and create external locations + storage credentials. Migration is a sequence of metastore-admin operations.
• Cloud IAM read access (AWS s3/IAM, Azure, or GCP) — required to diagnose a storage-credential failure (the D2 case: a table is cloud-native but the UC storage credential's IAM role cannot assume the bucket). Bash(aws:*) is allowed for read-only IAM inspection; the skill never mutates cloud IAM.
• Databricks CLI authenticated (databricks auth login, or the DATABRICKS_HOST + DATABRICKS_TOKEN env pair) and jq for parsing. Enumeration and grant SQL run through the CLI Statement Execution API.
• DATABRICKS_WAREHOUSE_ID set to a running SQL warehouse — every statement-execution call requires it.

Authentication. All auth comes from the environment: the CLI's DATABRICKS_HOST + DATABRICKS_TOKEN (or databricks auth login), and read-only cloud credentials from the standard provider chain (aws sts get-caller-identity to confirm). No secrets are hardcoded.

Instructions

The pipeline is verify role chain → detect readiness → plan → migrate → trace access → isolate envs. Do them in order; Step 1 is a hard gate.

Step 1: Verify the Role Chain (hard gate — fail fast, not mid-migration)

Before touching data, confirm the running principal holds the roles the migration needs. A missing role surfaces here, not three tables into a SYNC.

# Am I authenticated, and as whom?
databricks current-user me | jq -r '.userName'

# Metastore-admin check: only a metastore admin can read every grant. If this
# errors with PERMISSION_DENIED, the principal is not metastore-admin.
databricks api post /api/2.0/sql/statements --json "$(jq -n --arg wh "$DATABRICKS_WAREHOUSE_ID" \
  '{warehouse_id:$wh, wait_timeout:"30s",
    statement:"SELECT 1 FROM system.information_schema.catalog_privileges LIMIT 1"}')" \
  | jq -r '.status.state, (.status.error.message // "ok")'

Report the role chain status plainly: account-admin (needed to enable system schemas), metastore-admin (needed for grants + external locations), cloud IAM read (aws sts get-caller-identity). If any is missing, name who must grant it and STOP — do not start an audit you cannot finish.

Step 2: Enable & Confirm the System Schemas

The readiness audit reads system.information_schema; permission tracing reads system.access. Those schemas are individually gated — enabling one does not enable the others — behind an account-level flag AND a metastore-admin grant chain. If they are not enabled, run the bundled idempotent enabler (account-admin auth; a workspace PAT is rejected up front) to enable each schema and grant a group USE CATALOG system + USE SCHEMA + SELECT in one pass:

python3 "${CLAUDE_SKILL_DIR}/scripts/enable-system-schemas.py" \
  --account-id "$DATABRICKS_ACCOUNT_ID" --metastore-id "$METASTORE_ID" \
  --grant-to data-governance --dry-run   # drop --dry-run to apply

Then confirm a UC metastore is attached to this workspace (databricks metastores current) — without one there is nowhere to migrate to. The full two-layer access model (account-admin enables, metastore-admin grants, neither inherits SELECT) is in ${CLAUDE_SKILL_DIR}/references/system-tables-access-model.md.

Step 3: Detect — Run the Readiness Audit

Enumerate the HMS schema(s) and classify every table. The script owns the verdict; feed it either live (--live SCHEMA) or the enumerated rows as JSON.

OUT="${OUT:-$(pwd)/uc-migration-out}" && mkdir -p "$OUT"

# Live enumeration (needs DATABRICKS_WAREHOUSE_ID + an authenticated CLI):
python3 "${CLAUDE_SKILL_DIR}/scripts/audit-hms-readiness.py" \
  --live sales --summary --out "$OUT/readiness-sales.csv"

Every row lands in one of three buckets — READY (cloud-native path, SYNC-able), BLOCKED (a named condition), ORPHAN (dangling HMS entry, cleanup not migration). When a BLOCKED class appears, load ${CLAUDE_SKILL_DIR}/references/uc-migration-blockers.md for that class's canonical error, the physical relocation procedure, and the per-cloud variant. Do not skip the CLONE-drops-history gotcha in that file — a shallow CREATE TABLE ... CLONE breaks time travel post-migration; use DEEP CLONE.

Step 4: Plan — Dependency-Ordered Migration Plan

Hand the readiness CSV to the migration-planner subagent (see agents/migration-planner.md) with the org's catalog naming convention. It emits a numbered execution order with per-step rationale and the right verb per table (SYNC for ready Delta externals, DEEP CLONE/rewrite for blocked, CREATE VIEW after base tables, skip for orphans). Dependency ordering is the point: a view never migrates before the tables it reads.

Invoke it via /audit-hms-readiness (which runs Step 3 then routes here) or directly by handing the subagent the CSV path.

Step 5: Migrate — Execute the Plan

Walk the plan top to bottom:

• READY (Delta external, cloud path): SYNC TABLE <uc_cat>.<schema>.<table> FROM hive_metastore.<schema>.<table> — metadata-only, no data copy. SYNC SCHEMA does a whole schema when every table is ready.
• BLOCKED (legacy scheme / DBFS root): relocate the data first (DEEP CLONE for Delta, CREATE TABLE AS SELECT for non-Delta) to a UC-governed cloud path, register an external location + storage credential, then create the UC table at the new path. Full procedure per class in uc-migration-blockers.md.
• LEGACY_TABLE_ACL: the data may be ready, but the DENY-based grants cannot auto-map to UC's allow-only model — re-author them (Step 6 traces them).

Re-run Step 3 after a batch to confirm the migrated tables drop out of BLOCKED.