Databricks Bundle Medic

Databricks Bundle Medic

The deploy + infrastructure spine of the pack. Databricks Asset Bundles (DAB) is immature tooling — the replacement for the deprecated dbx, with a moving bug list across CLI versions — and the infrastructure operations around a deploy (encryption keys, networking, workspace config) bite production platform teams hardest. This skill turns the five worst deploy-time foot-guns into deterministic, reversible operations.

Overview

Five pains, all from the pack's deploy-ops research:

Asset Bundles (DAB). D4 — databricks bundle bind does NOT yet support UC catalogs or external locations (databricks/cli#4842), so existing UC resources cannot come under bundle management without hand-editing terraform.tfstate (hostile) or destroy-and-recreate (impossible with dependent tables). D5 — "unexpected EOF reading terraform.tfstate" on every redeploy after the first (databricks/cli#4986), which bricks the bundle until you switch to DATABRICKS_BUNDLE_ENGINE=direct. D6 — the schema GRANT-ordering bug (databricks/cli#4573): the first deploy to a fresh workspace fails with "User does not have CREATE TABLE on Schema", but the second succeeds because the first applied the grants before dying.

Deploy-time infrastructure. D8 — rotating a workspace's customer-managed key (CMK) requires terminating every cluster, instance pool, and SQL warehouse first: a hard maintenance window. D9 — the PrivateLink trap: enabling PrivateLink for the control plane is mistaken for "all traffic is private," but the data plane's S3 / STS / Kinesis calls still traverse the NAT, billing NAT-processing + cross-AZ transfer until each gets its own VPC endpoint.

The two hooks (this is the pack's only two-hook skill). A PreToolUse hook (hooks/bundle-deploy-guard.py) runs before every databricks bundle deploy: it validates the bundle's local terraform state parses as JSON, caches a timestamped known-good backup as a recovery escape hatch, and warns loudly if the state is corrupt or has shrunk (the D5 signature). A PostToolUse hook (hooks/bundle-grant-retry.py) watches a deploy's output and — ONLY on the exact D6 "does not have … on Schema" signature — adds context recommending one retry, with the reason. Both are advisory: the pre-hook never blocks a deploy, and the post-hook never masks a real error — it surfaces the diagnosis and stops recommending retries if the same failure persists.

Deterministic work lives in scripts/; deep knowledge in references/. The import-uc-resource-to-bundle.py D4 workaround is self-deprecating — it exists only until #4842 closes and says so. Control-plane state comes from the databricks-workspace-mcp server (external_locations_list, storage_credentials_list) or the CLI; advisory-mode fallback accepts pasted input.

Prerequisites

• Databricks CLI authenticated, plus terraform and jq on PATH — for the bundle, import, and state operations.
• databricks-workspace-mcp registered (optional) — for external_locations_list and storage_credentials_list. Absent → the CLI (databricks external-locations list) or advisory mode on pasted input.
• Account-level OAuth M2M (service principal) for CMK rotation — the D8 operations are account-scoped; a workspace PAT is insufficient. AWS/Azure/GCP CLI for the cloud-side key + VPC-endpoint work.
• The two hooks are plugin-level — installed with the pack. The pre-hook is silent unless a state looks corrupt; the post-hook is silent unless the exact D6 error fires.

Instructions

Pick the flow by symptom. Always name the exact error string / issue id — unexpected EOF reading terraform.tfstate (#4986), does not have CREATE TABLE on Schema (#4573), the bundle bind UC gap (#4842) — an operator greps for those.

Step 1: A bundle deploy that fails on state (D5)

"unexpected EOF reading terraform.tfstate" after the first deploy is #4986. The bundle-deploy-guard PreToolUse hook already cached a known-good backup; restore it, or switch engines:

DATABRICKS_BUNDLE_ENGINE=direct databricks bundle deploy -t "$TARGET"

Engine tradeoffs + the migration steps: ${CLAUDE_SKILL_DIR}/references/bundle-engine-tradeoffs.md.

Step 2: First deploy fails "does not have CREATE TABLE on Schema" (D6)

This is the transient GRANT-ordering bug (#4573). The bundle-grant-retry PostToolUse hook flags it; re-run the SAME deploy exactly once — the failed first pass already applied the grants. If the identical error persists after one retry, it is NOT this transient — treat it as a real missing privilege and stop retrying.

Step 3: A UC resource that will not bind (D4)

databricks bundle bind cannot take a UC catalog or external location yet (#4842). Generate a review-first Terraform import plan (never hand-edit state):

python3 "${CLAUDE_SKILL_DIR}/scripts/import-uc-resource-to-bundle.py" \
  --type external_location --name raw_zone --resource-key raw_zone_loc

The three workarounds, ranked by risk: ${CLAUDE_SKILL_DIR}/references/uc-resource-binding-workarounds.md.

Step 4: Rotate a customer-managed key (D8)

CMK rotation needs the whole workspace drained. Inventory the running compute (via external_locations_list / the CLI), then plan the drain — dry-run by default:

databricks clusters list --output json > /tmp/inv-clusters.json   # + warehouses, pools
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json          # dry-run
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --inventory inv.json --execute --manifest drain.json
# ... rotate the CMK per cloud, then:
python3 "${CLAUDE_SKILL_DIR}/scripts/drain-workspace.py" --resume drain.json

Per-cloud playbooks (AWS/Azure/GCP): ${CLAUDE_SKILL_DIR}/references/cmk-rotation-by-cloud.md.

Step 5: Audit the PrivateLink cost leak (D9)

PrivateLink covers the control plane only. Audit the data-plane VPC for the S3/STS/Kinesis endpoints and emit remediation Terraform for any that are missing:

python3 "${CLAUDE_SKILL_DIR}/scripts/audit-vpc-endpoints.py" \
  --present s3 --vpc-id vpc-abc --region us-east-1 --emit-terraform

The full per-service leak/fix map: ${CLAUDE_SKILL_DIR}/references/cost-leak-map.md.