Databricks Streaming Guardian

Databricks Streaming Guardian

The data-ops spine of the pack. Delta Lake, Liquid Clustering, Structured Streaming, and DLT each ship a different set of foot-guns that fire most visibly when production data flows through them at scale — and most of them are documented platform decisions that surprise engineers, not bugs. This skill's job is friction at trigger time (a hook that blocks the genuinely-irreversible op) plus deterministic recovery when something already broke.

Overview

Twelve foot-guns, grouped by the surface that triggers them. Eleven are owned outright (D01–D10, D12); the twelfth — D11, DLT rebuild cost — is shared with databricks-cost-leak-hunter: this skill checks the rebuild cost as part of pre-refresh safety, that skill owns ongoing cost optimization.

Delta write conflicts. D01 ConcurrentDeleteDeleteException — a manual OPTIMIZE colliding with auto-compaction, which is silently enabled on any table touched by MERGE/UPDATE/DELETE. D02 ConcurrentAppendException after moving to Liquid Clustering — LC keeps file-set-level writer conflicts; a fan-out MERGE breaks unless its predicate is narrowed to the clustering keys.

Streaming + checkpoint. D03 DELTA_FILE_NOT_FOUND_DETAILED — VACUUM deletes files the checkpoint pins to. D04 silent checkpoint corruption / reset to batch 0. D05 RocksDB state-store off-heap OOM (the heap looks fine while off-heap state pins multi-GB). D12 DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE — CREATE OR REPLACE mints a new UUID and kills every active consumer.

Migration + evolution. D06 Liquid-Clustering migration's hidden full-rewrite cost + downstream partition-predicate breakage. D07 time travel breaking silently when VACUUM crosses the retention boundary. D10 Auto Loader UnknownFieldException stopping the stream on every new column.

DLT. D08 the @dlt.table thread race (out-of-order registration). D09 full refresh silently dropping data from a non-replayable source. D11 the rebuild cost multiplier (checked before a full refresh; ongoing DLT cost is databricks-cost-leak-hunter's job).

The hook (AP02/AP06 — this pack's only blocking hook). A PreToolUse hook (hooks/streaming-guard-hook.py) intercepts a Bash command that runs DROP TABLE, CREATE OR REPLACE TABLE, or VACUUM against a table and — only when it confirms via system.streaming.query_progress that an active stream reads that table — blocks it with a message naming the consumers and the pain. It is precise by design: it matches only real SQL-execution surfaces (never a git commit mentioning "drop table"), and it fails open — if it cannot verify consumers, it allows rather than false-block. Blocking is reserved for the genuinely irreversible.

Deterministic work lives in scripts/; deep knowledge in references/; the Liquid-Clustering predicate rewrite in the merge-rewriter subagent. Two data planes: the databricks-workspace-mcp control plane (cluster/pipeline events) and the CLI Statement Execution API for system.* reads. Either absent → advisory mode on pasted input.

Prerequisites

• databricks-workspace-mcp registered — for clusters_events (RocksDB OOM correlation) and pipelines_get (DLT event log). Absent → advisory mode.
• Databricks CLI authenticated + jq, and DATABRICKS_WAREHOUSE_ID set — for the system.streaming.query_progress reads the hook and recovery flows use. The hook fails open (allows) if these are absent, so it never false-blocks.
• The hook is a plugin-level PreToolUse hook — it runs on Bash commands once the pack is installed. It is silent on everything except a confirmed-unsafe destructive op.

Instructions

Pick the flow by symptom. Always name the exact, searchable Databricks error string — ConcurrentAppendException, ConcurrentDeleteDeleteException, DELTA_FILE_NOT_FOUND_DETAILED, DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE, UnknownFieldException — even when the user paraphrases it or gives a short form; the full code is what an operator greps logs and docs for.

Step 1: Before a destructive op (the hook does this automatically)

Running DROP TABLE / CREATE OR REPLACE TABLE / VACUUM on a table? The hook checks for active streaming consumers first and blocks if any exist. To check manually, query system.streaming.query_progress for a stream whose source_description names the table. If consumers exist: do NOT CREATE OR REPLACE (use ALTER/in-place — D12) and do NOT VACUUM below the consumers' checkpoint lag (D03/D07). See ${CLAUDE_SKILL_DIR}/references/checkpoint-recovery.md.

Step 2: A Delta write conflict (D01, D02)

• ConcurrentDeleteDeleteException (D01) — before a manual OPTIMIZE, probe the table for auto-compaction:

  bash "${CLAUDE_SKILL_DIR}/scripts/pre-optimize-check.sh" --table main.sales.orders

  If it reports COLLISION RISK, don't run manual OPTIMIZE (or disable auto-compaction first). Details: ${CLAUDE_SKILL_DIR}/references/concurrency-conflicts.md.

• ConcurrentAppendException on a Liquid-Clustering table (D02) — hand the failing MERGE to the merge-rewriter subagent; it fetches the target's clustering keys via DESCRIBE DETAIL and narrows the ON predicate so writers touch disjoint file sets.

Step 3: A broken streaming source (D03, D04, D12)

Map the symptom to the failure class and its exact error code, then get the recovery tier from the decision tree:

• file-not-found → DELTA_FILE_NOT_FOUND_DETAILED (VACUUM deleted pinned files — D03)
• uuid-changed → DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE (CREATE OR REPLACE minted a new UUID — D12)
• checkpoint-reset → silent batchId regression / checkpoint corruption (D04)
• transient → a restartable blip with an intact checkpoint

python3 "${CLAUDE_SKILL_DIR}/scripts/recover-streaming-source.py" \
  --failure file-not-found --time-travel yes    # or uuid-changed / checkpoint-reset / transient

It echoes the canonical error code and recommends SAFE_RESTART / REPROCESS_FROM_OFFSET / RESTORE_FROM_TIME_TRAVEL / FULL_RESET_BACKFILL with the data-loss tradeoff stated up front. Name that full code in your answer — not just the short class. The full three-tier reasoning is in ${CLAUDE_SKILL_DIR}/references/checkpoint-recovery.md.

Step 4: RocksDB state-store OOM (D05)

A driver/executor OOM while the JVM heap looks healthy points at off-heap RocksDB state. Correlate the OOM to state size with clusters_events, then bound the memory and enable changelog checkpointing per ${CLAUDE_SKILL_DIR}/references/rocksdb-state-store-tuning.md.