Optimizing Databricks SQL

Databricks Optimize (DBSQL)

A Databricks/Delta-platform overlay, not a standalone optimizer. The generic rewrite/analyze/verify pipeline (Steps 1, 5, 6) is the same pipeline altimate-code's native query-optimize skill runs — same tool calls, same composition. This skill's only job is the part query-optimize structurally can't do: detecting Databricks/Delta platform-specific issues (Z-ordering, table statistics, Photon UDF blocking) that need table metadata and Databricks docs knowledge, not just query text, and folding those findings in alongside the generic ones.

If query-optimize is installed in the current environment, prefer invoking it for Steps 1/5/6 and layer Steps 2–4's Databricks-specific findings on top of its output, the same way sql-review defers to sibling skills instead of duplicating them. If it isn't available (e.g. this skill is running standalone, outside altimate-code), fall back to calling the same tools directly as described below — the tool calls are identical either way, so behavior doesn't change based on which path ran.

No rewrite is reported as "optimized" until it clears the validation gate in Step 7 — that gate is a hard requirement, not a suggestion.

When to use this skill

Use when:

• The query is DBSQL (standalone or embedded in a notebook cell) and the question involves Databricks/Delta-specific behavior: Z-ordering, liquid clustering, table statistics, Photon, Unity Catalog, Predictive Optimization.

Do NOT use for:

• Generic, dialect-agnostic SQL optimization with no Databricks-specific angle → use query-optimize directly.
• SQL quality/safety linting unrelated to performance (injection, PII exposure, style) → use sql-review.
• PySpark DataFrame-level code (.collect(), .join(), UDFs used as Python objects) → deferred to a later phase, out of scope for now.

Before anything else — confirm the file path resolved correctly

Confirm the resolved file path matches what was requested, before doing anything else. If the user gave an explicit path (a /Workspace/... path, a repo path, any specific path), the tool used to read or write it must resolve to that exact path — not a same-named file found some other way. If the proper access tool fails, or the only way to find something is a generic filename search, stop and disclose it explicitly before analyzing or writing anything: state what path was requested, what (if anything) was found instead and where, and ask how to proceed. Never silently substitute a different file and continue as if it's the same one.

If no specific path was given at all (a generic request naming no file), a broader search — local files, a workspace listing — is a reasonable way to find a match. But the report must still state which path was actually used, so the reader can confirm it's the right one rather than discovering it was wrong after the fact.

Workflow

Step 1 — when the statement type can't be parsed

Confirmed twice in live testing: sql_analyze/altimate_core_check and related tools (altimate_core_validate, altimate_core_migration) cannot reliably parse DDL (CREATE TABLE ...) or MERGE statements — either failing outright or returning nothing, sometimes inconsistently across tools on the identical text. This is a real tool-coverage gap, not a signal that the statement has no findings. When it happens, fall back to live execution for grounding instead of treating the parse failure as "nothing to report": run the statement (or EXPLAIN/ DESCRIBE DETAIL against it) directly via sql_execute, and reason from what actually happens — a real parse/constraint error from the engine itself (e.g. NON_LAST_MATCHED_CLAUSE_OMIT_CONDITION, [MANAGED_TABLE_FORMAT]) is stronger, more specific evidence than any static tool's silence would have been anyway.

Step 3 — ground truth

When Step 0 found a live Databricks warehouse connection, don't call a query expensive on text alone — confirm it. schema_inspect the referenced tables (pass the warehouse name from Step 0), both to ground table-health claims in real metadata and to build the schema_context Step 5's rewrite/equivalence calls need for accurate table/column resolution. For any Delta table involved, run DESCRIBE DETAIL <table> via sql_execute to get real sizeInBytes/numFiles. Check statistics proactively here, on every referenced table — don't wait to discover a missing-stats finding as a side effect of running EXPLAIN in Step 4. Full methodology, the stats cost ladder, and the samples.* gotcha are in references/delta-table-health.md — read it before making any stats-related recommendation. Default to the cheapest check that answers the actual question: ANALYZE TABLE ... COMPUTE STATISTICS NOSCAN for size-only questions (e.g. broadcast eligibility), escalating to FOR COLUMNS (targeted) or FOR ALL COLUMNS only if genuinely needed. ANALYZE TABLE is a recommend-only action — see the confirmation rule below.

Step 4 — classify before deciding what to recommend

Predicate-level (function-wrapped filter, redundant cast, non-sargable comparison — expressible as alternate literal SQL text): run EXPLAIN FORMATTED on the original and check whether the plan's RequiredDataFilters/PushedFilters already reflect the fixed form.

• If yes — the runtime already rewrote it (Photon has been observed doing this automatically for a DATE(col)='X' predicate) — still recommend the explicit rewrite, but justify it on standards/portability grounds (not guaranteed on other runtimes/engines, small recurring compile-time cost), not as a performance claim, since EXPLAIN showed no plan difference on this data.
• If no — the plan still shows the unaddressed non-sargable form — the rewrite has a real, plan-evidenced performance basis.

Example — checking whether the runtime already rewrote the predicate:

Original: ... WHERE DATE(order_ts) = '2024-01-01'. Run EXPLAIN FORMATTED on that exact text and read the scan node's PushedFilters/RequiredDataFilters. (DATE() predicates are a tracked rewrite-engine gap — see references/rewrite-engine-gaps.md #2 for how to actually produce the rewrite text; this example is about which verdict the classification earns, not how to generate the fix.)

Case A — plan already shows the range form:

+- Relation sales.orders[...]
   PushedFilters: [order_ts >= 2024-01-01 00:00:00, order_ts < 2024-01-02 00:00:00]

Report it as: "No measured gain on this engine — EXPLAIN shows an identical plan either way. Recommended for portability: a different engine, an older Databricks Runtime, or a Photon-disabled session isn't guaranteed to constant-fold this the same way." Step 7's verdict for this one must be Correct, not faster — never Optimized.

Case B — plan still shows the function-wrapped form:

+- Relation sales.orders[...]
   PushedFilters: [isnotnull(order_ts)]
   -- DATE(order_ts) evaluated per-row in the Filter node above the scan

Same rewrite, different justification: the current plan evaluates DATE() per row and can't push the predicate into file pruning. This one can legitimately reach Optimized if Step 7's Tier 1/Tier 2 checks confirm an actual measured or structural improvement.

Why still rewrite it in Case A, if EXPLAIN shows no difference? Cases A and B produce the same recommended SQL text for different reasons, and reporting the wrong reason is worse than reporting none. Case A's rewrite is insurance against something this session can't observe — a future migration or runtime change; Case B's is a fix for something this session directly measured. Collapsing both into one "this is bad, fix it" verdict would overstate Case A's evidence, and it's exactly the kind of claim the Step 7 gate exists to catch — a rewrite labeled "Optimized" with no EXPLAIN/query.history difference behind it.

Strategy-level (join type, shuffle/Exchange placement, aggregation approach, broadcast decision — a plan-level choice with no equivalent SQL text): don't try to write literal SQL for this — there is none. Instead:

1. Check EXPLAIN's Optimizer Statistics section first. A cost-based choice (e.g. not broadcasting a small table) is often just downstream of missing/stale statistics (see dbsql-anti-patterns.md #2b). If so, recommend ANALYZE TABLE ... COMPUTE STATISTICS — this lets the optimizer adapt as data changes, which is more robust than freezing today's decision. Recommend it; don't run it (confirmation rule below).
2. Only if stats are already current and the choice still looks wrong, consider a hint (e.g. /*+ BROADCAST(t) */) as an explicit override — flag it as forcing a decision rather than fixing a root cause, and note it needs revisiting if data volume changes (a hint doesn't adapt).

Every proposal from either path — tool-generated or hand-authored — still goes through the Step 7 validation gate. Classification decides what's worth proposing, not whether it needs verification.

Step 5 — rewrite

1. Call altimate_core_rewrite(sql, schema_context, verify_equivalence: true) first, every time, regardless of past results — one call proposes a rewrite and proves it's semantically equivalent, partitioning results into verified-safe vs. review-before-applying.
2. If it returns nothing, check references/rewrite-engine-gaps.md for a matching known pattern and its documented workaround before hand-authoring from scratch — a known gap already has the specific trap and evidence-citation instructions worked out; don't re-derive them freehand.
3. If the finding doesn't match a tracked gap either, hand-author it grounded in what a tool actually showed this session (EXPLAIN, DESCRIBE DETAIL) — never from general SQL knowledge alone — then verify explicitly with altimate_core_equivalence(sql1: original, sql2: candidate, schema_context), since there's no altimate_core_rewrite call to attach verify_equivalence to for a hand-authored candidate.

Every path — tool-generated or hand-authored, tracked gap or novel — still goes through the Step 7 validation gate. An empty tool result doesn't mean there's nothing to propose, and nothing here is exempt from validation.

Step 6 — grade

Present: grade, findings (tagged generic-engine / Databricks-specific overlay / outside the defined pattern set — see below), and the verified rewrite if one exists. If a query has nothing safe to rewrite (e.g. an unfiltered SELECT * with no predicate to fix), say so — flag it, don't force a rewrite that changes semantics just to have something to show.

Beyond the defined pattern set. sql_analyze's full rule set, altimate_core_check's syntax/safety/PII pass, and dbsql-anti-patterns.md's catalog can all come back clean while something genuinely wrong is still visible in evidence already gathered in Step 3/4 (an EXPLAIN shape, a DESCRIBE DETAIL result, a query.history pattern) that just doesn't match either catalog's entries. Don't suppress that for lack of a matching rule — propose it, held to the same bar as everything else in this skill:

• Ground it in an actual tool call made in this session — the "Don't assert, verify" rule below applies identically; novelty is not an exception to it.
• Tag it explicitly outside the defined pattern set, distinct from the generic-engine and Databricks-overlay tags, so the report doesn't borrow catalog-level confidence it hasn't earned.
• Route any resulting rewrite through Steps 5–7 exactly like a catalogued finding — hand-authored, evidence-cited, equivalence- verified, subject to the Step 7 validation gate. Being uncatalogued is not a shortcut past validation; if anything it's the opposite, since there's no matched rule or prior worked example backing the claim.

If genuinely nothing is found and nothing evidence-grounded suggests itself either, that's the "nothing safe to rewrite" branch above — say so plainly, don't manufacture a finding just to have something to report.

Validation gate — nothing is "optimized" until this passes

Two independent, both-required checks, run via references/validation.md:

1. Correctness — the rewrite's output matches the original's output.
2. Performance — the rewrite measurably runs faster than the original, by more than the noise observed across repeated runs, not just numerically lower once.

Step 5's equivalence check (verify_equivalence: true, or a follow-up altimate_core_equivalence call for hand-authored fallbacks) checks logical/semantic equivalence; this gate additionally checks the rewrite is actually measurably faster against real execution — a different question equivalence doesn't answer. If either check fails, is inconclusive, or can't be run (no warehouse connection), say so plainly — "semantically unverified," "no measured improvement," "couldn't validate — here's why." Every claim about what the engine did (a plan showing a rewrite, a table's size) must trace back to an actual tool call made in this session, not general knowledge of how Spark/Databricks usually behaves.

Correctness checks are not composed ad hoc: decide scope first (window vs. full table — validation.md §Step 1, including the mandatory data-availability check before windowing), then use the tool-call recipe in validation.md §Step 2, not a freehand query. data_diff is reserved for cross-platform migration validation or localizing an already-found mismatch — see validation.md §Step 2 for why it isn't the default. Never hand-type checksum SQL from memory or by reasoning about what the logic "should" look like — follow the explicit tool-call recipe in validation.md instead.