Debug Multi Launch Merge

Purpose

When opt-merge-multi-launch-kernels produces a fused ELF and aircc.py rejects it for hardware-resource reasons, this skill identifies which of the 6 documented constraints was hit. Use as a diagnostic decision tree, not a mechanical fix-applier — confirm the trigger matches your stderr before applying the corresponding remedy.

Knowledge base references

• programming_examples/kernel_registry/details/<Kernel>_bf16.md — per-kernel constraints / placeability notes (the authority for that kernel's hard limits): BD inner-dim 1024, GEMV K-DMA repeat, combined channel reads, L2 cap — the constraints the merge blockers below map back to (for GEMV-specific limits see details/GEMV_bf16.md)
• programming_examples/kernel_registry/supported_kernels.md — per-kernel constraints + silent-corruption traps (the merge constraints in context of each leaf kernel)
• programming_examples/llms/llama32_1b/multi_launch_builder/ — working fused-ELF builders to diff a failing merge against (what DOES merge, and the FA-stays-separate boundary)

Trigger pattern

This skill matches when ANY of these appear in aircc.py stderr:

• buffer descriptor / BD / out of buffer descriptors (BD exhaustion)
• channel routing / cannot route / channel allocation failed
• herd shape mismatch / herd dimension
• aie.tile location conflict
• airrt.herd_load not found / undefined symbol
• stride must be 1 for BF16 / sub-32b types

If your error doesn't match any of these patterns, the issue is likely NOT a multi-launch resource collision — invoke superpowers:systematic-debugging instead.

Diagnostic hypothesis tree

For each hypothesis: confirm the symptom-fit, then apply the listed remedy. Don't apply remedies speculatively.

Hypothesis 1: BD exhaustion

Symptom-fit: stderr mentions buffer descriptor / BD / out of buffer descriptors. Often triggered by a non-1024-aligned dim (see the kernel's details/<Kernel>_bf16.md placeability notes) because the DMA auto-splits into many shim BDs that exhaust the pool.

Diagnostic: identify which model dim is non-1024-aligned (e.g. emb_dim=1536). Check the merged ELF's launch count and whether each launch's DMA pattern is BD-friendly (see the kernel's details/<Kernel>_bf16.md).

Remedy (in order of preference):

1. Pad the offending dim to 1024-aligned via GQA-aware reindexed padding (technique in phase-2-single-block-validation Step 2 + the kernel's details/<Kernel>_bf16.md placeability notes)
2. If padding is infeasible: un-merge the most recently added launch from the merged ELF; recompile
3. Switch to kernel-first split-ELF path (more, smaller ELFs)

Hypothesis 2: Channel routing congestion

Symptom-fit: stderr mentions channel routing / cannot route / channel allocation failed. Adjacent launches use overlapping channel IDs that cannot coexist physically on the AIE2P fabric.

Diagnostic: print the merged MLIR (make print or --print-module-only); search for air.channel declarations and check IDs across the merged launches.

Remedy: rename channels in one of the offending launches. The _rename_all(text, prefix=...) helper in programming_examples/llms/llama_kernel_builder/stitching.py already prefixes every SSA name (including channels) per-kernel — confirm your stitching code used distinct prefixes per sub-kernel. If two sub-kernels were stitched with the same prefix, that's the bug. Re-run stitching with distinct prefixes; recompile.

Hypothesis 3: Herd shape conflict

Symptom-fit: stderr mentions herd shape mismatch / herd dimension. Two launches need different herd shapes (e.g., [8,4] for GEMM and [8,1] for RMSNorm) that the placement pass can't reconcile in one segment.

Diagnostic: identify the herd sizes=[N, M] of each sub-kernel. GEMM is typically [8, 4], GEMV/RMSNorm/RoPE/Eltwise are [8, 1].

Remedy: in practice these CAN coexist in one segment when both fit the chip's 8 columns (8 cols × max(M_i) = chip rows). If the compiler still rejects, the cleanest path is to keep the offending kernels as separate XRT calls (don't merge them). This isn't a regression — it's a scoping decision. Document in TODO and accept the slightly higher dispatch overhead.