Offline Packing Env Vars

Purpose / 用途

Use this skill when you set up or debug training-side sample packing for LLaVA-OneVision2 — i.e. when you need to decide which env vars to export in a training shell script (Stage-1 / Stage-1.5 / Stage-2) and want to understand why both OFFLINE_PACKING_BMR and OFFLINE_PACKED_DATA must be 1 to actually get padding-free attention.

在配置或调试 LLaVA-OneVision2 训练侧的样本 packing 时使用——比如要决定在训练 shell 脚本（Stage-1 / Stage-1.5 / Stage-2）中导出哪些环境变量，以及为什么必须 OFFLINE_PACKING_BMR=1 和 OFFLINE_PACKED_DATA=1 同时打开才能真正获得 padding-free 的 attention。

This skill is specifically for:

• Choosing the correct env var combination in training scripts
• Diagnosing cross-sample attention leakage in packed runs
• Understanding why cu_lengths is a dummy [[0]] in some runs and a real [B, P+1] tensor in others
• Avoiding the well-known OFFLINE_PACKING_VQA red herring (it is dead code)

Companion skill: cu-lengths-attention-flow covers the consumer side (how cu_lengths is fed into ViT/LLM attention). This skill covers the producer + gate side.

姊妹 skill：cu-lengths-attention-flow 讲消费端（cu_lengths 如何送入 ViT/LLM attention）。本 skill 讲生产端 + 开关。

TL;DR / 一句话总结

For packed training to work end-to-end, both env vars must be 1:

export OFFLINE_PACKING_BMR='1'   # data-layer gate: build real cu_lengths
export OFFLINE_PACKED_DATA='1'   # batch-layer gate: forward real cu_lengths to model

Setting only one is a silent bug. OFFLINE_PACKING_VQA is dead code; do not rely on it.

The Three Env Vars / 三个环境变量真相表

💡 The OFFLINE_PACKING_VQA red herring is the #1 source of confusion. Newcomers see it in shell scripts and assume it controls VQA packing. It does not. There is no third packing branch in task_encoder.py — only the BMR branch and the legacy captioning fallback.

💡 OFFLINE_PACKING_VQA 这个红鲱鱼是头号困惑源。新人在 shell 脚本里看到它，以为它控制 VQA packing。并不。task_encoder.py 里没有第三个 packing 分支——只有 BMR 分支和老的 captioning fallback。

Two-Stage Gate Architecture / 两段式 Gate 架构

Packing in this codebase is split into two orthogonal gates that must both fire. Understanding this is the whole point of the skill.

本仓库的 packing 拆成两个正交的 gate，必须都触发。理解这一点就是本 skill 的核心。

Gate 1 — Data Layer (OFFLINE_PACKING_BMR)

Where: aiak_training_llm/data/multimodal/task_encoder.py, inside the PackedCaptioningSample branch of the encoder dispatch (encode_sample ~line 186).

What it does:

• For each entry inside the packed sample (for idx in range(n_orig_sample):), if OFFLINE_PACKING_BMR == 1, it builds a MultiMixQASample carrying the full chat-format messages ({role: user, content: prompt}, {role: assistant, content: caption}) and routes it through encode_multi_mix_qa().
• If OFFLINE_PACKING_BMR != 1, it falls back to a plain CaptioningSample and encode_captioning() — losing the multi-turn / multi-image structure required for SFT.
• After the per-entry loop, regardless of the BMR flag, it calls self.pack_selected_samples(l_Qwen2VLImageTaskSample) (line 277), which constructs the per-sub-sample cumulative lengths cu_lengths = [0, len₁, len₁+len₂, ...] and attaches them to the resulting ImageTaskSamplePacked (line 473).

Net effect: enables the correct per-sub-sample encoding and produces real s.cu_lengths on each sample.

作用：启用正确的逐子样本编码，并在每个样本上产出真正的 s.cu_lengths。

⚠️ Even with BMR off, pack_selected_samples still attaches a cu_lengths tensor to the sample. But the sub-samples were encoded via the wrong path (legacy captioning), so the resulting boundaries don't match what the LLM actually sees. BMR off + PACKED_DATA on is a hidden corruption, not just a missing-feature.

⚠️ 即使 BMR 关掉，pack_selected_samples 仍然会给样本挂上 cu_lengths 张量。但子样本走的是错误的编码路径（老 captioning），结果 boundary 和 LLM 实际看到的 token 序列对不上。BMR 关 + PACKED_DATA 开是隐性数据损坏，不只是缺特性。

Gate 2 — Batch Layer (OFFLINE_PACKED_DATA)

Where: aiak_training_llm/data/multimodal/task_encoder.py:359-365, inside batch() (the collate function).

What it does:

# Cumulative sample lengths are needed for packing, otherwise use dummy values.
cu_lengths = torch.tensor([[0]], dtype=torch.int32)
max_lengths = torch.tensor([[0]], dtype=torch.int32)

if self.is_packing_enabled or int(os.environ.get("OFFLINE_PACKED_DATA", 0)) == 1:
    cu_lengths = torch.stack([s.cu_lengths for s in samples])
    max_lengths = torch.tensor([s.max_length for s in samples], dtype=torch.int32)

• Default: emit a dummy cu_lengths of shape [1, 1] containing only [[0]].
• When OFFLINE_PACKED_DATA == 1 (or the energon online-packing flag is set): stack the real per-sample cu_lengths produced by Gate 1 into shape [B, P+1].

Net effect: decides whether the consumer (model forward) sees real packing offsets or a dummy that says "no packing".

作用：决定消费端（模型 forward）看到的是真实的 packing 偏移，还是一个表示"没有 packing"的 dummy。

Why both gates must fire / 为什么必须两个都开

The consumer side at aiak_training_llm/train/pretrain/pretrain_llava_onevision2.py:153-168:

packed_seq_params = None
...
if cu_lengths.shape == torch.Size([1, 1]):
    pass                        # treat as not packed
else:
    assert cu_lengths.shape[0] == 1, "micro-batch-size must be 1 for packing"
    packed_seq_params = PackedSeqParams(
        qkv_format="thd",
        cu_seqlens_q=cu_lengths[0],
        cu_seqlens_kv=cu_lengths[0],
        ...
    )

So:

🔥 The "BMR=1, PACKED_DATA=0" footgun is the most dangerous combination. Training does not crash. Loss curves look reasonable. But every sub-sample in a packed sequence can attend to every other sub-sample's prefix. Use this skill's TL;DR snippet to avoid it.

🔥 "BMR=1, PACKED_DATA=0" 这个组合最危险。训练不会挂，loss 曲线看着也正常。但 packed 序列里每个子样本都能 attend 到别的子样本的 prefix。用本 skill 顶部的 TL;DR 片段避开它。