Compute Mamba Ratio

Optimal hybrid dual-pool ratio (--mamba-full-memory-ratio)

A hybrid model (attention layers + linear-attention layers — the recurrent-state family: Mamba/SSM, GDN, KDA, etc.) splits serving memory into two independently-budgeted pools, fixed once at startup:

• state pool (the linear-attention recurrent state) → caps concurrency (hard: whole slots, worst-case reserved, fail-loud)
• full-KV pool (attention KV) → caps context × concurrency (soft: paged, over-committable via retraction)

--mamba-full-memory-ratio r splits the post-weight budget: mamba_budget = rest · r/(1+r), i.e. mamba_budget : kv_budget = r. This skill picks the r (or the pin---max-mamba-cache-size alternative) at which neither pool bottlenecks first for the user's workload.

The formula

r*  =  (S + D) · token_equiv · dcp_size / L
token_equiv  =  state_bytes_per_slot / kv_bytes_per_token

• L = average context length per request (input + output tokens)
• token_equiv = full-KV token-equivalent of one state slot
• S = state slots per running request (cache-strategy dependent, table below)
• D = --speculative-num-draft-tokens (0 if NOSPEC); each running req carries D extra intermediate states
• dcp_size = --dcp-size (1 without DCP). DCP shards the per-rank KV by dcp_size, so KV gets ~dcp_size× cheaper per request → the balance shifts that much toward the state pool.

r is dimensionless (just the split). To also predict the actual concurrency you need rest (below).

Inputs to collect from the user

1. L — average context (input + output) in tokens.
2. The two per-GPU byte constants — one of:
   • (a) measured (preferred, exact) — from one boot log at any ratio:
     • Mamba Cache is allocated. ... ssm_state size X GB with max_mamba_cache_size: N → state_bytes_per_slot = X / N
     • KV Cache is allocated. #tokens: M, KV size: Y GB → kv_bytes_per_token = Y / M
   • (b) derived — model arch (linear-layer count + state dims d_state/d_conv/heads/head_dim; attention type + KV dims: MLA latent dim, or GQA kv_heads·head_dim·layers) × the dtypes below.
3. S — from --mamba-radix-cache-strategy, the overlap scheduler, and SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK (table below).
4. D — --speculative-num-draft-tokens (0 if NOSPEC; also 0 when ReplaySSM spec-verify is enabled — see caveats).
5. dcp_size — --dcp-size (1 if no DCP). If DCP and spec with a replicated draft KV, apply the draft caveat (below).
6. KV dtype (bf16 / fp8) and ssm dtype (fp32 / bf16) — they set the two byte constants (see the token_equiv 2×2).
7. (only to also predict the clamp, not just r) rest = per-GPU memory − weights at the chosen --mem-fraction-static. Read avail mem after Load weight end, or Memory pool end. avail mem + pool sizes, from the boot log.

S (state slots per running request), set by --mamba-radix-cache-strategy

All strategies keep prefix caching on; they differ in the track buffer that snapshots chunk-boundary state under the overlap scheduler. S = base + ping-pong, where base is 3 (live state + radix retention/COW headroom), and ping-pong = 2 (overlap on, non-lazy) / 1 (lazy, or overlap off) / 0 (no track buffer at all). This mirrors kv_cache_configurator._calculate_mamba_ratio; read it there if a release moves the constants.

SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1 takes the base to 2, since the decode-time skip frees one resident slot per running request. no_buffer is the exception and stays at an effective 3: it adds that slot straight back, because its binding limit is the prefill-to-decode peak, which a decode-time change does not shrink.

The skip is a real concurrency lever, not a rounding detail: at the default strategy it takes S from 5 to 4, so the same state budget admits 25% more requests and r* drops by the same factor.

Overlap counts as off when --disable-overlap-schedule is passed or --pp-size > 1 (pipeline parallelism turns it off for you). So a PP deployment left on the default strategy sits on the S = 4 row, not S = 5 — using 5 there overstates the state cost and starves KV.

token_equiv — compute it from the two byte constants

token_equiv = state_bytes_per_slot / kv_bytes_per_token. Both bytes are per-model, per-GPU — read them from a boot log (or derive from the arch + dtypes) and divide. That's the whole thing; the dtype effects below are just for a quick mental estimate.

The recurrent state has two parts: state_bytes = SSM_bytes + conv_bytes. The SSM part scales with --mamba-ssm-dtype; the conv part is always bf16 (SGLANG_MAMBA_CONV_DTYPE, fixed). How the dtype knobs move token_equiv:

• fp8 KV → ×2 (universal, exact): fp8 is 1 byte vs bf16's 2, so kv_bytes_per_token halves.
• fp32 ssm vs bf16 ssm → ×≈2: only the SSM tensor doubles, conv stays fixed, so the exact factor is (2·SSM + conv)/(SSM + conv) — just under 2. conv is usually small relative to SSM, so estimate ≈2 (e.g. one measured model: conv ≈7% of state → factor ≈1.9). Only bother with the exact split if a model's conv is unusually large.
• fp32 ssm + fp8 KV → ×≈4.

For an actual number, don't apply factors — read state_bytes_per_slot and kv_bytes_per_token for your model+dtype from the boot log and divide.

The GB the boot log prints for both pools is GiB (1024³). Convert both the same way and token_equiv is unaffected either way, since it is a ratio — but get it right before comparing an absolute byte figure against your own log.

Worked example (one measured model, TP8, fp32 ssm + fp8 KV), read off one boot log:

Mamba Cache is allocated. max_mamba_cache_size: 257, conv_state size: 0.46GB, ssm_state size: 13.04GB
KV Cache is allocated. dtype: torch.float8_e4m3fn, #tokens: 1167552, KV size: 15.03 GB

state_bytes_per_slot = (0.46 + 13.04) GiB / 257 ≈ 56.4 MB, kv_bytes_per_token = 15.03 GiB / 1167552 ≈ 13.8 KB → token_equiv ≈ **4080**.