Eval bug: DFlash on Qwen3.6-35B-A3B (qwen35moe) hard-crashes at draft load / first decode on single CUDA GPU (v0.3.2)

[Andgihat]

Name and Version

llama-server --version

version: 1 (a620cbd)
built with MSVC 19.44.35227.0 for Windows AMD64

Self-built from branch v0.3.2 (commit a620cbd), -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CUDA_FORCE_MMQ=ON -DGGML_CUDA_FA_ALL_QUANTS=ON, CUDA Toolkit 12.8. system_info confirms a healthy native build: ARCHS = 1200 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1.

Operating systems

Windows

GGML backends

CUDA

Hardware

Single GPU: NVIDIA RTX 5060 Ti 16 GB (Blackwell, sm_120) + Ryzen 5 9600X + 32 GB DDR5.

Models

• Target: Qwen3.6-35B-A3B, IQ3_S (qwen35moe architecture).
• Drafter: Anbeeld/Qwen3.6-35B-A3B-DFlash-GGUF → qwen36-35b-a3b-dflash-Q4_K_M.gguf (bit-identical to the HF repo file; 291,761,856 bytes).

Problem description & steps to reproduce

Flat DFlash on Qwen3.6-35B-A3B (qwen35moe) hard-crashes (silent process exit, no GGML_ASSERT/CUDA error printed) on a single CUDA GPU. The crash happens in the DFlash path right after the drafter is picked up:

load_model: loading draft model '...qwen36-35b-a3b-dflash-Q4_K_M.gguf'
load_model: DFlash draft model will use target output device CUDA0 by default; pass --spec-draft-device to override
<process exits, nothing further>

The architecture is routed through the new recurrent-backup path:

speculative_: flat DFlash target architecture qwen35moe will use recurrent-only backup cells; keeping attention streams at n_parallel=1

The exact stage shifts with flags but DFlash always dies:

• Stock-style flags (--n-gpu-layers 999, no --kv-unified): crash at draft load (as above).
• Quickstart-style flags (--kv-unified --no-mmap --mlock --no-host, -ngl all): gets past load (model loaded / server is listening), then hard-crashes on the first decode — last line is slot get_availabl: ... adaptive dm: reset state for LRU slot selection, consistent with the recurrent backup / decode path.

Minimal reproduce (the most deterministic case — crashes at draft load every time):

llama-server -m Qwen3.6-35B-A3B-IQ3_S.gguf \
  --spec-draft-model qwen36-35b-a3b-dflash-Q4_K_M.gguf \
  --spec-type dflash --spec-dflash-cross-ctx 1024 --spec-draft-ngl all \
  --ctx-size 32768 --n-gpu-layers 999 --n-cpu-moe 4 --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -b 1024 -ub 512 -np 1 --jinja

Isolation (what I ruled out)

The crash is specific to the DFlash path on this MoE, not to memory pressure or KV type:

• Not OOM: crashes at --ctx-size 32768 too, where the draft est is only 1329 MiB and 15 GB VRAM is free. (At 256K the est is 4913 MiB.)
• Not KV type: crashes identically with turbo3/turbo3 and kvarn3/kvarn3 (and q4_0/q4_0).
• Not expert placement: crashes with --n-cpu-moe 4 and --n-cpu-moe 12.
• Not mmap: --no-mmap (alone and with --mlock --no-host) does not help.
• Drafter is correct: the GGUF is bit-identical to the official Anbeeld/Qwen3.6-35B-A3B-DFlash-GGUF file; vocab_match=1 is reported.
• Target alone is fine: the same target at 256K runs perfectly without DFlash — both kvarn3/kvarn3 (~103 t/s) and turbo3/turbo3 (~91 t/s) decode cleanly. Only adding --spec-type dflash triggers the crash.
• GGML_DFLASH_CRASH_TRACE=1 + GGML_DFLASH_DEBUG=1 + -lv 3 produce no extra breadcrumbs — the fault precedes those hooks.

Likely related to the recurrent-state path; possibly the same root cause as #75 (segfault in llama_memory_recurrent::resize).

First Bad Commit

Not bisected — first time building from v0.3.2. DFlash on the 27B target is not affected on my side; this is specific to the qwen35moe 35B-A3B recurrent path.

Relevant log output

srv    load_model: loading model 'Qwen3.6-35B-A3B-IQ3_S-3.06bpw.gguf'
srv    load_model: [spec] estimated memory usage of draft model is 1329.44 MiB
srv  speculative_: flat DFlash target architecture qwen35moe will use recurrent-only backup cells; keeping attention streams at n_parallel=1
common_init_result: fitting params to device memory ...
common_init_from_params: warming up the model with an empty run - please wait ...
TCQ decode: context-adaptive V alpha enabled
srv    load_model: loading draft model 'qwen36-35b-a3b-dflash-Q4_K_M.gguf'
srv    load_model: DFlash draft model will use target output device CUDA0 by default; pass --spec-draft-device to override
<<< process exits here, no further output, exit without assert >>>

[Andgihat]

Update: this is not MoE-specific — it reproduces on the dense-hybrid Qwen3.6-27B (qwen35) too, so it looks like the whole Qwen3.5/3.6 recurrent family (linear_attention + full_attention hybrid) is affected via the recurrent-backup path.

Same build (a620cbd, v0.3.2, native sm_120, single RTX 5060 Ti 16 GB).

• Target: Qwen3.6-27B-UD-IQ3_XXS (qwen35, head_dim=256, 48× linear_attention + 16× full_attention).
• Drafter: Anbeeld/Qwen3.6-27B-DFlash-GGUF → Qwen3.6-27B-DFlash-Q4_K_M.gguf (vocab_match=1).

Unlike the 35B (which crashes at draft load with stock flags), the 27B loads fine and then hard-crashes on the first decode:

srv  speculative_: flat DFlash target architecture qwen35 will use recurrent-only backup cells; keeping attention streams at n_parallel=1
srv    load_model: [spec] estimated memory usage of draft model is 2942.22 MiB
srv    load_model: loading draft model 'Qwen3.6-27B-DFlash-Q4_K_M.gguf'
srv    load_model: expanded recurrent state to 2 cells before speculative GPU buffers
common_speculative_create_ctx_dft: draft model warmup complete
dflash: target/drafter info: ... vocab_match=1 capture_min=1 capture_max=61
srv  llama_server: model loaded
srv  llama_server: server is listening on http://127.0.0.1:8101
...
srv  params_from_: Chat format: peg-native
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot get_availabl: id  0 | task -1 | adaptive dm: reset state for LRU slot selection
<<< process exits here on first decode, no further output, no assert >>>

Repro:

llama-server -m Qwen3.6-27B-UD-IQ3_XXS.gguf \
  --spec-draft-model Qwen3.6-27B-DFlash-Q4_K_M.gguf \
  --spec-type dflash --spec-dflash-cross-ctx 512 --spec-draft-ngl all \
  --ctx-size 131072 --n-gpu-layers 999 --flash-attn on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -b 512 -ub 256 -np 1 --jinja

For reference, the same 27B target runs cleanly without --spec-type dflash (kvarn3/kvarn3 ~33 t/s, turbo3/turbo3 ~32 t/s @128k), so only the DFlash recurrent-backup path is involved. So between the 35B (draft-load crash) and the 27B (first-decode crash), the common factor is the recurrent backup on Qwen3.5/3.6 targets.