Skip to content

[DSv4] Refresh GB200 dynamo-vLLM 1k1k sweep #2190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Oseltamivir wants to merge 2 commits into
base: main
Choose a base branch
from
+485 −0
Open

[DSv4] Refresh GB200 dynamo-vLLM 1k1k sweep #2190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e93f00a
feat(dsv4): restore GB200 dynamo-vllm 1k1k sweep
Oseltamivir Jul 14, 2026
b9d6a08
Merge branch 'main' into dsv4-gb200-restore-1k1k
Oseltamivir Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions ...arks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/1k1k/disagg-gb200-1p1d-dep8-dep16.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
name: "dsv4-vllm-disagg-gb200-1p1d-dep8-dep16"

# 1k/1k mid-to-high throughput topology. Extrapolated from
# kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml adjusted for DSV4-Pro's
# DP>=8 minimum. Single prefill worker feeding a wide DP=16 decode handles
# conc 256-4096 cleanly for 1k prompts (prefill throughput per rank is high
# enough at this prompt length; see kimi precedent).
#
# Differences from our 8k1k 7p1d-dep8-dep16:
# * prefill_workers: 1 (vs 7) — 1k prompts don't need 14 prefill nodes
# * max-model-len: 3072 instead of auto
# * prefill max-num-seqs: 16 (fills 16384-token budget at 1k per seq)
# * decode max-num-seqs: 512 instead of 256 (shorter KV, more parallelism)
# * max-cudagraph-capture-size / max-num-batched-tokens (decode): 512
#
# Restored 2026-07 for the GB200 dsv4 1k1k refresh: these day-0 recipes
# (#1129) were dropped by the deepseek-v4-pro-sa curation (5b0347f43,
# "only 8k/1k in scope"), which froze the site's GB200 1k1k rows at the
# 2026-04-25 day-0 run. Container/dynamo/infra fields updated to match
# the current 8k1k -sa set (v0.20.0-ubuntu2404, dynamo wheel
# 1.2.0.dev20260426, dedicated NATS/etcd infra node); topology and engine
# tuning unchanged from day-0.

model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"

setup_script: vllm-container-deps.sh

# Also set slurm.time_limit explicitly (above srtslurm.yaml's 6h default) so
# a slow first-time Lustre load + cudagraph capture can't get cut off by the
# SLURM wall clock.
slurm:
time_limit: "8:00:00"

# Bumped from the 1800s default to 4 hours. DSV4-Pro weights load slowly from
# Lustre with multiple workers contending for the same OSTs — previous 1k/1k
# run hit the default 1800s. Make this *very* generous since the cost of an
# over-long deadline is just sitting idle, not wasted compute.
health_check:
max_attempts: 1440
interval_seconds: 10

resources:
gpu_type: "gb200"
gpus_per_node: 4
prefill_nodes: 2
decode_nodes: 4
prefill_workers: 1
decode_workers: 1
gpus_per_prefill: 8
gpus_per_decode: 16

infra:
etcd_nats_dedicated_node: true

frontend:
type: dynamo
enable_multiple_frontends: false

backend:
type: vllm
connector: null

prefill_environment:
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
VLLM_SERVER_DEV_MODE: "1"

decode_environment:
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
VLLM_SERVER_DEV_MODE: "1"

vllm_config:
prefill:
kv-transfer-config: '{"kv_connector": "NixlConnector", "kv_role": "kv_both"}'
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
kv-cache-dtype: "fp8"
tensor-parallel-size: 1
pipeline-parallel-size: 1
data-parallel-size: 8
data-parallel-rpc-port: 13345
enable-expert-parallel: true
enforce-eager: true
max-model-len: 3072
max-num-seqs: 16
max-num-batched-tokens: 16384
trust-remote-code: true
no-enable-prefix-caching: true
no-enable-flashinfer-autotune: true
block-size: 256
gpu-memory-utilization: 0.88
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true
Comment on lines +87 to +106

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Both new dep8-dep16 recipes (1p1d and 3p1d) omit tokenizer-mode: deepseek_v4 in their prefill and decode vllm_config blocks, unlike every other DeepSeek-V4-Pro recipe in the tree (including the tep8 sibling added in this same PR).

Extended reasoning...

What's wrong

disagg-gb200-1p1d-dep8-dep16.yaml and disagg-gb200-3p1d-dep8-dep16.yaml set no tokenizer-mode anywhere in their vllm_config.prefill or vllm_config.decode blocks. Every other DeepSeek-V4-Pro vLLM recipe in benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/ — all 8k1k GB200/B200/GB300 recipes, the agentic recipes, and critically the disagg-gb200-1p1d-dep8-tep8.yaml sibling added in this same PR (lines 125 and 147) — sets tokenizer-mode: deepseek_v4 in both the prefill and decode sections.

Why this happened

The header comment on the affected file states it was "Extrapolated from kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml adjusted for DSV4-Pro's DP>=8 minimum." Kimi-K2.5 is a different model that does not require the DeepSeek-specific tokenizer, so the extrapolation carried over everything except this one model-specific field. By contrast, the tep8 sibling recipe in the same PR was mirrored directly from NVIDIA's 8k1k DSV4 reference recipe, which is why it correctly retained tokenizer-mode: deepseek_v4.

Why nothing else catches it

tokenizer-mode is not injected by the config/recipe generation pipeline (generate_sweep_configs.py) — it is a raw pass-through field into the vLLM launch args from the recipe YAML itself. There's no default-injection or validation step downstream that would backfill it, and deepseek_v4 is a genuinely distinct, non-auto-selected tokenizer mode (see utils/bench_serving/benchmark_serving.py around line 1271, where it is listed as a separate choice alongside auto/slow/mistral/custom, with a comment noting the stock HF AutoTokenizer may not recognize deepseek_v4 and vLLM's own loader must be used instead).

Impact

Without this setting, vLLM falls back to its default (auto) tokenizer resolution for DeepSeek-V4-Pro, which will either fail to load a workable tokenizer at engine startup or silently mis-tokenize inputs/outputs. These two recipes back the conc-list: [128, 256, 1024, 2048, 4096] and conc-list: [4096, 8192] scenarios in configs/nvidia-master.yaml — i.e. most of the 1k1k sweep this PR is restoring. The result would either be failed CI runs or (worse) benchmark numbers published for an incorrectly-tokenized model, defeating the stated purpose of the PR.

Proof by comparison

  1. Grep the tree for tokenizer-mode: deepseek_v4 — it appears twice (prefill + decode) in every existing DSv4-Pro recipe, and twice in the new disagg-gb200-1p1d-dep8-tep8.yaml (lines 125, 147) added in this PR.
  2. The same grep against disagg-gb200-1p1d-dep8-dep16.yaml (lines 87-106 shown, prefill/decode vllm_config blocks) and disagg-gb200-3p1d-dep8-dep16.yaml returns zero matches.
  3. The fix is a one-line addition per section: add tokenizer-mode: deepseek_v4 to both the prefill: and decode: entries of vllm_config in both files, mirroring the tep8 recipe and every 8k1k recipe.


decode:
kv-transfer-config: '{"kv_connector": "NixlConnector", "kv_role": "kv_both"}'
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
kv-cache-dtype: "fp8"
tensor-parallel-size: 1
pipeline-parallel-size: 1
data-parallel-size: 16
data-parallel-rpc-port: 13345
enable-expert-parallel: true
max-model-len: 3072
max-num-seqs: 512
max-cudagraph-capture-size: 512
max-num-batched-tokens: 512
trust-remote-code: true
no-enable-prefix-caching: true
block-size: 256
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
gpu-memory-utilization: 0.9
stream-interval: 50
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true

benchmark:
type: "sa-bench"
isl: 1024
osl: 1024
concurrencies: "128x256x1024x2048x4096"
req_rate: "inf"
use_chat_template: false

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The three new 1k1k vLLM DSv4-Pro recipes set benchmark.use_chat_template: false with no custom_tokenizer, while every one of the 20 existing DeepSeek-V4-Pro vLLM 8k1k recipes uses use_chat_template: true + custom_tokenizer: sa_bench_tokenizers.vllm_deepseek_v4.VLLMDeepseekV4Tokenizer. This looks like a leftover from the kimi-k2.5 template cited in the header comments rather than a deliberate choice, and isn't called out in the PR description — worth a quick confirmation, though the actual measurement impact on this fixed-ISL/OSL sweep is likely small.

Extended reasoning...

All three new 1k1k recipes (disagg-gb200-1p1d-dep8-tep8.yaml, disagg-gb200-1p1d-dep8-dep16.yaml, disagg-gb200-3p1d-dep8-dep16.yaml) set benchmark.use_chat_template: false and omit custom_tokenizer entirely. Every one of the 20 existing DeepSeek-V4-Pro recipes under benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/ sets use_chat_template: true paired with custom_tokenizer: sa_bench_tokenizers.vllm_deepseek_v4.VLLMDeepseekV4Tokenizer, with zero exceptions. The header comments in the new files say they were extrapolated from kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml and NVIDIA's 8k1k recipe — neither of which uses the vLLM-DSv4 tokenizer/chat-template convention — which is consistent with this being a copy-paste leftover rather than an intentional change, and it is not mentioned anywhere in the PR description.

I verified the client-side code path in utils/bench_serving/benchmark_serving.py: use_chat_template gates whether _apply_chat_template() (and, when --dsv4 is set, the dedicated DeepSeek-V4 encoder in encoding_dsv4.py) is applied to generated prompts (lines 211-212, 334-335). There's also a hard guard, if dsv4 and not use_chat_template: raise ValueError(...), but I confirmed in benchmarks/benchmark_lib.sh (lines 430-432) that --dsv4 auto-enables --use-chat-template in the shell wrapper that builds the CLI invocation, so that ValueError cannot actually fire here — no crash risk, contrary to what a naive reading of the guard would suggest.

A verifier raised a solid refutation that I confirmed by grepping the repo: use_chat_template: false with no custom_tokenizer is not unique to these three new files — several existing sglang DeepSeek-V4-Pro 8k1k recipes (e.g. disagg-gb200-1p1d-dep8-dep16-6-c512.yaml, disagg-gb200-4p1d-dep8-dep16-12-c4096.yaml, disagg-gb200-2p1d-dep8-dep16-8-c1536.yaml) also set use_chat_template: false with no custom tokenizer field at all. So the "always true" convention is specific to the vLLM 8k1k DSv4 recipes (still 20/20 with no exceptions there, which is the directly comparable set for these new vLLM recipes), not a universal rule across every DSv4 recipe in the repo. This softens the "clear deviation" framing somewhat, though it doesn't fully explain away the discrepancy since these new files are vLLM recipes being compared against the vLLM convention.

On impact: since benchmark.type here is sa-bench with fixed isl/osl (synthetic random-token generation, not a real corpus), the chat template only changes how the ~1024 input tokens are wrapped/formatted, not the token count — benchmark_serving.py line 244-245 explicitly subtracts the chat-template overhead so the total input length still lands on the target ISL either way. Throughput/latency at a fixed ISL/OSL is driven primarily by topology and batch size, not by whether the tokens are wrapped in a chat template, so the "skews the whole sweep" framing in the original report is likely overstated for this fixed-seq-len benchmark type. The more defensible concern is data-comparability/methodology: the restored 1k1k rows would be generated by a different code path (default tokenizer, no DSv4 chat-template) than the comparable 8k1k rows for the same model, which is worth the author confirming is intentional before publishing to the site — but I couldn't find a concrete mechanism by which it changes the actual throughput/latency numbers with a fixed-ISL/OSL sweep.

Proof walkthrough: For disagg-gb200-1p1d-dep8-dep16.yaml, isl: 1024, osl: 1024. With use_chat_template: true (as in the 8k1k sibling recipes), benchmark_serving.py would apply _apply_chat_template(), compute chat_template_len, and generate 1024 - chat_template_len raw tokens so the templated prompt totals ~1024 tokens. With use_chat_template: false (as configured here), it skips that step and generates exactly 1024 raw tokens directly. In both cases the model receives ~1024 input tokens and produces up to 1024 output tokens — the throughput-determining quantities are the same; what differs is only the token content/formatting, plus which tokenizer encodes it (default HF tokenizer vs. the dedicated VLLMDeepseekV4Tokenizer). That difference matters for methodology consistency across the site's DSv4 rows, but not for whether the reported tok/s numbers for this recipe are individually valid.

Recommend the author confirm whether this was intentional (e.g., because these are day-0-derived recipes predating the -sa tokenizer convention) or fix it to match the 8k1k vLLM convention for consistency — but I don't think it rises to a blocking functional defect given the fixed-ISL/OSL benchmark type and the lack of a crash path.

155 changes: 155 additions & 0 deletions ...marks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/1k1k/disagg-gb200-1p1d-dep8-tep8.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
name: "dsv4-vllm-disagg-gb200-1p1d-dep8-tep8"

# 1k/1k variant of NVIDIA's 8k/1k 1p1d-dep8-tep8 recipe (mirrored from
# aflowers/gb200-dsv4-recipes branch). Same topology and tuning; only
# max-model-len shrinks from 9280 (8k+1k+pad) to 3072 (1k+1k+pad). No
# upstream NVIDIA reference for DSV4-Pro 1k/1k vLLM disagg yet.
#
# Topology: 1 prefill (DP=8) + 1 decode (TP=8). 4 nodes total. Targets
# very low concurrency (1-64).
#
# Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe — see
# ../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml for the full deviation list.
Comment on lines +10 to +12

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Two header-comment path references in the restored 1k1k recipes point to nonexistent files. In disagg-gb200-1p1d-dep8-tep8.yaml (lines 10-12), the comment points to '../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml' but the actual GB200 8k1k low-latency sibling is named disagg-gb200-low-latency.yaml. In disagg-gb200-1p1d-dep8-dep16.yaml (lines 2-4), the comment references 'kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml' but the actual directory is kimi-k2.5-fp4/1k1k/ (missing the -fp4 suffix). Both are comment-only with no runtime impact; worth a quick fix so future readers can actually follow the references.

Extended reasoning...

What the bug is: Two of the three newly-restored 1k1k recipe YAMLs contain header comments that cross-reference sibling/precedent files using paths that do not exist in this repository.

  1. In disagg-gb200-1p1d-dep8-tep8.yaml (lines 10-12), the comment reads: "Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe — see ../8k1k/disagg-gb200-1p1d-dep8-tep8.yaml for the full deviation list." There is no file named disagg-gb200-1p1d-dep8-tep8.yaml under benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/. That directory names its GB200 recipes descriptively (e.g. disagg-gb200-low-latency.yaml, disagg-gb200-low-middle-curve.yaml, disagg-gb200-high-tpt-megamoe.yaml) rather than by topology string. The actual 8k1k sibling implementing the same 1-prefill-DP8 / 1-decode-TP8 low-latency topology is disagg-gb200-low-latency.yaml.

  2. In disagg-gb200-1p1d-dep8-dep16.yaml (top-of-file comment), the precedent this recipe was extrapolated from is written as kimi-k2.5/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml. The only kimi directory that exists under vllm/ is kimi-k2.5-fp4/ (with the -fp4 suffix) — the referenced filename itself is correct and does exist, but at vllm/kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml, not the path written in the comment.

Code path / why nothing catches this: These are plain YAML comments (lines starting with #), never parsed by generate_sweep_configs.py, validate_perf_changelog.py, or any recipe loader — they're documentation for humans skimming the file. Nothing in the PR's stated validation (config generation, changelog validation, YAML parse) would ever touch these strings, so a broken reference here can't be caught by CI; it only surfaces when a person manually tries to open the referenced file.

Impact: Zero runtime impact — the sweep, config generation, and benchmark execution are all unaffected. The only cost is to a future engineer who reads the comment, tries to open the referenced file to check 'the full deviation list' or find the extrapolation precedent, and hits a dead path. For (1) this is a real dead-end since the correct file has a completely different name (disagg-gb200-low-latency.yaml vs. the referenced disagg-gb200-1p1d-dep8-tep8.yaml) and wouldn't be trivially guessable. For (2) the impact is milder — the filename itself is correct and unique in the repo, so a reader who greps for the filename (rather than manually cd'ing into the written path) finds it immediately; a verifier flagged this as too minor to be worth a review comment on its own.

Proof (bug 1): ls benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/ shows no disagg-gb200-1p1d-dep8-tep8.yaml; the GB200 file implementing the same DP8-prefill/TP8-decode low-latency topology is disagg-gb200-low-latency.yaml (confirmed via its name: svf-vllm-disagg-gb200-low-latency and decode tp:8/ep:1 config in configs/nvidia-master.yaml). Following the comment's literal path therefore resolves to a missing file.

Proof (bug 2): ls benchmarks/multi_node/srt-slurm-recipes/vllm/ shows the only kimi directory is kimi-k2.5-fp4; find for disagg-gb200-1p1d-dep4-dep16.yaml locates it only at kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml. The comment's written path kimi-k2.5/1k1k/... (no -fp4) does not exist as a directory.

Fix: Repoint the first comment at ../8k1k/disagg-gb200-low-latency.yaml, and add the missing -fp4 to the second comment's directory (kimi-k2.5-fp4/1k1k/disagg-gb200-1p1d-dep4-dep16.yaml). Both are one-line comment edits with no functional change.

On the refutation: One verifier argued bug 2 is too trivial to report because the filename is unique and easily grep-able despite the wrong parent directory. That's a fair point for bug 2 in isolation, but bug 1's broken reference is a harder dead-end (the correct file has an entirely different name, not a typo'd path), so the pair together is worth a single combined nit-level note — cheap for the author to fix in the same pass, not worth blocking merge over.

#
# Restored 2026-07 for the GB200 dsv4 1k1k refresh: these day-0 recipes
# (#1129) were dropped by the deepseek-v4-pro-sa curation (5b0347f43,
# "only 8k/1k in scope"), which froze the site's GB200 1k1k rows at the
# 2026-04-25 day-0 run. Container/dynamo/infra fields updated to match
# the current 8k1k -sa set (v0.20.0-ubuntu2404, dynamo wheel
# 1.2.0.dev20260426, dedicated NATS/etcd infra node); topology and engine
# tuning unchanged from day-0.

model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"

setup_script: vllm-container-deps.sh

slurm:
time_limit: "8:00:00"

health_check:
max_attempts: 1440
interval_seconds: 10

resources:
gpu_type: "gb200"
gpus_per_node: 4
prefill_nodes: 2
decode_nodes: 2
prefill_workers: 1
decode_workers: 1
gpus_per_prefill: 8
gpus_per_decode: 8

infra:
etcd_nats_dedicated_node: true

frontend:
type: dynamo
enable_multiple_frontends: false

backend:
type: vllm
connector: null

prefill_environment:
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
VLLM_SERVER_DEV_MODE: "1"
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB: "1024"
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE: "2048"
# VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
# VLLM_MOE_ROUTING_SIMULATION_STRATEGY: "uniform_random"
UCX_MEMTYPE_CACHE: "n"
UCX_MEMTYPE_REG_WHOLE: "n"
UCX_TLS: "cuda_copy,cuda_ipc,tcp"
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL

decode_environment:
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
NCCL_MNNVL_ENABLE: "1"
NCCL_NVLS_ENABLE: "1"
VLLM_SERVER_DEV_MODE: "1"
# VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
# VLLM_MOE_ROUTING_SIMULATION_STRATEGY: "uniform_random"
UCX_MEMTYPE_CACHE: "n"
UCX_MEMTYPE_REG_WHOLE: "n"
UCX_TLS: "cuda_copy,cuda_ipc,tcp"
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL

vllm_config:
prefill:
kv-transfer-config: '{"kv_connector": "NixlConnector", "kv_role": "kv_both"}'
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
kv-cache-dtype: "fp8"
tensor-parallel-size: 1
pipeline-parallel-size: 1
data-parallel-size: 8
data-parallel-rpc-port: 13345
enable-expert-parallel: true
enforce-eager: true
max-model-len: 3072
max-num-seqs: 16
max-num-batched-tokens: 32768
trust-remote-code: true
no-enable-prefix-caching: true
no-enable-flashinfer-autotune: true
no-async-scheduling: true
block-size: 256
gpu-memory-utilization: 0.8
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true
# CPU/DRAM expert offload — required for fit. Without these the prefill
# rank reports `Available KV cache memory: -16 GiB` and the engine
# refuses to start. Numa-bind from upstream is still off because our
# NVIDIA/srt-slurm@sa-submission-q2-2026 clone doesn't ship the
# vllm_numa_bind_hash_fix.py patch.
offload-group-size: 3
offload-num-in-group: 1
offload-prefetch-step: 2
tokenizer-mode: deepseek_v4

decode:
kv-transfer-config: '{"kv_connector": "NixlConnector", "kv_role": "kv_both"}'
served-model-name: "deepseek-ai/DeepSeek-V4-Pro"
kv-cache-dtype: "fp8"
tensor-parallel-size: 8
pipeline-parallel-size: 1
enable-expert-parallel: true
max-model-len: 3072
max-num-seqs: 64
max-cudagraph-capture-size: 64
max-num-batched-tokens: 64
trust-remote-code: true
no-enable-prefix-caching: true
block-size: 256
attention-config: '{"use_fp4_indexer_cache":true}'
compilation-config: '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","pass_config":{"fuse_allreduce_rms":false}}'
gpu-memory-utilization: 0.9
stream-interval: 50
no-disable-hybrid-kv-cache-manager: true
enable-sleep-mode: true
tokenizer-mode: deepseek_v4

benchmark:
Comment on lines +130 to +149

@claude claude Bot Jul 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 In the restored disagg-gb200-1p1d-dep8-tep8.yaml recipe, the decode block sets tensor-parallel-size: 8 and enable-expert-parallel: true (no data-parallel-size), so the decode engine actually runs with expert parallelism enabled. But configs/nvidia-master.yaml labels this decode as tp: 8, ep: 1, dp-attn: false (pure TEP), so the published conc 1-64 row for this topology will be mislabeled relative to what the engine actually runs. The fix is to comment out enable-expert-parallel in the decode block, matching the 8k1k disagg-gb200-low-latency.yaml sibling this recipe is explicitly based on.

Extended reasoning...

The bug: disagg-gb200-1p1d-dep8-tep8.yaml's decode block (lines 130-149) sets tensor-parallel-size: 8, enable-expert-parallel: true, and no data-parallel-size. In vLLM, EP world size = tp × dp, so with tp=8/dp=1 this actually shards MoE experts across the 8 TP ranks — a real, different execution mode from plain tensor-parallel decode with replicated experts. Meanwhile configs/nvidia-master.yaml's new search-space entry for this exact recipe (the conc-list: [1, 4, 8, 16, 32, 64] block referencing this CONFIG_FILE) labels decode as tp: 8, ep: 1, dp-attn: false — the pure-TEP convention.\n\nWhy the label matters: per AGENTS.md's srt-slurm section, srtctl reads only the recipe to actually launch the engine, while generate_sweep_configs.py reads nvidia-master.yaml for the frontend labels shown on the published site. These two are decoupled by design, and AGENTS.md explicitly warns that 'Recipe-only edits mislabel results.' That's exactly what happens here: the engine runs with EP enabled, but the site will report/plot the row as ep=1 pure-TEP.\n\nConfirmed against the established convention: I checked the analogous 8k1k sibling this recipe's own header comment claims to mirror ('Local deltas vs upstream 8k/1k sibling: same as the 8k/1k recipe'). 8k1k/disagg-gb200-low-latency.yaml's decode block (lines 112-133) has the identical topology (tp:8, no dp) but explicitly comments out enable-expert-parallel and data-parallel-size:\n\n tensor-parallel-size: 8\n pipeline-parallel-size: 1\n# data-parallel-size: 8\n# data-parallel-rpc-port: 13345\n# enable-expert-parallel: true\n\nAnd master's entry for that 8k1k sibling (lines 8741-8752) carries the same ep: 1, dp-attn: false decode label. So the convention across every GB200 dsv4 decode in this file is: ep:1/dp-attn:false label ⇔ enable-expert-parallel commented out in the recipe; ep:8+/dp-attn:true label ⇔ it's active with a matching data-parallel-size. The new 1k1k recipe is the only pure-TEP-labeled decode that leaves enable-expert-parallel: true active.\n\nStep-by-step proof:\n1. Master config (new entry, this PR): conc-list: [1,4,8,16,32,64] → decode tp:8, ep:1, dp-attn:false, referencing CONFIG_FILE=recipes/vllm/deepseek-v4/1k1k/disagg-gb200-1p1d-dep8-tep8.yaml.\n2. That recipe's decode block: tensor-parallel-size: 8, enable-expert-parallel: true, no data-parallel-size → vLLM launches with EP world size 8 (experts sharded across the 8 TP ranks), not plain TP.\n3. Compare to the 8k1k sibling with the same master label (ep:1, dp-attn:false): its recipe comments out enable-expert-parallel/data-parallel-size, so it truly runs plain TP=8 with replicated experts.\n4. Result: identical master label (ep:1) maps to two different actual decode execution modes — one is really EP=8, the other is really EP disabled. The 1k1k row's reported topology doesn't match what produced its numbers.\n\nFix: comment out (or delete) enable-expert-parallel: true in the decode block of disagg-gb200-1p1d-dep8-tep8.yaml (1k1k) to match the 8k1k sibling and the ep:1 label — consistent with the PR's own stated intent of re-basing this recipe on the current 8k1k -sa layout. This is a valid vLLM config either way (it won't crash CI), so it's a data-correctness/mislabeling issue rather than a functional break.

type: "sa-bench"
isl: 1024
osl: 1024
concurrencies: "1x4x8x16x32x64"
req_rate: "inf"
use_chat_template: false
Loading
Loading