[DSv4][bring-up] FP8 H100 Dynamo-vLLM multinode benchmark / DSv4 FP8 H100 Dynamo-vLLM 多节点基准测试（bring-up）#2196
[DSv4][bring-up] FP8 H100 Dynamo-vLLM multinode benchmark / DSv4 FP8 H100 Dynamo-vLLM 多节点基准测试（bring-up）#2196Oseltamivir wants to merge 18 commits into
Conversation
New SKU — no prior DeepSeek-V4-Pro data on H100. H100 (sm90) has no FP4 path, so this runs FP8 on the deepseek-v4-hopper image (shared with the H200 single-node dsv4 sglang recipe). ~671B FP8 weights don't fit 8xH100 (640GB), so prefill and decode workers each span 2 nodes at TP16 (matches dsr1-fp8-h100-dynamo-sglang); cross-node KV via NIXL. Conservative low-latency 1p1d STP topology (conc 1-64). Unvalidated on-cluster: needs the debug-runs loop for NIXL handshake + TP16 FP8 mem headroom before this leaves bring-up. pr-link intentionally TBD. 中文：搭建 H100 上 DeepSeek-V4-Pro FP8 分离式多节点 SGLang（Dynamo）脚手架 （H100 首次支持 dsv4，无原生 FP4 故走 FP8，复用 deepseek-v4-hopper 镜像； 671B 权重放不下 8xH100，故 prefill/decode 各跨 2 节点 TP16，跨节点 KV 用 NIXL）。尚未在集群验证，需 debug-runs 确认 NIXL 握手与 TP16 FP8 显存余量。
Oseltamivir
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Flesh out the H100 scaffold using same-SKU / same-model refs per recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro: - Add a throughput topology (1p1d TP16 prefill / DEP16 decode = tp16/ep16/dp-attn, DeepEP dispatch) alongside the low-latency TP16 pair, mirroring dsr1-fp8-h100-dynamo-sglang's max-throughput-TEP and max-throughput-DEP shapes. - Cover both 1k1k and 8k1k (4 recipes total). - Correct the FP8 sizing note (~960GB mixed -> ~1.05TB FP8). 中文：依据 recipes.vllm.ai 及同 SKU dsr1-h100、同型号 H200 参考，充实 H100 dynamo-sglang 拓扑：在低延迟 TP16 之外新增吞吐拓扑（DEP16 解码，DeepEP 跨节点分发），覆盖 1k1k 与 8k1k 共 4 个配方。仍需 debug-runs 集群验证。
Oseltamivir
force-pushed
the
dsv4-h100-dynamo-sglang-bringup
branch
from
3c0858f to
8b28eec
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Image: lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928" | ||
| - "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137 | ||
|
|
||
| - config-keys: | ||
| - dsv4-fp8-h100-dynamo-sglang | ||
| description: | ||
| - "Add DeepSeek-V4-Pro FP8 H100 disaggregated multinode SGLang benchmark via Dynamo (new SKU, previously no dsv4 data on H100)" | ||
| - "Extrapolated per recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro from the same-SKU dsr1-fp8-h100-dynamo-sglang (TP16 2-node disagg, NIXL, low-latency-TP + throughput-DEP topology pair) and the same-model H200 dsv4 sglang recipe (FP8 marlin serving)" | ||
| - "H100 (sm90) has no FP4 path, so the ~960GB mixed checkpoint runs in FP8 (~1.05TB), which doesn't fit 8xH100 (640GB); each worker spans 2 nodes at TP16, cross-node KV via NIXL" | ||
| - "Two topologies per seq-len (1k1k + 8k1k): low-latency 1p1d TP16/TP16 (conc 1-64) and throughput 1p1d TP16 prefill / DEP16 decode (tp16/ep16/dp-attn, DeepEP dispatch, conc 64-512); recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/{1k1k,8k1k}/disagg-h100-*.yaml" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The new perf-changelog.yaml entry uses pr-link: TBD, but the canonicalization tooling only recognizes XXX (or the literal PR URL) as a valid placeholder. This will make utils/prepare_perf_changelog_merge.py canonicalize (run by utils/merge_with_reuse.sh at merge time) raise a ChangelogValidationError and block the merge — please change it to XXX.
Extended reasoning...
The bug: The appended entry at the end of perf-changelog.yaml ends with pr-link: TBD instead of the canonical placeholder. utils/validate_perf_changelog.py defines:
PR_LINK_PLACEHOLDERS = {
"XXX",
"https://github.com/SemiAnalysisAI/InferenceX/pull/XXX",
}"TBD" is not a member of this set and does not match the canonical PR-link regex either, so it is not recognized as a valid stand-in for the real PR URL.
The triggering code path: The reuse-merge tooling (utils/merge_with_reuse.sh), at the step that finalizes a PR merge, invokes prepare_perf_changelog_merge.py canonicalize --base-ref origin/main --pr-number "$PR". That entry point calls canonicalize_appended_links(), which immediately calls compare_entries(base, head, pr_number). For every appended changelog entry, compare_entries calls validate_added_pr_link(link, pr_number) from validate_perf_changelog.py. Since pr_number is a real integer at merge time, the check becomes:
if link not in PR_LINK_PLACEHOLDERS and link != expected:
raise ChangelogValidationError(...)"TBD" fails both conditions (not in PR_LINK_PLACEHOLDERS and != expected canonical link), so ChangelogValidationError is raised and the merge-prep step aborts under set -euo pipefail in merge_with_reuse.sh.
Why the author's local check doesn't catch it: The PR description says validate_perf_changelog.py --base-ref main passes locally. That CLI's main() only calls validate_matrix_compatible_change → validate_generated_config, which shells out to process_changelog.py — a script that contains no pr-link/compare_entries logic at all. compare_entries/validate_added_pr_link are only reachable through prepare_perf_changelog_merge.py, which is invoked exclusively by the merge tooling, not by the documented local check. So the local validation passing is misleading — it gives no signal about whether the pr-link value will be accepted at merge time.
Step-by-step proof:
- PR is merged; CI/maintainer runs
utils/merge_with_reuse.shfor PR [DSv4][bring-up] FP8 H100 Dynamo-vLLM multinode benchmark / DSv4 FP8 H100 Dynamo-vLLM 多节点基准测试（bring-up） #2196. - The script reaches the canonicalize step:
prepare_perf_changelog_merge.py canonicalize --base-ref origin/main --pr-number 2196. canonicalize_appended_links()callscompare_entries(base_content, head_content, 2196).compare_entriesfinds the new entry (config-keydsv4-fp8-h100-dynamo-sglang) withpr-link: "TBD"and callsvalidate_added_pr_link("TBD", 2196).- Since
pr_number=2196is set, the function checks"TBD" not in {"XXX", ".../pull/XXX"}(True) and"TBD" != "https://github.com/SemiAnalysisAI/InferenceX/pull/2196"(True) → raisesChangelogValidationError. - The script exits non-zero;
merge_with_reuse.sh(running underset -e) aborts the merge, blocking the automated reuse-merge path until the entry is fixed.
Fix: Replace pr-link: TBD with pr-link: XXX (matching the convention used elsewhere in this file, which the tooling then auto-canonicalizes to the real PR URL at merge time), or use the literal https://github.com/SemiAnalysisAI/InferenceX/pull/2196 link directly.
| dsv4-fp8-h100-dynamo-sglang: | ||
| image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f | ||
| model: deepseek-ai/DeepSeek-V4-Pro | ||
| model-prefix: dsv4 | ||
| runner: h100-multinode | ||
| precision: fp8 | ||
| framework: dynamo-sglang | ||
| router: { name: dynamo-router, version: "0.8.0" } | ||
| kv-p2p-transfer: nixl | ||
| multinode: true | ||
| disagg: true | ||
| scenarios: | ||
| fixed-seq-len: | ||
| - isl: 1024 | ||
| osl: 1024 | ||
| search-space: | ||
| # Low latency: 1 prefill (TP16, 2 nodes) + 1 decode (TP16, 2 nodes). | ||
| - conc-list: [1, 2, 4, 8, 16, 32, 64] | ||
| prefill: | ||
| num-worker: 1 | ||
| tp: 16 | ||
| ep: 1 | ||
| dp-attn: false | ||
| additional-settings: | ||
| - "CONFIG_FILE=recipes/sglang/deepseek-v4/1k1k/disagg-h100-1p1d-tp16-tp16.yaml" | ||
| decode: |
There was a problem hiding this comment.
🔴 The new dsv4-fp8-h100-dynamo-sglang config (runner: h100-multinode) routes to runners/launch_h100-dgxc-slurm.sh, which is not updated by this PR and has three separate blockers: (1) its dynamo-sglang branch only accepts MODEL_PREFIX==dsr1 && PRECISION==fp8 and exits 1 for any other combination, including the new dsv4/fp8 config; (2) even past that, it hardcodes the SGLang container squash to the old v0.5.8-cu130 image rather than reading the config's deepseek-v4-hopper image; (3) it never stages the new SemiAnalysis-authored recipe YAMLs (recipes/sglang/deepseek-v4/...) into the cloned srt-slurm repo the way launch_b200-dgxc.sh/launch_gb200-nv.sh do, so srtctl apply would fail on a missing file. All three must be fixed in the launcher before any of the 4 new recipes can actually run on h100-multinode.
Extended reasoning...
The new dsv4-fp8-h100-dynamo-sglang config in configs/nvidia-master.yaml (lines 4603-4628) sets model-prefix: dsv4, precision: fp8, framework: dynamo-sglang, and runner: h100-multinode. That runner maps (via configs/runners.yaml) to runners/launch_h100-dgxc-slurm.sh, which is untouched by this PR. Reading that launcher directly turns up three independent, stacked blockers, each of which alone is sufficient to prevent the recipe from running:
1. Model-prefix hard gate (lines 19-26). The dynamo-sglang branch is:
if [[ $FRAMEWORK == "dynamo-sglang" ]]; then
if [[ $MODEL_PREFIX == "dsr1" && $PRECISION == "fp8" ]]; then
export MODEL_PATH="/mnt/nfs/lustre/models/dsr1-fp8"
export SRT_SLURM_MODEL_PREFIX="dsr1-fp8"
else
echo "Unsupported model prefix/precision for dynamo-sglang: $MODEL_PREFIX/$PRECISION"
exit 1
fiWith set -e/set -x active from the top of the script, MODEL_PREFIX=dsv4 fails the inner condition and the script exits 1 immediately — before the srt-slurm clone, container mapping, or srtctl apply logic is ever reached (those start around line 39 onward). This happens for every one of the 4 new recipe YAMLs, since they all share this config entry.
2. Wrong container squash file (lines 79-83). Even hypothetically past gate #1, the sglang container mapping is unconditional:
if [[ $FRAMEWORK == "dynamo-sglang" ]]; then
SQUASH_FILE="/mnt/nfs/lustre/containers/lmsysorg_sglang_v0.5.8.post1-cu130.sqsh"
CONTAINER_KEY="lmsysorg/sglang:v0.5.8-cu130"It never reads $IMAGE for sglang (only the dynamo-trt branch derives its squash/key from $IMAGE). The existing dsr1-fp8-h100-dynamo-sglang config happens to declare image: lmsysorg/sglang:v0.5.8-cu130, which is exactly this hardcoded value — that's why it has worked so far by coincidence. The new dsv4 config declares image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d5..., which this launcher has no knowledge of. The generated srtslurm.yaml containers map only ever points dynamo-sglang/latest/${CONTAINER_KEY} at the v0.5.8 squash — there is no deepseek-v4-hopper entry and no import step for it in the multinode path (the multinode branch never runs enroot import, unlike the single-node else-branch at ~line 314). The recipe would run in the wrong sglang build, one that predates DeepSeek-V4 support.
3. Recipe files never staged into the clone (lines 39-56, 138-148). The launcher clones NVIDIA/srt-slurm, checks out sa-submission-q2-2026, cds into it, and later runs srtctl apply -f "$CONFIG_FILE" with CONFIG_FILE=recipes/sglang/deepseek-v4/{1k1k,8k1k}/disagg-h100-1p1d-tp16-{tp16,dep16}.yaml — a path resolved relative to that clone root. I grepped the whole script for any cp/mkdir -p recipes staging step and found none (the only cp copies logs out after the run). Compare runners/launch_b200-dgxc.sh, which explicitly does this per model/framework:
mkdir -p recipes/vllm/deepseek-v4
cp -rT \"$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4\" recipes/vllm/deepseek-v4
...
mkdir -p recipes/sglang/dsr1/b200-fp4
cp -rT \"$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/dsr1/b200-fp4\" recipes/sglang/dsr1/b200-fp4The existing dsr1 H100 recipe uses recipes/h100/... paths that ship with the upstream srt-slurm branch, so the H100 launcher never previously needed a staging step. This PR introduces the first H100 recipes under recipes/sglang/deepseek-v4/... whose source lives in this repo's own benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/ tree — brand-new SemiAnalysis-authored files that do not exist in the pinned upstream branch. Without a copy step, sed -i on $CONFIG_FILE (line 145) and srtctl apply -f "$CONFIG_FILE" (line 148) fail with a missing-file error.
Step-by-step proof for gap #1 (the outermost blocker): a sweep job for dsv4-fp8-h100-dynamo-sglang sets FRAMEWORK=dynamo-sglang, MODEL_PREFIX=dsv4, PRECISION=fp8, IS_MULTINODE=true and invokes launch_h100-dgxc-slurm.sh. set -e/set -x are active. Execution enters the IS_MULTINODE==true branch, then FRAMEWORK == "dynamo-sglang" is true, then $MODEL_PREFIX == "dsr1" && $PRECISION == "fp8" evaluates dsv4 == dsr1 → false, so the else fires: echo "Unsupported model prefix/precision for dynamo-sglang: dsv4/fp8" followed by exit 1. The job dies here, in every one of the 4 new sweep configs, before the srt-slurm clone (gap #3's location) or the container mapping (gap #2's location) are ever reached.
Why this isn't caught by existing code/tests: generate_sweep_configs.py and validate_perf_changelog.py, which the PR description cites as local checks, only validate the YAML schema/shape — neither executes the runner launcher script, so none of these three gaps would surface locally. They only manifest at actual sweep-launch time on the h100-multinode pool.
Fix: launch_h100-dgxc-slurm.sh needs a dsv4/fp8 arm in the dynamo-sglang branch (setting MODEL_PATH/SRT_SLURM_MODEL_PREFIX), a container mapping that derives the squash/key from $IMAGE for sglang (or an explicit dsv4-hopper entry) instead of the hardcoded v0.5.8 values, and a mkdir -p recipes/sglang/deepseek-v4 && cp -rT $GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4 recipes/sglang/deepseek-v4 staging step mirroring the b200/gb200 launchers. None of this is in scope of the current 6-file diff, so as merged the 4 new recipes cannot launch at all. Given the PR is explicitly flagged as a bring-up scaffold requiring on-cluster debug-runs before it's considered validated, this is worth surfacing now since the debug-runs themselves cannot even start without a companion launcher change.
| SGLANG_ENABLE_THINKING: "1" | ||
| SGLANG_REASONING_EFFORT: "max" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "100000" | ||
| SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "100000" | ||
|
|
||
| sglang_config: | ||
| prefill: | ||
| served-model-name: "deepseek-ai/DeepSeek-V4-Pro" | ||
| model-path: "/model/" | ||
| trust-remote-code: true | ||
| disable-radix-cache: true | ||
|
|
||
| disaggregation-mode: "prefill" | ||
| disaggregation-transfer-backend: nixl | ||
|
|
There was a problem hiding this comment.
🟡 The DEP16 decode block in disagg-h100-1p1d-tp16-dep16.yaml (both 1k1k and 8k1k variants) sets moe-a2a-backend: deepep but never sets moe-runner-backend: marlin or disable-flashinfer-autotune, even though the prefill block in the same file, every worker in the sibling disagg-h100-1p1d-tp16-tp16.yaml, and the H200 dsv4 reference script all explicitly pin marlin for this FP8-on-sm90 checkpoint. Since moe-a2a-backend only controls all-to-all dispatch and is orthogonal to the MoE GEMM runner backend, the DEP16 decode worker (the higher-throughput topology) will fall back to SGLang's default runner instead of the validated FP8 marlin path used everywhere else in this PR.
Extended reasoning...
The bug: In both disagg-h100-1p1d-tp16-dep16.yaml files (1k1k lines 61-76, 8k1k lines 61-76), the decode block under sglang_config sets moe-a2a-backend: deepep along with expert-parallel-size: 16 and enable-dp-attention: true, but omits moe-runner-backend: marlin and disable-flashinfer-autotune: true. Those two settings ARE present on:
- the
prefillblock in the same file (lines 45-46 in each variant), - both
prefillanddecodeblocks of the siblingdisagg-h100-1p1d-tp16-tp16.yaml(same PR, same SKU/precision), - the pre-existing H200 dsv4 single-node reference (
benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang.sh:41,43:--moe-runner-backend marlin,--disable-flashinfer-autotune).
Why this isn't a false alarm from a misunderstanding of the two flags: moe-a2a-backend selects the all-to-all dispatch/combine implementation used to route tokens between experts across ranks (deepep vs. the default). moe-runner-backend selects the local expert grouped-GEMM kernel (e.g. marlin, deep_gemm, flashinfer_trtllm, triton/auto). These are orthogonal knobs in SGLang — picking a dispatch backend does not implicitly pin a GEMM runner backend. Reference configs elsewhere in this repo confirm both are set together on DEP decode blocks that use deepep, e.g. qwen3.5/gb200-fp8/.../1p1d-dep4-dep16.yaml decode sets moe-runner-backend: deep_gemm and moe-a2a-backend: deepep, and the dsr1 B200 FP4 DEP decode sets moe-runner-backend: flashinfer_trtllm alongside its a2a backend. So pinning a runner backend on a deepep-dispatch DEP decode block is the established pattern in this codebase, not an incompatible combination.
The code path that triggers it: The PR's own description states H100 (sm90) has no FP4 kernels, so this FP8 checkpoint must run through the marlin MoE path — that's exactly why the author forced moe-runner-backend: marlin on the prefill worker and on every TP16 decode worker. The DEP16 decode worker still executes the same MoE layers (it's expert-parallel across 16 GPUs, not skipping MoE), so it needs a working FP8 MoE GEMM path just as much as prefill does. With the setting unset, SGLang falls back to its default/auto runner-backend selection for this launch, which is inconsistent with the marlin path validated/selected everywhere else in this PR for sm90 FP8.
Impact: At best, the DEP16 decode worker silently runs a different (unvalidated, possibly slower or numerically different) MoE kernel than every other worker in this PR, undermining the throughput-topology benchmark's comparability to the TP16 low-latency topology and to the intended marlin baseline. At worst, since the FP8 marlin path was deliberately chosen because H100 lacks the FP4 fast path, the default runner could fail to launch, crash, or silently underperform on sm90 for this checkpoint — a real risk for exactly the highest-throughput topology (conc 64-512) the PR is trying to benchmark.
Proof by walkthrough: (1) Open disagg-h100-1p1d-tp16-dep16.yaml, decode block: keys present are tensor-parallel-size: 16, data-parallel-size: 16, expert-parallel-size: 16, enable-dp-attention: true, moe-a2a-backend: deepep, kv-cache-dtype: fp8_e4m3 — no moe-runner-backend key anywhere in that block. (2) Scroll up ~15 lines to the prefill block in the same file: moe-runner-backend: marlin and disable-flashinfer-autotune: true are both present. (3) Open the sibling disagg-h100-1p1d-tp16-tp16.yaml: both prefill and decode blocks set moe-runner-backend: marlin + disable-flashinfer-autotune: true. (4) Open benchmarks/single_node/fixed_seq_len/dsv4_fp8_h200_sglang.sh: confirms marlin is the established FP8 MoE runner for this model family on Hopper-class GPUs. (5) Conclude: every other MoE-running worker across this PR and its stated references pins marlin; only the DEP16 decode block (in both 1k1k and 8k1k) omits it — an inconsistency, not a deliberate design choice.
Fix: Add moe-runner-backend: marlin and disable-flashinfer-autotune: true to the decode block's sglang_config in both disagg-h100-1p1d-tp16-dep16.yaml files (1k1k and 8k1k), matching the prefill block in the same file and the sibling TP16 decode block.
Note: this PR is explicitly an unvalidated bring-up scaffold gated on debug-runs before it leaves bring-up, so this omission will surface and can be corrected during that validation pass rather than causing a silent production issue today — hence nit rather than a hard merge blocker.
dsv4-fp8-h100-dynamo-sglang used runner: h100-multinode, copied from the dsr1-fp8-h100-dynamo-* references. No registered runner carries the h100-multinode label, so run-sweep jobs (runs-on = the pool key verbatim) queued indefinitely (run 29326475700). Add an h100-disagg pool to runners.yaml (mirrors the mi300x-disagg / mi355x-disagg convention) mapping to the 3 runners that actually carry the h100-disagg label (h100-dgxc-slurm_17-19), and point the config at it. The launcher is selected from the runner name (h100-dgxc-slurm_NN -> launch_h100-dgxc-slurm.sh), so routing is unchanged. 中文：修复 H100 dynamo-sglang 的 runner 标签。原先用的 h100-multinode 无任何 runner 携带该标签，导致作业永久排队。新增 h100-disagg 资源池（对应真实携带 h100-disagg 标签的 3 台 runner），并将配置指向它。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29326475700 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29327734075 |
Replace the SGLang scaffold with a vLLM TEP16/TEP16 deployment, including the H100 launcher, DeepGEMM setup, NIXL recipe wiring, and append-only changelog update. 中文：将 SGLang 脚手架改为遵循 vLLM 配方的 TEP16/TEP16 部署，并补齐 H100 启动器、DeepGEMM 安装、NIXL 配方接入及追加式更新日志。
Oseltamivir
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29329644045 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29329644045 |
Install GNU make only when the H100 login-runner image does not provide it, then continue through the same upstream make setup ARCH=x86_64 flow used by the other srt-slurm launchers.\n\n中文：仅在 H100 登录 runner 镜像未提供 GNU make 时安装该依赖，随后继续使用与其他 srt-slurm 启动器一致的上游 make setup ARCH=x86_64 流程。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29330562124 |
Keep the H100 launcher aligned with the other srt-slurm launchers by invoking make setup directly and leaving GNU Make provisioning to the runner image.\n\n中文：让 H100 启动器与其他 srt-slurm 启动器保持一致，直接调用 make setup，并由 runner 镜像负责提供 GNU Make，不在宿主机安装软件包。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29332540598 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29332540598 |
Use the existing dsv4-fp8 checkpoint on the H100 cluster and exclude nodes without healthy pod-network RDMA, including the node with mismatched mlx5 numbering.\n\n中文：使用 H100 集群现有的 dsv4-fp8 检查点，并排除 pod 网络 RDMA 异常的节点以及 mlx5 设备编号不一致的节点。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29333296843 |
Replace the custom DeepGEMM bootstrap with NVIDIA srt-slurm's standard vllm-container-deps.sh and wire both H100 recipes to it.\n\n中文：将自定义 DeepGEMM 引导脚本替换为 NVIDIA srt-slurm 标准的 vllm-container-deps.sh，并让两套 H100 配方统一使用该脚本。
Split each two-node worker as TP8 x PP2 with EP8 so FP8 weight partitions remain divisible by the checkpoint's 128-column quantization blocks while retaining 16 GPUs of memory.\n\n中文：将每个双节点 worker 调整为 TP8 x PP2 并使用 EP8，在保留 16 张 GPU 显存容量的同时，确保 FP8 权重分片可被检查点的 128 列量化块整除。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29333848295 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29334346000 |
The multinode run failed with torch.OutOfMemoryError during decode CUDA-graph capture: each TP8/PP2 worker reached ~74.6/80GB and could not allocate the last ~1GB (~3.93GB was tied up in CUDA-graph pools). Fixes: - gpu-memory-utilization 0.95 -> 0.90 (frees ~4GB headroom for capture) - add PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to prefill/decode env (the allocator's own fragmentation hint from the OOM message) - max-model-len 200000 -> 3072 (1k1k) / 9472 (8k1k): benchmark-appropriate, drops the oversized KV/metadata reservation Applied to both disagg-h100-1p1d-tep8pp2-tep8pp2.yaml recipes. 中文：修复 H100 TP8/PP2 模型加载阶段的 CUDA OOM。每个 worker 在 0.95 显存利用率 下达到 ~74.6/80GB，decode CUDA graph 捕获时无法再分配 ~1GB。将显存利用率降到 0.90、加入 expandable_segments 以减少碎片、并把 max-model-len 从 200000 改为 基准合适的 3072/9472。
The TP8xPP2 disaggregated prefill hung ~300s on the 8k1k prefill then the EngineCore died (dynamo-disagg + pipeline-parallel + NIXL KV-transfer stall; the server loaded and served, so OOM is resolved — this is a PP/disagg stall). PP is not a standard vLLM DeepSeek-V4 strategy; switch to the recipe's multi_node_dep pattern: TP8 x DP2 x EP16. TP8 stays intra-node (preserves the FP8 128-column block divisibility that cross-node TP16 breaks), data-parallel (not a pipeline) spans the 2 nodes, EP16 shards experts. Per-GPU ~= dense/8 + experts/16 ~= 46GB, fits 80GB. Applied to both 1k1k and 8k1k recipes. 中文：H100 由 TP8xPP2 改为 TP8 x DP2 x EP16（recipe 的 multi_node_dep 策略）。 PP2 分离式在 8k1k 预填充阶段挂起 ~300s 导致 EngineCore 死亡（PP+disagg+NIXL 传输停滞）。DP2 用数据并行跨 2 节点，消除流水线停滞；TP8 保持节点内，避免 TP16 破坏 FP8 128 列量化块整除性。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29350015600 |
origin/main advanced (H200 refresh #2191 merged, adding dsv4-fp8-h200-* entries) so the branch's changelog was stale and check-changelog flagged the main-only entries as deletions. Reset perf-changelog.yaml to origin/main and re-append only the dsv4-fp8-h100-dynamo-vllm entry. 中文：将 H100 分支的 perf-changelog 重置到 origin/main 并仅重新追加 H100 条目， 修复 origin/main 推进（#2191 合并）导致的 append-only 校验删除报错。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29353482591 |
TP8xDP2 failed at launch: dynamo assigns a data_parallel_rank per GPU, so
data-parallel-size must equal the 16-GPU count ("data_parallel_rank (9) must
be in range [0,2)"). Switch to the framework's DEP mode: tensor-parallel-size=1,
data-parallel-size=16, expert-parallel across the 2 nodes — matching the GB200
dsv4 dynamo-vLLM DEP recipe (data-parallel-size=gpus, tp1). tp1 also removes the
FP8 128-column TP-divisibility issue that motivated the original PP2.
中文：H100 改为纯 DP16+EP16（tp1）。TP8xDP2 启动失败，因 dynamo 按 GPU 分配
data_parallel_rank，故 data-parallel-size 必须等于 GPU 数（16）。参照 GB200
dsv4 DEP 配方（data-parallel-size=gpus, tp1）。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29354041797 |
DP16 on 2 nodes OOM'd during model load (dense replicated ~16GB + experts EP16 ~62GB > 80GB). Aggregated DP32+EP32 across 4 nodes shards experts to ~31GB/GPU, leaving headroom. TP1 keeps FP8 128-col block divisibility. Aggregated recipe (prefill worker 1 / decode worker 0), enforce-eager for the first green shot to isolate the memory-fit variable.
Oseltamivir
force-pushed
the
dsv4-h100-dynamo-sglang-bringup
branch
from
690ea9b to
3e2cbfc
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29356153316 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29356883417 |
DP32 'aggregated' collapsed onto 1 node: this framework sizes nodes as tp*pp/gpus_per_node and EP/DP-attn add no GPUs, so tp1 gave 1 node and vLLM could not place 32 DP ranks on 8 GPUs. Use the memory-proven TP8xPP2 topology (the original disagg run loaded + reached prefill; only the NIXL P/D transfer stalled) but AGGREGATED (decode num-worker 0) so there is no P/D transfer. tp8*pp2 = 16 GPUs = 2 nodes; TP8 keeps FP8 128-col divisibility.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29358425633 |
On-cluster prototype (srtctl job 11256, 2-node TP8xPP2, 1x8) confirmed FULL_AND_PIECEWISE graphs fit at gpu-mem-util 0.92 (no OOM) and cut decode TPOT from the eager ~144ms to ~15ms (~9.4x). The eager sweep served correctly but was dispatch-bound and could not finish in 8h; graphs make it viable on the same 2 nodes (no extra nodes needed).
8k1k conc512 = 5120 x 8k-token requests needs ~3.6h for that single level; the run blew the 8h SLURM limit mid-conc512 (1k1k + eval were already green). conc256 completes in ~71min and fits with margin. 1k1k keeps conc512 (short requests finish fast). Optimized cudagraph config otherwise unchanged.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29384384052 |
1 participant
Summary
Bring-up configuration; H100 remains unverified upstream and requires on-cluster validation before merge.
This revision follows the official vLLM DeepSeek-V4-Pro recipe instead of the earlier SGLang extrapolation:
vllm/vllm-openai:v0.21.0with FP8 KV cache, block size 256, expert parallelism, Hopper memory settings, and the required DeepGEMM setup.benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/{1k1k,8k1k}/and run on theh100-disaggpool.Validation completed:
python -m pytest utils/matrix_logic/ -v: 221 passed.srtctl dry-run, each requesting four H100 nodes.中文说明
这是 bring-up 配置；上游尚未验证 H100，合并前仍需在集群上完成验证。
本次修改不再沿用此前的 SGLang 外推方案，而是遵循官方 vLLM DeepSeek-V4-Pro 配方：
vllm/vllm-openai:v0.21.0，启用 FP8 KV cache、256 block size、专家并行(EP)及 Hopper 显存配置，并补齐配方要求的 DeepGEMM 安装步骤。benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/{1k1k,8k1k}/，使用h100-disagg资源池。已完成以下验证：
python -m pytest utils/matrix_logic/ -v：221 项测试通过。srtctl dry-run，每个作业申请四个 H100 节点。