Skip to content

Qwen3.5 FP4 GB300 disagg dynamo-sglang: bump image to v0.5.14-cu130 + NVFP4-V2 checkpoint #2238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hshrivastava-droid wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Qwen3.5 FP4 GB300 disagg dynamo-sglang: bump image to v0.5.14-cu130 + NVFP4-V2 checkpoint #2238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9af6bf5
Qwen3.5 FP4 GB300 disagg dynamo-sglang: bump image to v0.5.14-cu130 +…
hshrivastava-droid Jul 16, 2026
38d77d6
Update perf-changelog pr-link for #2238
hshrivastava-droid Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ...ti_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/8k1k_stp_lowlat_0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Qwen3.5-397B-A17B-NVFP4 Disaggregated 1P1D: TP4 Prefill + TP4 Decode
# Qwen3.5-397B-A17B-NVFP4-V2 Disaggregated 1P1D: TP4 Prefill + TP4 Decode
# Pure tensor parallel, no expert parallel (STP)
# 8k1k sa-bench concurrency sweep on GB300
#
Expand Down Expand Up @@ -83,7 +83,7 @@ backend:

sglang_config:
prefill:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down Expand Up @@ -125,7 +125,7 @@ backend:
log-level: "info"

decode:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down
6 changes: 3 additions & 3 deletions ...ti_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/8k1k_stp_maxtpt_0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Qwen3.5-397B-A17B-NVFP4 Disaggregated 5P1D wide-EP
# Qwen3.5-397B-A17B-NVFP4-V2 Disaggregated 5P1D wide-EP
# Prefill: 5 workers @ TP4/DP4/EP4 with DP-attn (per-node, DEP4)
# Decode: 1 worker @ TP16/DP16/EP16 with DP-attn + TBO (DEP16, 4 nodes)
# Total: 36 GB300 GPUs (5*4 + 4*4); 8k1k concurrency 1024/2048/3072.
Expand Down Expand Up @@ -93,7 +93,7 @@ backend:

sglang_config:
prefill:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down Expand Up @@ -128,7 +128,7 @@ backend:
linear-attn-decode-backend: "flashinfer"

decode:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down
6 changes: 3 additions & 3 deletions ...ti_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/8k1k_stp_maxtpt_1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Qwen3.5-397B-A17B-NVFP4 Disaggregated 6P1D wide-EP
# Qwen3.5-397B-A17B-NVFP4-V2 Disaggregated 6P1D wide-EP
# Prefill: 6 workers @ TP4/DP4/EP4 with DP-attn (per-node, DEP4)
# Decode: 1 worker @ TP16/DP16/EP16 with DP-attn + TBO (DEP16, 4 nodes)
# Total: 40 GB300 GPUs (6*4 + 4*4); 8k1k concurrency 5120.
Expand Down Expand Up @@ -91,7 +91,7 @@ backend:

sglang_config:
prefill:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down Expand Up @@ -126,7 +126,7 @@ backend:
linear-attn-decode-backend: "flashinfer"

decode:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down
6 changes: 3 additions & 3 deletions ...ti_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp4/8k1k/disagg/stp/8k1k_stp_maxtpt_2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Qwen3.5-397B-A17B-NVFP4 Disaggregated 7P1D wide-EP
# Qwen3.5-397B-A17B-NVFP4-V2 Disaggregated 7P1D wide-EP
# Prefill: 7 workers @ TP4/DP4/EP4 with DP-attn (per-node, DEP4)
# Decode: 1 worker @ TP16/DP16/EP16 with DP-attn + TBO (DEP16, 4 nodes)
# Total: 44 GB300 GPUs (7*4 + 4*4); 8k1k concurrency 5120.
Expand Down Expand Up @@ -91,7 +91,7 @@ backend:

sglang_config:
prefill:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down Expand Up @@ -126,7 +126,7 @@ backend:
linear-attn-decode-backend: "flashinfer"

decode:
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4"
served-model-name: "nvidia/Qwen3.5-397B-A17B-NVFP4-V2"
model-path: "/model/"
trust-remote-code: true

Expand Down
4 changes: 2 additions & 2 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11745,8 +11745,8 @@ glm5-fp4-gb200-dynamo-sglang:
dp-attn: false

qwen3.5-fp4-gb300-dynamo-sglang:
image: lmsysorg/sglang:nightly-dev-cu13-20260624-b2c8f7a2
model: nvidia/Qwen3.5-397B-A17B-NVFP4
image: lmsysorg/sglang:v0.5.14-cu130
model: nvidia/Qwen3.5-397B-A17B-NVFP4-V2

@functionstackx functionstackx Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is difference between nvidia/Qwen3.5-397B-A17B-NVFP4 & nvidia/Qwen3.5-397B-A17B-NVFP4-v2

model-prefix: qwen3.5
runner: gb300
precision: fp4
Expand Down
8 changes: 7 additions & 1 deletion perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4859,4 +4859,10 @@
description:
- "Add GB300 DeepSeek-V4 dynamo-sglang agentic recipes (agg-tp4 + dep8 disagg pareto)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2157


- config-keys:
- qwen3.5-fp4-gb300-dynamo-sglang
description:
- "Bump SGLang image to v0.5.14-cu130 for the Qwen3.5 FP4 GB300 disaggregated dynamo-sglang config"
- "Switch to the nvidia/Qwen3.5-397B-A17B-NVFP4-V2 checkpoint (model pre-staged on cluster as Qwen3.5-397B-A17B-NVFP4-V2)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2238
Comment on lines 4859 to +4868

@claude claude Bot Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This PR strips the two trailing spaces from the pre-existing blank separator line in perf-changelog.yaml (the one PR #2157 left as \n) before appending the new entry, turning it into a bare empty line. AGENTS.md explicitly forbids this exact change ("Never delete or modify whitespace in perf-changelog.yaml ... Altering it breaks CI"), and it will fail the check-changelog CI gate. Fix by restoring the original trailing-space line byte-for-byte and appending the new entry after it.

Extended reasoning...

What's wrong: The base file (as of PR #2157, commit 3355132) ends with the byte sequence ...pull/2157\n \n — the blank separator line after the last entry carries two trailing spaces. This PR's diff shows that line changing from - to + (a bare empty line) before the new entry is appended:

   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2157
-  
+
+- config-keys:
+    - qwen3.5-fp4-gb300-dynamo-sglang
...

Why it's forbidden: AGENTS.md is explicit and unambiguous on this point: "Never delete or modify whitespace in perf-changelog.yaml - CI depends on exact whitespace (including trailing spaces on blank separator lines). Altering it breaks CI." This diff does precisely the prohibited operation.

Why it will actually fail CI (not just a style nit): utils/validate_perf_changelog.py::validate_raw_change() is invoked by the check-changelog job in .github/workflows/run-sweep.yml (--base-ref origin/${{ github.base_ref }}). For any PR that only appends entries, it enforces:

if not head_raw.startswith(base_raw):
    raise ChangelogValidationError(
        "appended entries changed historical perf-changelog.yaml bytes; "
        "restore the base file byte-for-byte and append at the end"
    )

Step-by-step proof (verified directly against the repo history):

  1. git show 9af6bf5^:perf-changelog.yaml | tail -c 20 → b'renceX/pull/2157\n \n' — base ends with \n \n (two trailing spaces before EOF).
  2. git show 9af6bf5:perf-changelog.yaml (this PR's head) at the same offset reads ...pull/2157\n\n- config-keys:... — the trailing spaces are gone.
  3. A direct byte comparison (head_raw.startswith(base_raw)) confirms this is False: the first divergence is at byte offset 255901, where base has pull/2157\n \n and head has pull/2157\n\n- config-.
  4. Because additions is nonzero for this PR (a new - config-keys: block was appended), validate_raw_change() takes the if not head_raw.startswith(base_raw) branch and raises ChangelogValidationError, which fails the check-changelog job — and downstream jobs (needs: check-changelog) gate on its success.

The fix: Keep the original last line of the base file exactly as-is ( — two trailing spaces, no more, no less) and append the new entry as a new block after it, so head_raw starts with base_raw byte-for-byte. Concretely, restore the line to before the new - config-keys: block rather than collapsing it to an empty line.

2 changes: 1 addition & 1 deletion runners/launch_gb300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ elif [[ $MODEL_PREFIX == "kimik2.5" && $PRECISION == "fp4" ]]; then
elif [[ $MODEL_PREFIX == "qwen3.5" && $PRECISION == "fp4" ]]; then
# SRT_SLURM_MODEL_PREFIX must match the model.path alias used in our
# Qwen3.5 sglang recipes (qwen3.5-fp4).
export MODEL_PATH=/scratch/models/Qwen3.5-397B-A17B-NVFP4
export MODEL_PATH=/scratch/models/Qwen3.5-397B-A17B-NVFP4-V2
export SRT_SLURM_MODEL_PREFIX="qwen3.5-fp4"
elif [[ $MODEL_PREFIX == "qwen3.5" && $PRECISION == "fp8" ]]; then
# SRT_SLURM_MODEL_PREFIX must match the model.path alias used in our
Expand Down
Loading