[NV] Add gb300 nvfp4 qwen3.5 MTP 8k1k recipes

[richardhuo-nv]

Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM + Dynamo MTP benchmarks (8k1k)

Summary

This PR adds multi-token prediction (MTP, 3 speculative tokens) benchmark recipes for Qwen3.5-397B-A17B-NVFP4 on GB300 in a disaggregated TensorRT-LLM + Dynamo setup, covering the 8k1k (ISL 8192 / OSL 1024) scenario.

Runner and master config changes follow the same pattern as PR #2206 (STP recipes), which must be merged (or the runner branch must be compatible) before CI jobs launched from this PR will succeed.

Changes

9 MTP recipe configs — benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/mtp/

All configs use:

• Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18
• Dynamo installed at runtime: 1.3.0.dev20260708, TCP request plane
• Model alias: qwen3.5-fp4 → /scratch/models/Qwen3.5-397B-A17B-NVFP4
• Prefill always DEP2 (2 GPUs, 1 node); decode topologies vary
• 3 MTP speculative tokens (mtp3 in filename)

Config	Prefill	Decode	Concurrency
ctx1dep2_gen1tep16_mtp3_c48	1P DEP2	1D TEP16	48
ctx1dep2_gen4tp16_mtp3_c4	1P DEP2	4D TP16	4
ctx1dep2_gen7tep8_mtp3_c42	1P DEP2	7D TEP8	42
ctx2dep2_gen15tp4_mtp3_c90	2P DEP2	15D TP4	90
ctx3dep2_gen4tep8_mtp3_c192	3P DEP2	4D TEP8	192
ctx3dep2_gen7tep8_mtp3_c168	3P DEP2	7D TEP8	168
ctx4dep2_gen1dep16_mtp3_c384	4P DEP2	1D DEP16	384
ctx6dep2_gen5tep8_mtp3_c480	6P DEP2	5D TEP8	480
ctx9dep2_gen1dep16_mtp3_c1536	9P DEP2	1D DEP16	1536

runners/launch_gb300-nv.sh

• Added elif branch for dynamo-trt + qwen3.5 + fp4: clones NVIDIA/srt-slurm@sa-submission-q2-2026 and overlays benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ (covers both stp/ and mtp/ subdirs)
• Added --no-preflight for this path — model lives on compute-node /scratch/ NVMe, not visible from the GHA runner pod

configs/nvidia-master.yaml

• New entry qwen3.5-fp4-gb300-dynamo-trt-mtp: 9 search-space points, all spec-decoding: "mtp", disagg: true, kv-p2p-transfer: nixl

perf-changelog.yaml

• New entry for qwen3.5-fp4-gb300-dynamo-trt-mtp

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Not approving — the pr-link: TBD issue flagged inline is real and will hard-fail the standard merge_with_reuse.sh canonicalization step (verified against utils/validate_perf_changelog.py and utils/prepare_perf_changelog_merge.py), so this needs a fix before merge.

Extended reasoning...

Overview

This PR adds 9 new MTP (multi-token prediction) benchmark recipe YAMLs for Qwen3.5-397B-A17B-NVFP4 on GB300, plus a corresponding configs/nvidia-master.yaml entry, a perf-changelog.yaml entry, and a new elif branch in runners/launch_gb300-nv.sh to route this model/precision/framework combination to the correct srt-slurm checkout and recipe overlay directory.

Security risks

None. This is benchmark configuration and a shell-script routing branch for an internal CI runner; no user input, auth, or secrets handling is involved.

Level of scrutiny

This falls into the repo's standard recipe-addition pattern (new YAML configs + master-config entry + changelog + runner routing), which is normally simple and mechanical. However, I independently verified the inline bug report's core claim by reading utils/validate_perf_changelog.py and utils/prepare_perf_changelog_merge.py: PR_LINK_PLACEHOLDERS is exactly {"XXX", ".../pull/XXX"}, and validate_added_pr_link/canonicalize_appended_links raise ChangelogValidationError for any pr-link value that isn't the canonical URL or one of those placeholders. The new changelog entry uses the literal string TBD, which matches neither, so the standard merge_with_reuse.sh path used by the merge-prs skill would hard-fail on this PR. That's a legitimate blocker for the standard merge tooling, even though it doesn't show up in PR-time CI checks (which don't invoke this validator).

Other factors

This is a one-line, easy fix (swap TBD for XXX or the real PR link), and the nit about mislabeling the recipes as "STP" instead of "MTP" in the changelog description is cosmetic. Neither issue affects the correctness of the recipe YAMLs, master-config entries, or runner routing logic themselves. Given the merge-blocking nature of the pr-link issue, I'm deferring rather than approving so a human/author can land the fix.

[claude]

🔴 The new perf-changelog.yaml entry (line 4792) sets pr-link: TBD, but the standard merge tooling (utils/merge_with_reuse.sh → prepare_perf_changelog_merge.py canonicalize) only accepts the canonical .../pull/2218 URL or the literal XXX placeholder — TBD matches neither and will raise ChangelogValidationError, hard-failing the merge step even though PR-time checks stay green. Fix by changing it to https://github.com/SemiAnalysisAI/InferenceX/pull/2218 (or XXX, which the tooling auto-replaces at merge time).

Extended reasoning...

What the bug is: Every other entry in perf-changelog.yaml fills pr-link with either a concrete https://github.com/SemiAnalysisAI/InferenceX/pull/<n> URL or the sanctioned XXX placeholder that merge tooling auto-fills. This PR's new entry (appended at line 4792) instead uses the literal string TBD, which is not a value the changelog tooling recognizes.

Where it breaks: utils/validate_perf_changelog.py defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. Its validate_added_pr_link(link, pr_number) function checks: if a PR number is known, the link must equal the expected canonical URL or be one of PR_LINK_PLACEHOLDERS — otherwise it raises ChangelogValidationError("new PR entry must use '<url>' or an XXX placeholder; found '<link>'"). TBD satisfies neither condition.

Why PR-time checks don't catch it: The sweep-gate path (run-sweep.yml → validate_perf_changelog.py main → process_changelog.py) does not call validate_added_pr_link at all — and separately, ChangelogEntry.pr_link in matrix_logic/validation.py is an unconstrained str, so any string passes Pydantic validation. This is exactly why the issue is invisible in the PR's green checks.

Where it does fire: utils/merge_with_reuse.sh, the repository's standard reuse-assisted merge path (used by the merge-prs skill), unconditionally runs prepare_perf_changelog_merge.py canonicalize --pr-number <PR> after merging main (line 181), regardless of whether a conflict occurred. canonicalize_appended_links() calls compare_entries(base, head, pr_number), which for every appended entry calls validate_added_pr_link(entry['pr-link'], pr_number). With pr_number=2218 and link='TBD', this raises ChangelogValidationError: new PR entry must use 'https://github.com/SemiAnalysisAI/InferenceX/pull/2218' or an XXX placeholder; found 'TBD'.

Step-by-step proof:

1. PR [NV] Add gb300 nvfp4 qwen3.5 MTP 8k1k recipes  #2218 is merged/rebased via merge_with_reuse.sh.
2. After the main-merge step, the script calls prepare_perf_changelog_merge.py canonicalize --changelog-file perf-changelog.yaml --base-ref origin/main --pr-number 2218 --repo SemiAnalysisAI/InferenceX.
3. canonicalize_appended_links parses base vs. head, finds one appended entry (qwen3.5-fp4-gb300-dynamo-trt-mtp), and calls validate_added_pr_link('TBD', 2218).
4. expected = 'https://github.com/SemiAnalysisAI/InferenceX/pull/2218'; 'TBD' not in PR_LINK_PLACEHOLDERS and 'TBD' != expected → ChangelogValidationError is raised.
5. merge_with_reuse.sh exits non-zero at that step, blocking the automated merge until someone manually fixes the changelog entry.

Impact and fix: Had the author used the sanctioned XXX placeholder, canonicalize_appended_links would have auto-rewritten it to the real /pull/2218 link and the merge would proceed silently. Using TBD instead turns what should be a no-op auto-fix into a hard failure of the standard merge tooling. The fix is a one-line change: replace TBD with either XXX or the literal https://github.com/SemiAnalysisAI/InferenceX/pull/2218.

[claude]

🟡 The new perf-changelog.yaml entry for qwen3.5-fp4-gb300-dynamo-trt-mtp mislabels the 9 new recipes as "9 STP configs" (line ~4790), but this PR adds MTP (multi-token prediction) recipes, not STP — likely a copy-paste leftover from the sibling STP PR #2206. Should read "9 MTP configs" for accuracy.

Extended reasoning...

The new perf-changelog.yaml entry for qwen3.5-fp4-gb300-dynamo-trt-mtp is internally self-contradictory. The first description bullet correctly states "Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM+Dynamo MTP (3-token) benchmarks for 8k1k," but the third bullet begins "9 STP configs: 1P1D DEP2+1D TEP16 (conc 48), ...". STP (single-token prediction, i.e. no speculative decoding) and MTP (multi-token prediction, 3 speculative draft tokens) are different decoding modes, and this entry unambiguously documents the latter.

Every piece of evidence in this PR confirms these are MTP recipes: the master-config key is qwen3.5-fp4-gb300-dynamo-trt-mtp with every search-space point set to spec-decoding: "mtp"; all 9 new recipe filenames under benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/mtp/ contain mtp3; and every recipe's speculative_config block sets decoding_type: MTP with max_draft_len: 3. There is no STP recipe anywhere in this diff.

The likely root cause is that this changelog entry was drafted by copying the changelog entry from the sibling STP recipes PR (#2206), which does add STP configs, and the "STP" wording in the count line was never updated to match the MTP content being described here — while the first bullet was correctly edited to say "MTP."

Step-by-step proof:

1. Read perf-changelog.yaml lines 4784-4792 (new entry): bullet 1 says "...MTP (3-token) benchmarks..."; bullet 3 says "9 STP configs: ...".
2. Read configs/nvidia-master.yaml, new key qwen3.5-fp4-gb300-dynamo-trt-mtp: every one of the 9 search-space entries sets spec-decoding: "mtp".
3. Read any of the 9 new recipe YAMLs, e.g. ctx1dep2_gen1tep16_mtp3_c48.yaml: both prefill.speculative_config.decoding_type and decode.speculative_config.decoding_type are MTP with max_draft_len: 3.
4. Filenames themselves encode mtp3 (3-token MTP), not stp.
5. Therefore "STP" in bullet 3 is factually wrong for the content it describes and contradicts bullet 1 of the same entry.

This is purely a documentation string in a changelog file with no functional/runtime impact — it doesn't affect recipe execution, CI, or the master config — so it does not block merge. The fix is a one-word edit: change "9 STP configs" to "9 MTP configs" on line 4790.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29378593024
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29378593024