[WIP] feat(minimaxm3): add B200 disaggregated configs

[xinli-sw]

No description provided.

[claude]

🔴 All 4 new B200 MiniMax-M3 recipes set dynamo.install: false (version 1.2.1) under a frontend.type: dynamo, but the byte-identical container image is used by the existing B300-fp4 recipes, which all require dynamo.install: true (version 1.3.0.dev20260614) to run. If dynamo isn't pre-bundled in this image (which the B300 requirement implies), the disaggregated dynamo frontend/router will fail to start on B200, producing no benchmark results for any of the 4 new configs. Same issue on lines ~129-132 of the other 3 new recipe files (3p2d-c256, 3p2d-c512, 4p2d-c1024).

Extended reasoning...

All 4 new B200 recipes (2p2d-dep2-tp4-c128, 3p2d-dep2-tp4-c256, 3p2d-dep2-tp4-c512, 4p2d-dep2-tp4-c1024) set:

dynamo:
  install: false
  version: 1.2.1

with frontend.type: dynamo (a disaggregated dynamo frontend/router). The exact same container image, vllm/vllm-openai:vllm-minimax-m3-perf-x86_64-13.0.1-8b00f41, is already used by all 8 existing MiniMax-M3 B300-fp4/8k1k recipes, and every one of those sets dynamo.install: true with version: 1.3.0.dev20260614. A repo-wide grep confirms these 4 new files are the only recipes anywhere under srt-slurm-recipes that combine a dynamo frontend with install: false (158 recipes use install: true; the rest omit the key entirely — none use false). A comment on an unrelated sglang/deepseek-v4 recipe (disagg-gb300-*.yaml) explicitly documents the rationale for this field elsewhere in the tree: dynamo.install: true is needed because "the container ... has no dynamo baked in." Since the vLLM MiniMax-M3 image is byte-identical between B200 and B300, the natural reading is that it likewise lacks a pre-installed dynamo runtime, and disabling install on B200 would leave the disaggregated frontend/router with nothing to launch — causing the job to fail at startup and the sweep to produce zero results.

I checked runners/launch_b200-dgxc.sh for the new B200 branch this PR adds (elif [[ $FRAMEWORK == "dynamo-vllm" && $MODEL_PREFIX == "minimaxm3" ... ]]): it only clones srt-slurm and copies the recipe directory in; there is no separate, out-of-band dynamo install step, so the recipe's own dynamo.install field is what governs whether the runtime gets installed for this launch path (same as for every other multinode recipe in the tree).

Addressing the refutation: one reviewer pointed out that version: 1.2.1 is not a stray leftover — it exactly matches the router: { name: dynamo-router, version: "1.2.1" } entry newly added to configs/nvidia-master.yaml for minimaxm3-fp4-b200-dynamo-vllm, and argued this reflects a deliberate, coordinated change (both install and version moved together) rather than a copy/paste mistake, possibly because B200 provisions dynamo through a different path tied to that master-config router entry. I looked into this: the same router: { name: dynamo-router, version: X } field exists on dozens of other master-config entries (e.g. qwen3.5-fp4-gb300-dynamo-sglang → version 1.1.0, matching that recipe's dynamo: version: "1.1.0", which also has no install key at all — not install: false). This is the established pattern across the repo: the master-config router.version simply mirrors whatever version is declared in the recipe, and it is not shown anywhere in this repo (launch scripts, utils/matrix_logic, ingest code) to trigger a separate dynamo-provisioning mechanism that would make install: false safe. So while the version pin is clearly deliberate rather than accidental, that doesn't establish that flipping install to false is also correct — it's equally consistent with the author having intentionally set the version to match the new master-config entry while mistakenly (or unverifiedly) also disabling the install step, not realizing this image needs it just like its B300 counterpart.

Step-by-step, the likely failure mode:

1. launch_b200-dgxc.sh hits the new elif branch for dynamo-vllm/minimaxm3/fp4, clones srt-slurm, copies in recipes/vllm/minimax-m3/b200-fp4, and later runs uv pip install -e . (installs srtctl only — not dynamo).
2. srtctl apply -f "$CONFIG_FILE" ... reads the recipe YAML, sees dynamo.install: false, and skips installing the dynamo runtime package/binaries into the container/venv used to launch the job.
3. The recipe's frontend.type: dynamo still expects to start a dynamo frontend/router process for the disaggregated prefill/decode topology.
4. Because the image itself apparently doesn't bundle dynamo (as evidenced by the B300 recipes on the identical image needing install: true to function), the frontend/router process fails to start (missing binary/import error).
5. The job fails at launch for all 4 new configs, and the PR's stated goal — adding validated B200 disaggregated benchmark points — produces no usable results.

Fix: set dynamo.install: true for all 4 new recipes (mirroring the B300 recipes on the same image), and confirm whether version: 1.2.1 is actually the correct/compatible version to install for this image, or whether it should match the validated 1.3.0.dev20260614 used on B300. Given the residual uncertainty raised by the refutation, it would also be worth a quick sanity check against srt-slurm's handling of dynamo.install (or a smoke-test launch) before merge, since the intent behind the coordinated version change is plausible even though the install: false value itself looks like the actual mistake.

[claude]

🟡 All four new B200 minimax-m3 recipes set slurm.time_limit: "01:00:00", well below the launcher's 4h default and the 3-8h used elsewhere in the tree, and none define a health_check block, so the launcher's sed that bumps max_attempts 360→720 (meant to absorb cold-cache loads off shared FS) is a no-op here. The model loads from shared scratch FS, so a slow cold load could eat the 1h walltime and kill the job in TIMEOUT with no results — worth double-checking against the sibling B300 recipes (no time_limit override, health_check present) before merge.

Extended reasoning...

The bug: all four new recipes (2p2d-c128, 3p2d-c256, 3p2d-c512, 4p2d-c1024) set slurm.time_limit: "01:00:00". This overrides the generated srtslurm.yaml's default_time_limit: "4:00:00" (set in runners/launch_b200-dgxc.sh) downward to a quarter of that. Across the whole srt-slurm-recipes tree, the only other explicit time_limit values used are 3h and 8h — 1h is an outlier by a wide margin, and it's the only class of minimax-m3 recipe that sets it this low.\n\nWhy the launcher's usual safety net doesn't apply: runners/launch_b200-dgxc.sh has a sed step specifically to absorb slow cold-cache model loads off shared FS:\n\nThis only works if the recipe already has a health_check.max_attempts line to rewrite (going from 360×10s=3600s to 720×10s=7200s). None of the four new recipes define a health_check section at all, so this sed matches nothing and is a silent no-op. Compare to the sibling B300-fp4 minimax-m3 recipes (same checkpoint, same container image), which set no slurm.time_limit override (inheriting the 4h default) and still carry health_check.max_attempts: 360, letting the sed extend their health-check budget to 2h. The new B200 recipes get neither protection.\n\nWhy this matters here specifically: MODEL_PATH for minimaxm3+fp4 resolves to /scratch/fsw/models/MiniMax-M3-NVFP4, shared scratch FS — the same class of cold-cache-slow-load risk that motivated both the sed and the 8h time_limit/1440 max_attempts used by the sibling deepseek-v4 disagg recipes (comment there: "to absorb cold-cache model loads"). If a cold cache causes a slow multi-worker MoE weight load (the 4p2d/c1024 config spans 4 nodes and has the most startup coordination), the health-check window and remaining walltime could both be exhausted before srtctl ever runs the benchmark sweep, and SLURM kills the job in TIMEOUT — a multi-node B200 allocation consumed with zero results collected, which is a much worse failure mode than a clean error.\n\nProof sketch: launch job → sed looks for ^ max_attempts: [0-9]* in the recipe, finds no match (none of the 4 files have a health_check key), no-op → srtctl applies the recipe with its default health-check window (360×10s=3600s=1h) → model load from /scratch/fsw/models/MiniMax-M3-NVFP4 under cold cache takes long enough to approach that 1h health-check budget → SLURM's own 1h wall-clock limit (time_limit: "01:00:00") expires around the same time the load might still be settling → job terminates in TIMEOUT before the benchmark concurrency point runs → no results_concurrency_*.json produced, wasted allocation.\n\nCounterpoint worth weighing: a reviewer flagged that each of these recipes runs a single concurrency point (skip_initial_test: true, one value in concurrencies) rather than a multi-point sweep like the B300 sibling that relies on the 4h default, and that MiniMax-M3-NVFP4 is a much smaller 4-bit checkpoint than the ~680GB DSR1-FP8 case the sed comment references — so 1h may be a deliberate, proportionately-sized choice rather than an oversight, and the perf-changelog even labels these "validated" configurations. That's plausible, but it doesn't fully explain why these are the only minimax-m3 recipes with no health_check block at all (even the B300 siblings running fewer/no sweep points keep one), and a single occurrence of cold cache during CI reruns would still zero out the run. Given the WIP status and that this is easy to align with the siblings, it's worth a second look before merge.\n\nFix: either drop the slurm.time_limit override (inherit the 4h default like the B300 siblings) or add a health_check block (e.g. max_attempts: 360) so the launcher's sed can extend it, matching the pattern used everywhere else in the tree.

[claude]

🟡 The new dynamo-vllm/minimaxm3/fp4 branch clones NVIDIA/srt-slurm and copies recipes in without a git checkout <ref>, so it floats on whatever the default branch resolves to at run time. Every sibling branch in this if/elif chain (dsv4, dsr1/fp4, glm5/fp8, and the catch-all else) explicitly pins a ref after cloning — this looks like a forgotten step rather than a deliberate choice. Consider adding git checkout main (or a pinned ref/commit) to match the pattern used by the other branches.

Extended reasoning...

What the bug is: In runners/launch_b200-dgxc.sh the new elif arm for FRAMEWORK == "dynamo-vllm" && MODEL_PREFIX == "minimaxm3" && PRECISION == "fp4" (lines 115-121) does:

git clone https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR"
cd "$SRT_REPO_DIR" || exit 1
mkdir -p recipes/vllm/minimax-m3/b200-fp4
cp -rT ...

There is no git checkout <ref> between the clone and the recipe copy, so the working tree stays on whatever git clone resolves to as HEAD — normally the repository's default branch — at whatever commit exists on it at run time.

Why this stands out: every other branch in the same if/elif chain pins explicitly:

• dsv4 (the other dynamo-vllm branch): git checkout aflowers/vllm-gb200-v0.20.0
• dsr1/fp4 (sglang): checks out the literal commit a98738de9b2233459b5456e9ed71af09ce893f92, with a comment a few lines below explaining exactly why this matters: "Pin srt-slurm: newer commits stopped honoring the hash-pinned dynamo build and fall back to a dynamo release that is incompatible with this sglang image (worker fails at import)... Do not float on main -- the srtctl + dynamo-install toolchain is unpinned there."
• glm5/fp8: explicit git checkout main (redundant today, but a deliberate, documented choice)
• the catch-all else: git checkout sa-submission-q2-2026

The new minimaxm3/fp4 branch is the only one that neither pins a ref nor explicitly selects main — it's an implicit, undocumented float.

Why this isn't prevented elsewhere: srtctl is installed from this same cloned repo and invoked via srtctl apply for every framework, including dynamo-vllm, so the exact hazard documented in the dsr1 comment (upstream changes to the srtctl/dynamo-install toolchain silently breaking or changing worker behavior) applies equally to this new branch. Nothing else in the script re-pins or validates the srt-slurm ref later.

Step-by-step proof of the failure mode:

1. Today, git clone https://github.com/NVIDIA/srt-slurm.git checks out srt-slurm's current default branch (e.g. main) at whatever commit is HEAD right now.
2. This sweep runs successfully against that commit, since the recipes it copies in are only YAML config, not the srtctl/dynamo toolchain code itself.
3. At some future date, an unrelated upstream commit lands on srt-slurm main that changes srtctl behavior or the dynamo build/install path — exactly the kind of change the dsr1 comment describes ("newer commits stopped honoring the hash-pinned dynamo build and fall back to a dynamo release that is incompatible").
4. The next run of this minimaxm3/fp4 branch clones the same URL, lands on the new main commit with no checkout to override it, and either fails outright (worker import failure) or silently produces different benchmark numbers — with no corresponding change to this repo to explain why.
5. Compare to dsv4 or dsr1, which are immune to this because they pin a ref; the minimaxm3/fp4 branch has no such protection.

Suggested fix: add an explicit ref selection after the clone, mirroring the sibling branches — e.g. git checkout main (matching glm5/fp8's deliberate choice) or a pinned commit/branch if a specific known-good state exists, so the sweep's toolchain version is explicit and reviewable rather than implicit.

Severity: All verifiers agree this is nit, not normal. Nothing fails today — the clone lands on the repo's current default branch, which is exactly what the glm5/fp8 sibling deliberately uses via git checkout main and which works currently. The concern is speculative future drift (the exact mechanism the dsr1 comment warns about), not a concrete failure at merge time, and the recipe here also sets dynamo: install: false, partially mitigating the dynamo-install portion of that documented hazard. This is a real reproducibility/maintainability gap worth flagging and fixing, but it does not block the PR.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29458030349
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29458030349

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29461201009
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29461201009

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29463463002
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29463463002

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29468181344
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29468181344

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471584525
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29471584525