Skip to content

fix(agentx): rerun DeepSeek-V4 B300 SGLang with OOM headroom / 修复 AgentX：为 DeepSeek-V4 B300 SGLang 重跑预留 OOM 余量 #2226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 2 commits into
base: main
Choose a base branch
from
+74 −15
Open

fix(agentx): rerun DeepSeek-V4 B300 SGLang with OOM headroom / 修复 AgentX：为 DeepSeek-V4 B300 SGLang 重跑预留 OOM 余量 #2226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d50b4d0
fix(agentx): rerun B300 SGLang with OOM headroom
cquil11 Jul 15, 2026
8295312
chore(perf): trigger B300 AgentX rerun
cquil11 Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 62 additions & 10 deletions benchmarks/single_node/agentic/dsv4_fp4_b300_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ install_agentic_deps
SERVER_LOG="$RESULT_DIR/server.log"
mkdir -p "$RESULT_DIR"

export SGLANG_DEFAULT_THINKING=1
export SGLANG_DSV4_REASONING_EFFORT=high
export SGLANG_ENABLE_UNIFIED_RADIX_TREE=1
export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1

Expand Down Expand Up @@ -105,7 +107,28 @@ PARALLEL_ARGS=(--tp "$TP")
METRICS_ARGS=(--enable-metrics)
MEM_FRACTION_STATIC=0.88
CHUNKED_PREFILL_SIZE=8192
SWA_FULL_TOKENS_RATIO=0.1
# Default (non-DP) attention path: compressed attention, shared-expert fusion off.
MODEL_ARGS=(
--attention-backend compressed
--page-size 256
--disable-shared-experts-fusion
)
CUDA_GRAPH_ARGS=()
EXTRA_ARGS=()
if [ "$DP_ATTENTION" = "true" ]; then
# DP-attention high-throughput agentic recipe (SGLang cookbook: DSV4-Pro FP4
# B300, high-throughput, single-node): MegaMoE DeepGEMM MoE (DeepEP dispatch +
# fused shared experts + kernel autotune), tuned for conc128.
# Measured DEP8 conc128 vs the previous flashinfer_mxfp4 path:
# 24,466 -> 33,220 tok/s/gpu (surpasses the vLLM reference of 28,962).
export SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1
export SGLANG_OPT_FIX_HASH_MEGA_MOE=1
export SGLANG_OPT_USE_FAST_MASK_EP=1
export SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1
export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320
export SGLANG_OPT_FIX_NEXTN_MEGA_MOE=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0
PARALLEL_ARGS+=(
--dp "$TP"
--tokenizer-worker-num "$TP"
Expand All @@ -115,29 +138,57 @@ if [ "$DP_ATTENTION" = "true" ]; then
--stream-interval 20
--dist-init-addr "127.0.0.1:$((PORT + 2000))"
--ep-size "$EP_SIZE"
--moe-a2a-backend megamoe
)
MEM_FRACTION_STATIC=0.85
SWA_FULL_TOKENS_RATIO=0.075
# MegaMoE requires the default (non-compressed) attention path and fused shared experts.
MODEL_ARGS=()
EXTRA_ARGS=(--enable-prefill-delayer)
# chunked-prefill is divided by dp_size (=TP) under DP-attention; keep the
# effective chunk at 8192. Size the decode graph to served concurrency:
# DEP8 (dp 8) serves conc up to 512, DEP4 (dp 4) tops out near 128.
if [ "$TP" = "8" ]; then
CHUNKED_PREFILL_SIZE=65536
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544)
else
# DEP4 (TP4/EP4) shards the model across only 4 GPUs, so per-GPU weights
# are ~2x DEP8 (233 GB loaded; profiler floor ~0.879). At 0.93, run
# 29367944489 left only 2.9-4.8 GiB free while the sparse-attention
# indexer requested 4.2-6.1 GiB, causing CUDA OOMs during warmup. Use
# 0.90 to restore about 8.6 GiB/GPU of activation/workspace headroom
# while remaining above the profiler floor.
MEM_FRACTION_STATIC=0.90
CHUNKED_PREFILL_SIZE=32768
CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 128)
Comment on lines +143 to +163

@claude claude Bot Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The DP-attention branch launches sglang.launch_server with --cuda-graph-max-bs-decode (lines 153/163), a flag name that appears nowhere else in this repo — every other launcher, including the sibling MegaMoE DEP8 recipe sizing the identical 544 decode-graph batch (fixed_seq_len/dsv4_fp4_b300_sglang.sh:113), uses the plain --cuda-graph-max-bs. If SGLang's argparse rejects the -decode suffix, the server will fail to start for every dp-attn: true config (3 of the 5 search-space rows), which is exactly the MegaMoE DEP configuration this PR exists to rerun. Please confirm against the pinned SGLang image's server_args that this flag is real before merging; separately, the comment above it ("DEP8 ... serves conc up to 512") is now stale since this PR's own nvidia-master.yaml diff drops 512 from the DEP8 conc-list, leaving 196 as the new max.

Extended reasoning...

The flag. CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544) (TP8, line 153) and (--cuda-graph-max-bs-decode 128) (TP4, line 163) are only reachable when DP_ATTENTION=true, which covers 3 of the 5 search-space rows in dsv4-fp4-b300-sglang-agentic-hicache (DEP4-none, DEP4-hicache, DEP8-none). A repo-wide search shows --cuda-graph-max-bs-decode exists nowhere else in InferenceX — not in any other SGLang launcher, and a git log -S pickaxe search shows this commit is the very first time the string has ever appeared in this repository's history. The strongest counter-evidence sits right next to it: benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh runs the same DeepSeek-V4 FP4 B300 MegaMoE DEP8 configuration (same --moe-a2a-backend megamoe, same SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK) and sizes the decode graph to the identical value, 544, but with the plain --cuda-graph-max-bs (line 113). This script's own non-DP fallback (line 191) also falls back to the plain flag. That is about as close to an apples-to-apples control as you get, and it points at a typo/transcription slip rather than an intentional, SGLang-supported decode-specific flag.\n\nWhy it matters. sglang.launch_server parses its CLI via argparse, which aborts the process with SystemExit(2) ("unrecognized arguments") on any unknown flag, before CUDA context creation or model load. If --cuda-graph-max-bs-decode isn't recognized by the pinned image (lmsysorg/sglang:nightly-dev-cu13-20260707-b4155233), every DP-attention job in this sweep — the exact B300 MegaMoE DEP recipe this PR is explicitly trying to preserve and rerun — fails immediately at startup. The PR's stated validation (bash -n, YAML parsing, load_config_files) only checks shell/YAML syntax; none of it constructs or exercises the actual sglang.launch_server argv, so this would slip through and only be caught once the GPU jobs actually run.\n\nAddressing the refutation. One verifier argued this is refuted by the PR's own evidence: the OOM analysis in the description says jobs from run 29367944489 reached CUDA warmup and died inside deep_gemm.fp8_paged_mqa_logits — which is downstream of argument parsing, so if the flag were invalid, those jobs would have died at parse time instead, before ever touching CUDA. That's a real observation, but it rests on an unverifiable assumption: that predecessor PR #2145's script used the exact same -decode spelling and that this PR's DEP4 lines are a byte-for-byte carryover. I could not find #2145 in this repository's git history (no remote is configured, and no prior commit ever contains "cuda-graph-max-bs-decode" per git log -S), and the PR description itself calls this a rebuild "on the latest main" of a PR whose diff had to be manually reconstructed (excluding B200 and changelog changes). A manual reconstruction is exactly the kind of step where a flag name could be mistyped even if the original #2145 script used the correct plain flag — we simply cannot confirm from what's in front of us that the run 29367944489 logs correspond to a config using the -decode spelling rather than the correct one. Given that ambiguity, and given that the one piece of concrete, in-repo, apples-to-apples evidence (the sibling fixed_seq_len recipe) uses the plain flag for the identical 544 value, I still think this needs the author to explicitly confirm the flag against the pinned SGLang image before merge rather than relying on an inference about an unrelated historical run.\n\nStep-by-step proof of the failure mode (if the flag is indeed invalid):\n1. A search-space entry with dp-attn: true, e.g. tp: 8, ep: 8, is scheduled — one of the 3 affected rows.\n2. The script enters the DP_ATTENTION=true branch (line ~119) and, since TP=8, sets CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544) (line 153).\n3. SGLANG_CMD is assembled including "" (line ~229) and executed via "" >> "" 2>&1 &.\n4. sglang.launch_server's argparse.ArgumentParser.parse_args() encounters --cuda-graph-max-bs-decode, finds no matching add_argument, and calls parser.error(...), which prints "unrecognized arguments: --cuda-graph-max-bs-decode 544" and exits with status 2 — before any CUDA/model initialization.\n5. wait_for_server_ready polls the health port, never gets a response (the process already exited), and the job fails/hangs on the timeout instead of ever reaching the benchmark — for all 3 dp-attn rows, defeating the PR's stated purpose of rerunning exactly this B300 MegaMoE DEP configuration.\n\nFix. Confirm --cuda-graph-max-bs-decode is a recognized argument in the pinned SGLang nightly's server_args.py/launch_server before merge (e.g. via python -m sglang.launch_server --help against the actual image, or grepping the vendored SGLang source in the container). If it isn't, change both occurrences (lines 153, 163) to the plain --cuda-graph-max-bs, consistent with the sibling MegaMoE recipe and this script's own non-DP fallback. While touching this block, the comment directly above ("DEP8 (dp 8) serves conc up to 512") is also worth updating since this PR's own configs/nvidia-master.yaml diff removes 512 from the DEP8 conc-list, leaving 196 as the new max — a cosmetic follow-up, not a blocker on its own.

fi
elif { [ "$TP" = "8" ] || [ "$TP" = "4" ]; } && [ "${CONC:-999}" -le 16 ]; then
# TP-only low-latency (TP4 or TP8, non-DP, conc <= 16): SGLang cookbook
# DSV4-Pro FP4 B300 low-latency single-node recipe (mirrors the fixed_seq_len
# MTP TP-only path) with speculative decoding (EAGLE) removed. Default
# attention + fused shared experts + DSV4 FP4 sparse-attention indexer;
# mem-fraction 0.90.
PARALLEL_ARGS+=(
--moe-runner-backend flashinfer_mxfp4
--disable-flashinfer-autotune
--enable-deepseek-v4-fp4-indexer
)
MEM_FRACTION_STATIC=0.95
CHUNKED_PREFILL_SIZE=16384
MEM_FRACTION_STATIC=0.90
MODEL_ARGS=(--page-size 256)
else
PARALLEL_ARGS+=(
--moe-runner-backend flashinfer_mxfp4
--disable-flashinfer-autotune
)
fi

MODEL_ARGS=(
--attention-backend compressed
--page-size 256
--disable-shared-experts-fusion
)

# AgentX concurrency counts live session trees, not individual requests.
# Allow subagent fan-out to exceed CONC without clipping request bursts.
MAX_RUNNING_REQUESTS=$((2 * CONC))
CUDA_GRAPH_MAX_BS=$CONC
[ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64
# Non-DP path keeps the concurrency-scaled decode graph; DP path set 544 above.
[ ${#CUDA_GRAPH_ARGS[@]} -eq 0 ] && CUDA_GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS")

export PYTHONNOUSERSITE=1
export TORCH_CUDA_ARCH_LIST=10.0
Expand Down Expand Up @@ -172,9 +223,9 @@ SGLANG_CMD=(
--trust-remote-code
"${PARALLEL_ARGS[@]}"
--mem-fraction-static "$MEM_FRACTION_STATIC"
--swa-full-tokens-ratio 0.1
--swa-full-tokens-ratio "$SWA_FULL_TOKENS_RATIO"
--max-running-requests "$MAX_RUNNING_REQUESTS"
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
"${CUDA_GRAPH_ARGS[@]}"
--allow-auto-truncate
--chunked-prefill-size "$CHUNKED_PREFILL_SIZE"
--tool-call-parser deepseekv4
Expand All @@ -183,6 +234,7 @@ SGLANG_CMD=(
--watchdog-timeout 1800
"${MODEL_ARGS[@]}"
"${METRICS_ARGS[@]}"
"${EXTRA_ARGS[@]}"
"${CACHE_ARGS[@]}"
)

Expand Down
10 changes: 5 additions & 5 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14264,11 +14264,11 @@ dsv4-fp4-b300-sglang-agentic-hicache:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 4, kv-offloading: none, conc-list: [1, 4, 8, 16, 20, 24, 32] }
- { tp: 8, kv-offloading: none, conc-list: [1, 4, 8, 16, 32, 40, 48, 52, 56, 60, 64, 72] }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, conc-list: [8, 16, 24, 32, 40, 64], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 40, 48, 56, 64, 72, 80, 88, 96, 128], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [52, 72, 100, 128, 144, 196, 512], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 8, 16] }
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8, 16, 32, 40] }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, conc-list: [8, 16, 24, 32, 40], router: { name: sglang-router, version: "0.3.2" } }
Comment on lines 14264 to +14269

@claude claude Bot Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This PR changes the B300 benchmark recipe (mem-fraction-static, MegaMoE DEP path, new low-latency TP path, several SGLANG_* env vars) and the corresponding search-space in configs/nvidia-master.yaml, but appends no entry to perf-changelog.yaml. Since run-sweep.yml is path-filtered on perf-changelog.yaml and process_changelog.py derives the entire sweep matrix from newly-appended changelog lines, merging as-is means the B300 rerun this PR exists to produce will never execute. Please append a perf-changelog.yaml entry keyed to dsv4-fp4-b300-sglang-agentic-hicache (config-key, description, PR link) at the end of the file.

Extended reasoning...

What the bug is: The PR's stated purpose is to "rerun only the B300 configuration" for dsv4-fp4-b300-sglang-agentic-hicache with a lowered mem-fraction-static (0.93 → 0.90) to fix an observed CUDA OOM, plus a broader recipe overhaul (MegaMoE DEP path, a new low-latency TP-only path, several new SGLANG_* env vars, and revised conc-list search-space entries). The changed-files set for this PR is exactly two files — benchmarks/single_node/agentic/dsv4_fp4_b300_sglang.sh and configs/nvidia-master.yaml. No entry is appended to perf-changelog.yaml.

The code path that triggers it: .github/workflows/run-sweep.yml gates both its push (lines 25-26) and pull_request (lines 35-36) triggers with paths: ["perf-changelog.yaml"]. Since this PR never touches that file, the entire sweep workflow (including check-changelog, the reuse-sweep gate, and the actual benchmark jobs) will not fire on this PR, and will not fire on the post-merge push to main either. Even setting aside the workflow-level path filter, the sweep's matrix is itself derived from perf-changelog.yaml: utils/process_changelog.py's get_added_lines() git-diffs only that file between base and head, and main() explicitly raises ValueError("No additions found in the changelog file.") when there are no appended lines (utils/process_changelog.py:128-129). The config-keys from each appended changelog entry are what get resolved against the master config via get_config_keys_from_master() to build the actual sweep — so an empty changelog diff means zero configs get selected, config-key resolution never runs, and the dsv4-fp4-b300-sglang-agentic-hicache search-space this PR just edited is never benchmarked.

Why existing code doesn't prevent it: There is no other trigger path. Labels like full-sweep-fail-fast only affect fail-fast behavior of matrix jobs that are already running — they don't bypass the perf-changelog.yaml path filter. validate_perf_changelog.py (or equivalent validation) does not error out when the changelog is left unchanged, so nothing blocks the merge; the sweep just silently never runs. Recent config-changing PRs in this repo (e.g. #2158/#2222, #2197, #2198) consistently append a changelog entry alongside their config edits, and the immediate predecessor to this PR (#2145) also lacks one, showing this omission isn't a one-off outlier but a real gap that AGENTS.md and the PR checklist both explicitly call out as required ("append a perf-changelog.yaml entry (required — triggers benchmarks)").

Impact: Concretely, merging this PR as-is makes it a functional no-op for its own stated purpose. The mem-fraction-static OOM fix, the MegaMoE DEP path, the new low-latency TP-only path, and the revised concurrency search-space all land on main, but none of them get exercised by CI — the exact scenario (DEP4 CUDA OOM in deep_gemm.fp8_paged_mqa_logits) that motivated this PR will not be re-validated. A maintainer or the author would need a follow-up PR just to actually trigger the benchmark.

Step-by-step proof:

  1. git diff main...HEAD -- perf-changelog.yaml for this PR is empty (only dsv4_fp4_b300_sglang.sh and nvidia-master.yaml are in <changed-files>).
  2. On pull_request, GitHub Actions evaluates run-sweep.yml's trigger: the event's changed-file set intersected with paths: ["perf-changelog.yaml"] is empty, so the workflow does not run at all, regardless of labels applied.
  3. Suppose the workflow were manually forced to run anyway (e.g., via workflow_dispatch): the setup job calls process_changelog.py --base-ref ... --head-ref ..., which calls get_added_lines() — a git diff scoped to perf-changelog.yaml only. Since no lines were added there, added_yaml.strip() is empty and main() raises ValueError("No additions found in the changelog file."), hard-failing before any config-key resolution or sweep construction happens.
  4. Either way, the dsv4-fp4-b300-sglang-agentic-hicache config — the one this PR just modified — never gets included in a benchmark run.

How to fix: Append a perf-changelog.yaml entry at the end of the file with config-key: dsv4-fp4-b300-sglang-agentic-hicache, a short description of the mem-fraction/MegaMoE/low-latency changes, and a link to this PR — mirroring the format used by the sibling PRs (#2158/#2222, #2197, #2198) that touched nvidia-master.yaml.

- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 128], router: { name: sglang-router, version: "0.3.2" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [52, 72, 100, 128, 136, 144, 152, 196], router: { name: sglang-router, version: "0.3.2" } }

# DEP8 prefill uses an 8K batch because 16K OOMs in the FP4 MoE intermediate;
# decode uses FULL_DECODE_ONLY after the controlled graph test restored decode
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4795,3 +4795,10 @@
- "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)."
- "TP8/TP4 GPU-only KV points plus a TP4 native CPU-offload ladder via SimpleCPUOffloadConnector with lazy_offload off; TP4/DCP4 high-concurrency points (conc 32/64) using num_speculative_tokens=3 and synthetic_acceptance_length=2.88."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2222

- config-keys:
- dsv4-fp4-b300-sglang-agentic-hicache
description:
- "Follow up on PR #2145 with the B300-only DeepSeek-V4-Pro FP4 SGLang AgentX MegaMoE DEP and low-latency TP recipes."
- "Reduce DEP4 mem-fraction-static from 0.93 to 0.90 after run 29367944489 exposed sparse-attention indexer CUDA OOMs during warmup; preserve the refined B300 concurrency search space and rerun only this configuration."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2226
Loading