Skip to content

[AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 3 commits into
base: main
Choose a base branch
from
+90 −42
Open

[AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ebc6af9
feat(agentic): split DeepSeek-V4 B200 vLLM recipe
cquil11 Jul 15, 2026
64973b3
docs(changelog): link B200 replacement PR
cquil11 Jul 15, 2026
455eabc
chore: fix mooncake dev and conc
ivanium Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 74 additions & 32 deletions benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
set -eo pipefail

@claude claude Bot Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Line 2 changed set -euo pipefail to set -eo pipefail, dropping nounset — this is the only script among all 24 in benchmarks/single_node/agentic/ that doesn't use -u. The rewrite replaced every ${VAR:-default}-guarded reference ($DCP_SIZE, $SLURM_JOB_ID, $SLURMD_NODENAME, $MODEL_PATH, $KV_OFFLOAD_BACKEND) with bare $VAR forms, which is why -u had to go; the near-identical dsv4_fp4_b300_vllm.sh shows the same optional-var pattern works fine with -u kept and ${VAR:-} guards. Recommend restoring set -euo pipefail and guarding the new bare references, to keep this script's safety net consistent with its siblings.

Extended reasoning...

The bug: Line 2 of dsv4_fp4_b200_vllm.sh was changed from set -euo pipefail to set -eo pipefail, silently dropping the nounset (-u) flag. Every other script in benchmarks/single_node/agentic/ — including the near-identical dsv4_fp4_b300_vllm.sh, plus dsv4_fp4_mi355x_vllm.sh, dsv4_fp4_b200_sglang.sh, dsv4_fp8_h200.sh, and all the kimik2.5_*/qwen3.5_*/minimaxm3_* variants — still uses set -euo pipefail. This script is now the sole outlier in the directory.\n\nWhy it happened: The diff rewrote several previously-guarded optional-variable references into bare, unguarded forms: DCP_SIZE=${DCP_SIZE:-1} became an if [ -z "$DCP_SIZE" ]; then DCP_SIZE=1; fi block (which still reads the bare $DCP_SIZE first), ${SLURM_JOB_ID:-}/${SLURMD_NODENAME:-unknown} became bare $SLURM_JOB_ID/$SLURMD_NODENAME, ${MODEL_PATH:-} became bare $MODEL_PATH, and the new case "$KV_OFFLOAD_BACKEND" in "") branch reads $KV_OFFLOAD_BACKEND unguarded (note it isn't even part of check_env_vars, so it can legitimately be unset). None of these would survive set -u if the variable is genuinely unset, so -u had to be dropped to make the rewritten code work as written.\n\nWhy this is avoidable: The sibling dsv4_fp4_b300_vllm.sh proves -u and this exact optional-variable pattern coexist fine: it keeps set -euo pipefail while using DCP_SIZE="${DCP_SIZE:-1}", declare -p SLURM_JOB_ID >/dev/null 2>&1 && [ -n "$SLURM_JOB_ID" ], and similar guards for MODEL_PATH. The new case statement here could equally have matched on "${KV_OFFLOAD_BACKEND:-}" instead of the bare variable. So dropping -u wasn't required by the case-statement refactor — it was a side effect of un-guarding several other references at the same time.\n\nImpact: With -u disabled, a future typo in a variable name anywhere in this script (e.g. RESULT_DIR misspelled as RESUT_DIR) will silently expand to an empty string and continue executing rather than aborting immediately with an "unbound variable" error — exactly the failure mode -u exists to catch. As written today the script runs correctly (all currently-referenced optional vars are handled, and required vars are separately validated by check_env_vars), so there's no concrete runtime failure right now. This is a latent robustness/consistency regression rather than an active bug.\n\nProof by example: Suppose a future edit adds echo "Writing to $RESUT_DIR" (a typo of $RESULT_DIR) somewhere in this script. With set -eo pipefail (current state), $RESUT_DIR expands to "", the echo prints "Writing to ", and the script continues — the typo goes unnoticed until output is inspected downstream. With set -euo pipefail (every sibling script's setting), the same line would immediately abort with RESUT_DIR: unbound variable, catching the bug at the point of introduction. This is precisely why the convention exists across all 23 other scripts in the directory.\n\nFix: Restore set -euo pipefail on line 2 and guard the newly-unguarded references the same way dsv4_fp4_b300_vllm.sh does: DCP_SIZE="${DCP_SIZE:-1}"/PCP_SIZE="${PCP_SIZE:-1}", [[ -n "${SLURM_JOB_ID:-}" ]] / ${SLURMD_NODENAME:-unknown}, [[ -n "${MODEL_PATH:-}" ]], and case "${KV_OFFLOAD_BACKEND:-}" in.

set -x

# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B200 using vLLM.
Expand All @@ -14,21 +14,25 @@ set -x
# Highest aggregate throughput at large CONC.
#
# Image is configured in nvidia-master.yaml. block_size=256,
# kv-cache-dtype=fp8, FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph
# capture with custom_ops=all (per the vLLM blog recipe at
# https://vllm.ai/blog/deepseek-v4).
# kv-cache-dtype=fp8, FLASHINFER_MLA_SPARSE_DSV4 attention with the FP4 indexer
# cache, FULL_DECODE_ONLY cudagraph capture, and (in EP tiers) mega-MoE backend.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
#
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake.
# Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host
# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple or mooncake.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION

DCP_SIZE="${DCP_SIZE:-1}"
PCP_SIZE="${PCP_SIZE:-1}"
if [ -z "$DCP_SIZE" ]; then
DCP_SIZE=1
fi
if [ -z "$PCP_SIZE" ]; then
PCP_SIZE=1
fi
VLLM_CP_ARGS=()
if [ "$DCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE")
Expand All @@ -37,21 +41,21 @@ if [ "$PCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE")
fi

GPU_COUNT="${GPU_COUNT:-$((TP * PCP_SIZE))}"
GPU_COUNT=$TP
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
exit 1
fi
export GPU_COUNT

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ -n "$MODEL_PATH" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
Expand All @@ -65,12 +69,6 @@ nvidia-smi
resolve_trace_source
install_agentic_deps

# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings,
# which fails DSV4 indexer compilation with mlir_global_dtors(..., data).
# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved.
agentic_pip_install --quiet --force-reinstall --no-deps \
'nvidia-cutlass-dsl-libs-cu13==4.5.2'

# vllm-project/router expands the one HTTP backend into one logical worker per
# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's
# X-Correlation-ID is stable for every turn of a conversation; alias it to the
Expand All @@ -94,6 +92,10 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600
# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's
# store mask. 32k matches the trace-replay tuning validated for this workload.
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_USE_RUST_FRONTEND=1
export VLLM_DSV4_MEGA_FP8_COMBINE=1
export VLLM_RPC_TIMEOUT=600000

# ---- Server config ----------------------------------------------------------
SERVER_LOG="$RESULT_DIR/server.log"
Expand All @@ -106,8 +108,34 @@ ROUTER_PID=""
MOONCAKE_MASTER_PID=""

OFFLOAD_ARGS=()

if require_agentic_kv_offload_backend mooncake; then
case "$KV_OFFLOAD_BACKEND" in
"")
require_agentic_kv_offload_none
;;
vllm-simple)
require_agentic_kv_offload_backend vllm-simple
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
# Identical prefixes must hash to identical block keys across DP ranks.
export PYTHONHASHSEED=42
OFFLOAD_CONFIG=$(cat <<EOF
{
"kv_connector": "SimpleCPUOffloadConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"cpu_bytes_to_use_per_rank": ${CPU_BYTES_PER_RANK},
"lazy_offload": false,
"enable_cross_layers_blocks": "true"
}
}
EOF
)
OFFLOAD_ARGS=(
--kv-transfer-config
"$OFFLOAD_CONFIG"
)
;;
mooncake)
require_agentic_kv_offload_backend mooncake
# Embedded mode contributes one segment per GPU rank to a shared
# distributed store, so pre-divide the aggregate host-memory budget.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))
Expand All @@ -127,16 +155,14 @@ if require_agentic_kv_offload_backend mooncake; then
"global_segment_size": "${PER_RANK_GB}GB",
"local_buffer_size": "4GB",
"protocol": "rdma",
"device_name": "mlx5_0",
"device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_10,mlx5_11",
"enable_offload": false
}
EOF
export MOONCAKE_CONFIG_PATH
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
# Identical prefixes must hash to identical store keys across DP ranks.
export PYTHONHASHSEED=0
# B200 GPU memory registration works through DMA-BUF, but the compute
# nodes do not expose nvidia_peermem. Force Mooncake's DMA-BUF
# GPUDirect RDMA path instead of its legacy ibv_reg_mr path.
export WITH_NVIDIA_PEERMEM=0
export MC_SLICE_SIZE=1048576
export MC_WORKERS_PER_CTX=4
Expand Down Expand Up @@ -167,16 +193,27 @@ EOF
--kv-transfer-config
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
)
fi
;;
*)
echo "Error: unsupported B200 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
exit 1
;;
esac

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
fi

EP_ARGS=()
FAST_MOE_ARGS=()
if [ "$EP_SIZE" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
FAST_MOE_ARGS=(
--moe-backend deep_gemm_amxf4_mega_moe
--enable-ep-weight-filter
--prefill-schedule-interval 16
)
fi

# AgentX concurrency counts live session trees, not individual requests.
Expand All @@ -197,18 +234,23 @@ VLLM_CMD=(
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
--attention_config.use_fp4_indexer_cache=True
--max-model-len 1048576
--gpu-memory-utilization 0.92
--numa-bind
--enable-cumem-allocator
--no-enable-flashinfer-autotune
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--enable-prefix-caching
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
--no-disable-hybrid-kv-cache-manager
--disable-uvicorn-access-log
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
--max-num-seqs "$MAX_NUM_SEQS"
--max-cudagraph-capture-size "$MAX_NUM_SEQS"
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
"${FAST_MOE_ARGS[@]}"
"${OFFLOAD_ARGS[@]}"
)
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
Expand Down
18 changes: 8 additions & 10 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1771,7 +1771,7 @@ dsv4-fp4-b200-vllm:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 }

dsv4-fp4-b200-vllm-agentic:
image: vllm/vllm-openai:v0.23.0
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b200-dgxc
Expand All @@ -1782,14 +1782,12 @@ dsv4-fp4-b200-vllm-agentic:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Pure TP is only competitive at very low concurrency.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5] }
# Sample the useful MooncakeStore range without repeating its collapsed
# high-concurrency tail.
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [8, 10, 16] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 38, 44, 50], router: { name: vllm-router, version: "0.1.14" } }
# Retain the external-cache transition and peak-throughput region.
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [16, 38, 44, 56, 64, 66, 68], router: { name: vllm-router, version: "0.1.14" } }
# Pure TP at low concurrency.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] }
# DEP
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }

dsv4-fp4-b200-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
Expand Down Expand Up @@ -3003,7 +3001,7 @@ dsr1-fp8-b200-trt:
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 64, conc-end: 128 }
- { tp: 4, ep: 1, conc-start: 8, conc-end: 16 }
- { tp: 4, ep: 1, conc-start: 8, conc-end: 16 }
- { tp: 8, ep: 1, conc-start: 4, conc-end: 8 }
- isl: 8192
osl: 1024
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4788,3 +4788,11 @@
- "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)."
- "TP8/TP4 GPU-only KV points plus a TP4 native CPU-offload ladder via SimpleCPUOffloadConnector with lazy_offload off; TP4/DCP4 high-concurrency points (conc 32/64) using num_speculative_tokens=3 and synthetic_acceptance_length=2.88."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2222

- config-keys:
- dsv4-fp4-b200-vllm-agentic
description:
- "Update B200 AgentX: KV offload, sparse DSV4 attention, mega-MoE, and FULL_DECODE_ONLY CUDA graphs."
- "Image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec"
- "B200: GPU-resident TP8 at conc [1,2,4,6,8] and SimpleCPU-offload TP8 at conc [8,12,16]; DEP8 at conc [8,16,24,32,40,48,56,64,68,72] with both SimpleCPU and Mooncake 0.3.11.post1."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2224