Skip to content

[AgentX] vLLM DeepSeek-V4 B300 aggregate / vLLM DeepSeek-V4 B300 聚合配置 #2225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cquil11 wants to merge 3 commits into
base: main
Choose a base branch
from
+147 −100
Open

[AgentX] vLLM DeepSeek-V4 B300 aggregate / vLLM DeepSeek-V4 B300 聚合配置 #2225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fee87d8
feat(agentic): split DeepSeek-V4 B300 vLLM recipe
cquil11 Jul 15, 2026
2d1725b
docs(changelog): link B300 replacement PR
cquil11 Jul 15, 2026
407404b
chore: update conc
ivanium Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 130 additions & 88 deletions benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,60 +1,46 @@
#!/usr/bin/env bash
set -euo pipefail
set -eo pipefail
set -x

# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B300 using vLLM.
# Mirrors the fixed-seq-len parallelism options (pure TP and DEP) so the
# agentic sweep can probe both interactivity and throughput regimes:
# pure TP (DP_ATTENTION=false, EP_SIZE=1): attention TP-sharded across
# all $TP GPUs in a single engine. Lower TPOT, lower batch.
# TP+EP (DP_ATTENTION=false, EP_SIZE>1): attention TP-sharded, MoE
# experts EP-sharded within the TP group.
# DEP (DP_ATTENTION=true, EP_SIZE>1): per-DP-rank attention with
# experts EP-sharded across DP ranks (per the vLLM blog recipe).
# Highest aggregate throughput at large CONC.
# v4pro-b300.yaml TP4 and DEP4 recipe. SimpleCPUOffload / MooncakeStore
#
# Image is vllm/vllm-openai:v0.20.0-cu130. block_size=256, kv-cache-dtype=fp8,
# FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph capture with
# custom_ops=all (per the vLLM blog recipe at https://vllm.ai/blog/deepseek-v4).
# Image is configured in nvidia-master.yaml. The recipe uses FP8 KV cache,
# sparse DeepSeek-V4 FlashInfer attention with an FP4 indexer cache, mega-MoE,
# and FULL_DECODE_ONLY CUDA graphs with every batch size captured explicitly.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
#
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake.
# KV_OFFLOADING=none is used by TP4. DEP4 uses KV_OFFLOADING=dram with
# KV_OFFLOAD_BACKEND=vllm-simple or mooncake.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION

DCP_SIZE="${DCP_SIZE:-1}"
PCP_SIZE="${PCP_SIZE:-1}"
VLLM_CP_ARGS=()
if [ "$DCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE")
fi
if [ "$PCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE")
fi

GPU_COUNT="${GPU_COUNT:-$((TP * PCP_SIZE))}"
GPU_COUNT=$TP
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
exit 1
fi
export GPU_COUNT

if declare -p SLURM_JOB_ID >/dev/null 2>&1 && [ -n "$SLURM_JOB_ID" ]; then
SLURM_NODE=unknown
if declare -p SLURMD_NODENAME >/dev/null 2>&1 && [ -n "$SLURMD_NODENAME" ]; then
SLURM_NODE="$SLURMD_NODENAME"
fi
echo "JOB $SLURM_JOB_ID running on $SLURM_NODE"
# Under DP-attention the DP world size equals TP, and the DEP recipe sizes
# per-rank batch as MAX_NUM_SEQS = 2*CONC/TP, which must be an integer.
if [ "$DP_ATTENTION" = "true" ] && [ $((2 * CONC % TP)) -ne 0 ]; then
echo "Error: DEP requires 2*CONC divisible by TP, got CONC='$CONC' and TP='$TP'" >&2
exit 1
fi

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
# Either way, MODEL_PATH is what the server is launched with.
if declare -p MODEL_PATH >/dev/null 2>&1 && [ -n "$MODEL_PATH" ]; then
if [[ -n "$MODEL_PATH" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
Expand All @@ -68,17 +54,9 @@ nvidia-smi
resolve_trace_source
install_agentic_deps

# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings,
# which fails DSV4 indexer compilation with mlir_global_dtors(..., data).
# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved.
agentic_pip_install --quiet --force-reinstall --no-deps \
'nvidia-cutlass-dsl-libs-cu13==4.5.2'

# vllm-project/router expands the one HTTP backend into one logical worker per
# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's
# X-Correlation-ID is stable for every turn of a conversation; alias it to the
# router's preferred X-Session-ID header. This also keeps affinity correct when
# testing older wheels that prioritize per-request X-Request-ID.
# DP rank. Bind every turn of a conversation to the same rank by mapping
# AIPerf's stable correlation ID to the router's X-Session-ID header.
USE_VLLM_ROUTER=false
VLLM_BACKEND_PORT="$PORT"
if [ "$DP_ATTENTION" = "true" ]; then
Expand All @@ -91,13 +69,13 @@ if [ "$DP_ATTENTION" = "true" ]; then
agentic_pip_install --quiet "vllm-router==$VLLM_ROUTER_VERSION"
fi

# DeepSeek-V4-Pro weights are large; engine startup can exceed default 600s.
# Match the environment used by v4pro-b300.yaml.
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600

# vllm-project/vllm#43447 keeps local SWA prefix-cache tails sparsely, while
# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's
# store mask. 32k matches the trace-replay tuning validated for this workload.
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
export VLLM_DSV4_MEGA_FP8_COMBINE=1
export NCCL_NVLS_ENABLE=1
export VLLM_USE_RUST_FRONTEND=1

# ---- Server config ----------------------------------------------------------
SERVER_LOG="$RESULT_DIR/server.log"
Expand All @@ -109,13 +87,40 @@ SERVER_PID=""
ROUTER_PID=""
MOONCAKE_MASTER_PID=""

# The generated TOTAL_CPU_DRAM_GB budget is proportional to allocated GPUs.
# On cluster:b300-nv, dram-utilization=0.80 and DEP4 resolve to roughly the
# source recipe's 280 GiB per DP rank. TP4 remains GPU-resident.
OFFLOAD_ARGS=()
if require_agentic_kv_offload_backend mooncake; then
# Mooncake embedded mode contributes one global segment per GPU rank to
# a shared distributed store. Pre-divide the aggregate host budget
# across those rank-contributed segments.
case "$KV_OFFLOAD_BACKEND" in
"")
require_agentic_kv_offload_none
;;
vllm-simple)
require_agentic_kv_offload_backend vllm-simple
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
# Identical prefixes must hash to identical block keys across DP ranks.
export PYTHONHASHSEED=42
OFFLOAD_CONFIG=$(cat <<EOF
{
"kv_connector": "SimpleCPUOffloadConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"cpu_bytes_to_use": ${CPU_BYTES_PER_RANK},
"enable_cross_layers_blocks": "true"
}
}
EOF
)
OFFLOAD_ARGS=(
--kv-transfer-config
"$OFFLOAD_CONFIG"
Comment on lines +98 to +116

@claude claude Bot Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new vllm-simple branch (lines 98-116) builds a --kv-transfer-config naming SimpleCPUOffloadConnector but never exports VLLM_USE_SIMPLE_KV_OFFLOAD=1, unlike every other recipe in the repo that constructs this same connector (including ones using the identical explicit-JSON form, e.g. kimik2.5_int4_h200.sh:50-51, kimik2.5_fp4_b300_mtp.sh:74). Without it, the DEP4 sweep arm using kv-offload-backend: vllm-simple (conc [8,16,24,32,40,48,56,64,72] in nvidia-master.yaml) may not actually engage the intended SimpleCPUOffloadConnector semantics, risking misleading KV-offload benchmark results for that entire sweep row. Fix: add export VLLM_USE_SIMPLE_KV_OFFLOAD=1 right before building OFFLOAD_CONFIG in the vllm-simple case, mirroring the sibling recipes.

Extended reasoning...

The bug: In the new vllm-simple case of the KV_OFFLOAD_BACKEND switch (dsv4_fp4_b300_vllm.sh:98-116), the script builds a --kv-transfer-config JSON with kv_connector: SimpleCPUOffloadConnector, but nowhere in that branch does it export VLLM_USE_SIMPLE_KV_OFFLOAD=1. This is the only branch in the entire repository that constructs a SimpleCPUOffloadConnector transfer config without setting that env var.

Repo-wide convention (verified directly): Every sibling recipe that builds a SimpleCPUOffloadConnector kv-transfer-config sets VLLM_USE_SIMPLE_KV_OFFLOAD=1 immediately beforehand:

  • kimik2.5_fp4_b300_mtp.sh:74 — identical explicit JSON form.
  • kimik2.5_fp4_b300.sh:46, kimik2.5_int4_b200.sh:42, kimik2.5_int4_h100.sh:42 — shortcut form.
  • kimik2.5_int4_h200.sh:50-51 — explicit JSON form, and the surrounding comment explicitly says this JSON form is chosen 'rather than the --kv_offloading_backend native shortcut' to pass lazy_offload=true, and it still sets the env var on the line immediately before. This rules out the theory that the var is only needed for the shortcut-resolution path — it's required for the connector to actually take effect even when named explicitly in JSON.

Conversely, every branch that intentionally uses a different connector (OffloadingConnector, MooncakeStoreConnector, lmcache) defensively unsets the var: this same file's mooncake branch (line 170), dsv4_fp4_mi355x_vllm.sh:108/202/275, dsv4_fp4_b200_vllm.sh:165, kimik2.5_fp4_b200.sh:97, kimik2.5_fp4_mi355x.sh:68. The comment at dsv4_fp4_mi355x_vllm.sh:113-120 spells out the mechanism explicitly: the backend 'resolves to OffloadingConnector by default; setting VLLM_USE_SIMPLE_KV_OFFLOAD=1 would switch it to SimpleCPUOffloadConnector. We intentionally leave that env var UNSET here so the regular OffloadingConnector path is used.' That unset is only meaningful if a sibling branch is expected to set it — which is exactly the missing line in the new vllm-simple branch of this file.

Why existing code doesn't prevent it: There is no validation that the env var matches the requested connector; require_agentic_kv_offload_backend vllm-simple only checks that the backend name is allowed, not that the corresponding env var is exported. The script will run to completion and launch vLLM successfully either way — the failure mode is silent (a benchmark that runs and reports numbers, just for a possibly-wrong KV path), not a crash.

Step-by-step proof of the gap:

  1. A DEP4 sweep point runs with KV_OFFLOAD_BACKEND=vllm-simple, kv-offloading=dram.
  2. The script enters case "" in vllm-simple) at line 99.
  3. It computes CPU_BYTES_PER_RANK, sets PYTHONHASHSEED=42, and builds OFFLOAD_CONFIG naming kv_connector: SimpleCPUOffloadConnector — but at no point in this branch does it export VLLM_USE_SIMPLE_KV_OFFLOAD.
  4. Compare to kimik2.5_int4_h200.sh:50-51, which builds the functionally identical explicit-JSON SimpleCPUOffloadConnector config but sets the env var on the immediately preceding line — the validated, working pattern this PR's branch was clearly modeled on (matching field names kv_connector, kv_role: kv_both, kv_connector_extra_config) but missing this one line.
  5. vllm serve is invoked with "${OFFLOAD_ARGS[@]}" containing the JSON, but without the env var vLLM may not engage the SimpleCPUOffloadConnector-specific runtime behavior the recipe intends (the unanimous repo convention treats the var as required alongside the JSON, not redundant with it).
  6. This affects the entire DEP4 vllm-simple sweep row added in configs/nvidia-master.yaml (conc-list: [8,16,24,32,40,48,56,64,72]), whose purpose is specifically to characterize the SimpleCPU-offload cliff against the Mooncake row it sits beside — if the connector semantics aren't actually engaged, those 9 benchmark points measure the wrong thing while looking like valid data.

Fix: Add export VLLM_USE_SIMPLE_KV_OFFLOAD=1 in the vllm-simple) case, right before (or after) setting PYTHONHASHSEED=42, mirroring every sibling recipe.

Residual uncertainty: This PR pins a brand-new nightly image (vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec), so it's conceivable upstream vLLM changed connector resolution so the env var is no longer needed when the connector is named explicitly. However, the convention is unanimous across every other recipe in the repo — including ones using the identical explicit-JSON form — and the sibling mooncake branch in this very file still defensively unsets the var, which only makes sense if a peer branch sets it. The burden is on proving the exception, and the safe, minimal fix (add the one-line export) matches the validated pattern everywhere else.

)
;;
mooncake)
require_agentic_kv_offload_backend mooncake
# Embedded mode contributes one global segment per DP rank to the
# shared store, so divide the aggregate host budget across ranks.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))

MOONCAKE_VERSION=0.3.11.post1
agentic_pip_install --quiet --no-cache-dir --no-deps \
--force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION"
Expand All @@ -139,9 +144,7 @@ EOF
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
# Identical prefixes must hash to identical store keys across DP ranks.
export PYTHONHASHSEED=0
# Large agentic KV writes can exceed Mooncake Store's fixed 60-second
# transfer deadline at the default 64 KiB RDMA slice size. Reduce
# per-transfer bookkeeping and give the shared RNIC more workers.
export WITH_NVIDIA_PEERMEM=0
export MC_SLICE_SIZE=1048576
export MC_WORKERS_PER_CTX=4

Expand All @@ -165,54 +168,93 @@ EOF
fi

unset VLLM_USE_SIMPLE_KV_OFFLOAD
OFFLOAD_ARGS=(
--kv-transfer-config
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
)
fi
OFFLOAD_CONFIG='{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
OFFLOAD_ARGS=(--kv-transfer-config "$OFFLOAD_CONFIG")
;;
*)
echo "Error: unsupported B300 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
exit 1
;;
esac

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
fi

EP_ARGS=()
TP_ARGS=()
if [ "$DP_ATTENTION" = "true" ]; then
export PYTORCH_ALLOC_CONF=expandable_segments:True
else
export VLLM_ALLREDUCE_USE_FLASHINFER=1
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto
TP_ARGS+=(--disable-custom-all-reduce)
fi

MODE_ARGS=()
if [ "$EP_SIZE" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
MODE_ARGS+=(
--enable-expert-parallel
--enable-ep-weight-filter
--moe-backend deep_gemm_amxf4_mega_moe
)
fi
if [ "$DP_ATTENTION" = "true" ]; then
MODE_ARGS+=(
--prefill-schedule-interval 8
--max-num-batched-tokens 8192
)
fi

# AgentX concurrency counts live session trees, not individual requests.
# Subagent fan-out can push instantaneous request concurrency above CONC, so
# leave 2x headroom rather than clipping those bursts at the scheduler.
MAX_NUM_SEQS=$((2 * CONC))
if [ "$MAX_NUM_SEQS" -eq 128 ]; then
MAX_NUM_SEQS=136
if [ "$DP_ATTENTION" = "true" ]; then
# The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS.
MAX_NUM_SEQS=$((2 * CONC / TP))
else
# Preserve the previous TP4 scheduler headroom for agentic fan-out.
MAX_NUM_SEQS=$((2 * CONC))
fi
CUDA_GRAPH_CAPTURE_SIZES=""
for ((capture_size = 1; capture_size <= MAX_NUM_SEQS; capture_size++)); do
if [ -n "$CUDA_GRAPH_CAPTURE_SIZES" ]; then
CUDA_GRAPH_CAPTURE_SIZES+=","
fi
CUDA_GRAPH_CAPTURE_SIZES+="$capture_size"
done
COMPILATION_CONFIG="{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[${CUDA_GRAPH_CAPTURE_SIZES}],\"mode\":0}"

echo "Starting vllm server..."
export TORCH_CUDA_ARCH_LIST="10.0"
export PYTHONNOUSERSITE=1
export VLLM_FLOAT32_MATMUL_PRECISION=high

vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \
--host 0.0.0.0 \
--port "$VLLM_BACKEND_PORT" \
--trust-remote-code \
--kv-cache-dtype fp8 \
--block-size 256 \
"${PARALLEL_ARGS[@]}" \
"${VLLM_CP_ARGS[@]}" \
"${EP_ARGS[@]}" \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
--attention_config.use_fp4_indexer_cache=True \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--enable-prefix-caching \
--no-disable-hybrid-kv-cache-manager \
--max-num-seqs "$MAX_NUM_SEQS" \
"${OFFLOAD_ARGS[@]}" > "$SERVER_LOG" 2>&1 &
{ set +x; } 2>/dev/null
VLLM_CMD=(
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
--host 0.0.0.0
--port "$VLLM_BACKEND_PORT"
--gpu-memory-utilization 0.96
--trust-remote-code
--no-enable-flashinfer-autotune
--no-disable-hybrid-kv-cache-manager
--max-num-seqs "$MAX_NUM_SEQS"
--kv-cache-dtype fp8
--block-size 256
--max-model-len 1048576
--attention-config '{"use_fp4_indexer_cache":true,"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}'
--disable-uvicorn-access-log
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--compilation-config "$COMPILATION_CONFIG"
"${PARALLEL_ARGS[@]}"
"${TP_ARGS[@]}"
"${MODE_ARGS[@]}"
"${OFFLOAD_ARGS[@]}"
)
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt"
"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!
echo "Server PID: $SERVER_PID"

Expand Down
21 changes: 9 additions & 12 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3003,7 +3003,7 @@ dsr1-fp8-b200-trt:
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 64, conc-end: 128 }
- { tp: 4, ep: 1, conc-start: 8, conc-end: 16 }
- { tp: 4, ep: 1, conc-start: 8, conc-end: 16 }
- { tp: 8, ep: 1, conc-start: 4, conc-end: 8 }
- isl: 8192
osl: 1024
Expand Down Expand Up @@ -3217,7 +3217,7 @@ dsv4-fp4-b300-vllm:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 }

dsv4-fp4-b300-vllm-agentic:
image: vllm/vllm-openai:v0.23.0
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b300-nv
Expand All @@ -3228,16 +3228,13 @@ dsv4-fp4-b300-vllm-agentic:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Compare native GPU-cache and MooncakeStore CPU-offload cliffs.
- { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16] }
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [16, 18, 20, 24] }
# TP8 remains cache-resident through conc 52.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16, 32, 40, 48, 52] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [52] }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, conc-list: [8, 16], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [32], router: { name: vllm-router, version: "0.1.14" } }
# TP8 DEP retains representative low, peak, and transition points.
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [52, 72, 100, 128, 144], router: { name: vllm-router, version: "0.1.14" } }
# Preserve the previous GPU-resident TP4 search space on the new image.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16] }
- { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16] }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 72], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [32, 64, 96, 128, 160, 192, 224, 228], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [40, 72, 104, 136, 168, 200], router: { name: vllm-router, version: "0.1.14" } }

dsv4-fp4-b300-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4788,3 +4788,11 @@
- "Add EAGLE3 speculative-decoding arm for the Kimi K2.6 NVFP4 B300 AgentX recipe (draft lightseekorg/kimi-k2.6-eagle3-mla, TOKENSPEED_MLA attention backend with TRT-LLM ragged MLA kernel)."
- "TP8/TP4 GPU-only KV points plus a TP4 native CPU-offload ladder via SimpleCPUOffloadConnector with lazy_offload off; TP4/DCP4 high-concurrency points (conc 32/64) using num_speculative_tokens=3 and synthetic_acceptance_length=2.88."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2222

- config-keys:
- dsv4-fp4-b300-vllm-agentic
description:
- "Update B300 AgentX: KV offload, sparse DSV4 attention, mega-MoE, and FULL_DECODE_ONLY CUDA graphs."
- "Image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec"
- "B300: GPU-resident TP4 at conc [1,2,4,6,8,16]; DEP4 at conc [8,16,24,32,40,48,56,64,72]; DEP8 at conc [32,64,96,128,160,192,224,228]; with both SimpleCPU and Mooncake 0.3.11.post1."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2225
Loading