diff --git a/.github/codeowner-signoff-verify-prompt.md b/.github/codeowner-signoff-verify-prompt.md index ad0879c17..5f400f6ea 100644 --- a/.github/codeowner-signoff-verify-prompt.md +++ b/.github/codeowner-signoff-verify-prompt.md @@ -121,7 +121,7 @@ For the commit that passed Check 1, confirm the eval numbers are real and meet t the same inference-engine image as this PR's config. FAIL if evals are skipped/failed/empty/below bar, or the image differs — say exactly which. -## Check 3 — Recipe linked AND complete (SINGLE-NODE recipes only) +## Check 3 — Recipe linked, MERGED, AND complete (SINGLE-NODE recipes only) APPLICABILITY — read this first: the recipe-link requirement covers SINGLE-NODE recipes only, because the official upstream recipe sources (vLLM recipes, SGLang cookbook) publish single-node serve commands. Disaggregated / multi-node @@ -134,23 +134,31 @@ exclusively multi-node/disagg — files under `benchmarks/multi_node/**` (includ single-node recipes only` and DO NOT fail it. A sign-off note like "this is a disagg submission, no recipe update required" is a legitimate statement of that fact, not a violation. If the PR touches BOTH single-node and multi-node recipes, -apply (a)/(b) below to the single-node portion only. +apply (a)/(b)/(c) below to the single-node portion only. The InferenceX "recipe" for this PR = the files it changes under `benchmarks/single_node/**` plus its entry in `configs/*-master.yaml`. The merge -standard is: the community must be able to reproduce this benchmark from a public -recipe. +standard is: the community must be able to reproduce this benchmark from merged, +public upstream documentation. - (a) LINK PRESENT: The sign-off's "Additional detail section" MUST contain a link to - the corresponding recipe — a PR/commit in + the corresponding merged recipe PR in `https://github.com/vllm-project/recipes` or `https://github.com/sgl-project/sglang` (cookbook under `docs_new`), or the published recipe page (`https://recipes.vllm.ai/` or `https://docs.sglang.io/cookbook/...`). If no such link is present, FAIL. -- (b) MAJOR SERVER ARGS MATCH: Fetch the linked recipe (use the `fetch` MCP tool or - WebFetch; for a recipe PR, read its diff via `gh pr diff` against that repo if - accessible) and compare it to this PR's launch command. The recipe only needs to - match the MAJOR, deployment-defining server args — NOT every flag, and explicitly - NOT the knobs that are specific to InferenceX benchmark/harness tuning. +- (b) UPSTREAM CHANGE MERGED: For a linked GitHub PR, query the upstream repository + directly (for example, `gh pr view --json state,mergedAt,url`) and require + `state: MERGED` with a non-null `mergedAt`. An open PR, draft PR, closed-unmerged + PR, bare branch, or bare commit does NOT pass. A published recipe/cookbook page + containing the required recipe counts as merged upstream documentation. If the + linked artifact's merge/publication status cannot be verified, FAIL; never infer + that it merged from an approval, a green check, or the sign-off author's claim. +- (c) MAJOR SERVER ARGS MATCH: Fetch the merged or published recipe (use the `fetch` + MCP tool or WebFetch; for a merged recipe PR, read its diff via `gh pr diff` against + that repo if accessible) and compare it to this PR's launch command. The recipe + only needs to match the MAJOR, deployment-defining server args — NOT every flag, + and explicitly NOT the knobs that are specific to InferenceX benchmark/harness + tuning. MAJOR (must match — these define the model, parallelism, precision, and which kernels run, so they determine the perf profile): - model / model-path, hardware/SKU @@ -165,13 +173,14 @@ recipe. `--scheduler-recv-interval`, `--chunked-prefill-size`, `--disable-piecewise-cuda-graph`, `SGLANG_RADIX_FORCE_MISS` and similar env toggles, concurrency / sequence-length sweep ranges, ports, result filenames, and image tag/version. - FAIL only if a MAJOR arg in this PR is missing from (or contradicts) the recipe; - list exactly those. Treat the InferenceX-specific diffs as expected and mention them - only as a brief informational note, not as blockers. If a flag's effect is - equivalent to a recipe default (e.g. quantization auto-detected from an FP4 model), - say so and do not count it against the recipe. + FAIL if a MAJOR arg in this PR is missing from (or contradicts) the merged/published + recipe; list exactly those. Treat the InferenceX-specific diffs as expected and + mention them only as a brief informational note, not as blockers. If a flag's effect + is equivalent to a recipe default (e.g. quantization auto-detected from an FP4 + model), say so and do not count it against the recipe. - Note: a bare "recipes are already similar to the official ones" claim WITHOUT a - link does not pass this workflow's standard — a link is required. + link to merged/published upstream documentation does not pass this workflow's + standard. ## Check 4 — Reuse-sweep command explicitly posted The supported merge path for an approved PR is reuse (`utils/merge_with_reuse.sh`), diff --git a/benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh b/benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh index 35cc55c04..a16114a49 100755 --- a/benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -set -euo pipefail +set -eo pipefail set -x # Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B200 using vLLM. @@ -14,21 +14,25 @@ set -x # Highest aggregate throughput at large CONC. # # Image is configured in nvidia-master.yaml. block_size=256, -# kv-cache-dtype=fp8, FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph -# capture with custom_ops=all (per the vLLM blog recipe at -# https://vllm.ai/blog/deepseek-v4). +# kv-cache-dtype=fp8, FLASHINFER_MLA_SPARSE_DSV4 attention with the FP4 indexer +# cache, FULL_DECODE_ONLY cudagraph capture, and (in EP tiers) mega-MoE backend. # # Required env vars: # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR # -# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake. +# Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host +# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple or mooncake. source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION -DCP_SIZE="${DCP_SIZE:-1}" -PCP_SIZE="${PCP_SIZE:-1}" +if [ -z "$DCP_SIZE" ]; then + DCP_SIZE=1 +fi +if [ -z "$PCP_SIZE" ]; then + PCP_SIZE=1 +fi VLLM_CP_ARGS=() if [ "$DCP_SIZE" -gt 1 ]; then VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE") @@ -37,21 +41,21 @@ if [ "$PCP_SIZE" -gt 1 ]; then VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE") fi -GPU_COUNT="${GPU_COUNT:-$((TP * PCP_SIZE))}" +GPU_COUNT=$TP if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2 exit 1 fi export GPU_COUNT -if [[ -n "${SLURM_JOB_ID:-}" ]]; then - echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" +if [[ -n "$SLURM_JOB_ID" ]]; then + echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" fi # `hf download` creates the target dir if missing and is itself idempotent. # When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE # Either way, MODEL_PATH is what the server is launched with. -if [[ -n "${MODEL_PATH:-}" ]]; then +if [[ -n "$MODEL_PATH" ]]; then if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then hf download "$MODEL" --local-dir "$MODEL_PATH" fi @@ -65,12 +69,6 @@ nvidia-smi resolve_trace_source install_agentic_deps -# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings, -# which fails DSV4 indexer compilation with mlir_global_dtors(..., data). -# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved. -agentic_pip_install --quiet --force-reinstall --no-deps \ - 'nvidia-cutlass-dsl-libs-cu13==4.5.2' - # vllm-project/router expands the one HTTP backend into one logical worker per # DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's # X-Correlation-ID is stable for every turn of a conversation; alias it to the @@ -94,6 +92,10 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600 # vllm-project/vllm#44774 applies the same reachability policy to Mooncake's # store mask. 32k matches the trace-replay tuning validated for this workload. export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 +export VLLM_USE_V2_MODEL_RUNNER=1 +export VLLM_USE_RUST_FRONTEND=1 +export VLLM_DSV4_MEGA_FP8_COMBINE=1 +export VLLM_RPC_TIMEOUT=600000 # ---- Server config ---------------------------------------------------------- SERVER_LOG="$RESULT_DIR/server.log" @@ -106,8 +108,34 @@ ROUTER_PID="" MOONCAKE_MASTER_PID="" OFFLOAD_ARGS=() - -if require_agentic_kv_offload_backend mooncake; then +case "$KV_OFFLOAD_BACKEND" in + "") + require_agentic_kv_offload_none + ;; + vllm-simple) + require_agentic_kv_offload_backend vllm-simple + CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT )) + # Identical prefixes must hash to identical block keys across DP ranks. + export PYTHONHASHSEED=42 + OFFLOAD_CONFIG=$(cat <&2 + exit 1 + ;; +esac PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1) if [ "$DP_ATTENTION" = "true" ]; then @@ -175,8 +206,14 @@ if [ "$DP_ATTENTION" = "true" ]; then fi EP_ARGS=() +FAST_MOE_ARGS=() if [ "$EP_SIZE" -gt 1 ]; then EP_ARGS=(--enable-expert-parallel) + FAST_MOE_ARGS=( + --moe-backend deep_gemm_amxf4_mega_moe + --enable-ep-weight-filter + --prefill-schedule-interval 16 + ) fi # AgentX concurrency counts live session trees, not individual requests. @@ -197,18 +234,23 @@ VLLM_CMD=( --trust-remote-code --kv-cache-dtype fp8 --block-size 256 - "${PARALLEL_ARGS[@]}" - "${VLLM_CP_ARGS[@]}" - "${EP_ARGS[@]}" - --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' - --attention_config.use_fp4_indexer_cache=True + --max-model-len 1048576 + --gpu-memory-utilization 0.92 + --numa-bind + --enable-cumem-allocator + --no-enable-flashinfer-autotune --tokenizer-mode deepseek_v4 - --tool-call-parser deepseek_v4 - --enable-auto-tool-choice --reasoning-parser deepseek_v4 - --enable-prefix-caching + --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}' --no-disable-hybrid-kv-cache-manager + --disable-uvicorn-access-log + --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' --max-num-seqs "$MAX_NUM_SEQS" + --max-cudagraph-capture-size "$MAX_NUM_SEQS" + "${PARALLEL_ARGS[@]}" + "${VLLM_CP_ARGS[@]}" + "${EP_ARGS[@]}" + "${FAST_MOE_ARGS[@]}" "${OFFLOAD_ARGS[@]}" ) printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" diff --git a/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh b/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh index 3a5be7b57..72f2736fe 100755 --- a/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh +++ b/benchmarks/single_node/agentic/dsv4_fp4_b300_vllm.sh @@ -1,60 +1,46 @@ #!/usr/bin/env bash -set -euo pipefail +set -eo pipefail set -x # Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B300 using vLLM. -# Mirrors the fixed-seq-len parallelism options (pure TP and DEP) so the -# agentic sweep can probe both interactivity and throughput regimes: -# pure TP (DP_ATTENTION=false, EP_SIZE=1): attention TP-sharded across -# all $TP GPUs in a single engine. Lower TPOT, lower batch. -# TP+EP (DP_ATTENTION=false, EP_SIZE>1): attention TP-sharded, MoE -# experts EP-sharded within the TP group. -# DEP (DP_ATTENTION=true, EP_SIZE>1): per-DP-rank attention with -# experts EP-sharded across DP ranks (per the vLLM blog recipe). -# Highest aggregate throughput at large CONC. +# v4pro-b300.yaml TP4 and DEP4 recipe. SimpleCPUOffload / MooncakeStore # -# Image is vllm/vllm-openai:v0.20.0-cu130. block_size=256, kv-cache-dtype=fp8, -# FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph capture with -# custom_ops=all (per the vLLM blog recipe at https://vllm.ai/blog/deepseek-v4). +# Image is configured in nvidia-master.yaml. The recipe uses FP8 KV cache, +# sparse DeepSeek-V4 FlashInfer attention with an FP4 indexer cache, mega-MoE, +# and FULL_DECODE_ONLY CUDA graphs with every batch size captured explicitly. # # Required env vars: # MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR # -# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake. +# KV_OFFLOADING=none is used by TP4. DEP4 uses KV_OFFLOADING=dram with +# KV_OFFLOAD_BACKEND=vllm-simple or mooncake. source "$(dirname "$0")/../../benchmark_lib.sh" check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION -DCP_SIZE="${DCP_SIZE:-1}" -PCP_SIZE="${PCP_SIZE:-1}" -VLLM_CP_ARGS=() -if [ "$DCP_SIZE" -gt 1 ]; then - VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE") -fi -if [ "$PCP_SIZE" -gt 1 ]; then - VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE") -fi - -GPU_COUNT="${GPU_COUNT:-$((TP * PCP_SIZE))}" +GPU_COUNT=$TP if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2 exit 1 fi export GPU_COUNT -if declare -p SLURM_JOB_ID >/dev/null 2>&1 && [ -n "$SLURM_JOB_ID" ]; then - SLURM_NODE=unknown - if declare -p SLURMD_NODENAME >/dev/null 2>&1 && [ -n "$SLURMD_NODENAME" ]; then - SLURM_NODE="$SLURMD_NODENAME" - fi - echo "JOB $SLURM_JOB_ID running on $SLURM_NODE" +# Under DP-attention the DP world size equals TP, and the DEP recipe sizes +# per-rank batch as MAX_NUM_SEQS = 2*CONC/TP, which must be an integer. +if [ "$DP_ATTENTION" = "true" ] && [ $((2 * CONC % TP)) -ne 0 ]; then + echo "Error: DEP requires 2*CONC divisible by TP, got CONC='$CONC' and TP='$TP'" >&2 + exit 1 +fi + +if [[ -n "$SLURM_JOB_ID" ]]; then + echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" fi # `hf download` creates the target dir if missing and is itself idempotent. -# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE +# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE. # Either way, MODEL_PATH is what the server is launched with. -if declare -p MODEL_PATH >/dev/null 2>&1 && [ -n "$MODEL_PATH" ]; then +if [[ -n "$MODEL_PATH" ]]; then if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then hf download "$MODEL" --local-dir "$MODEL_PATH" fi @@ -68,17 +54,9 @@ nvidia-smi resolve_trace_source install_agentic_deps -# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings, -# which fails DSV4 indexer compilation with mlir_global_dtors(..., data). -# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved. -agentic_pip_install --quiet --force-reinstall --no-deps \ - 'nvidia-cutlass-dsl-libs-cu13==4.5.2' - # vllm-project/router expands the one HTTP backend into one logical worker per -# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's -# X-Correlation-ID is stable for every turn of a conversation; alias it to the -# router's preferred X-Session-ID header. This also keeps affinity correct when -# testing older wheels that prioritize per-request X-Request-ID. +# DP rank. Bind every turn of a conversation to the same rank by mapping +# AIPerf's stable correlation ID to the router's X-Session-ID header. USE_VLLM_ROUTER=false VLLM_BACKEND_PORT="$PORT" if [ "$DP_ATTENTION" = "true" ]; then @@ -91,13 +69,13 @@ if [ "$DP_ATTENTION" = "true" ]; then agentic_pip_install --quiet "vllm-router==$VLLM_ROUTER_VERSION" fi -# DeepSeek-V4-Pro weights are large; engine startup can exceed default 600s. +# Match the environment used by v4pro-b300.yaml. +export VLLM_USE_V2_MODEL_RUNNER=1 export VLLM_ENGINE_READY_TIMEOUT_S=3600 - -# vllm-project/vllm#43447 keeps local SWA prefix-cache tails sparsely, while -# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's -# store mask. 32k matches the trace-replay tuning validated for this workload. export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 +export VLLM_DSV4_MEGA_FP8_COMBINE=1 +export NCCL_NVLS_ENABLE=1 +export VLLM_USE_RUST_FRONTEND=1 # ---- Server config ---------------------------------------------------------- SERVER_LOG="$RESULT_DIR/server.log" @@ -109,13 +87,40 @@ SERVER_PID="" ROUTER_PID="" MOONCAKE_MASTER_PID="" +# The generated TOTAL_CPU_DRAM_GB budget is proportional to allocated GPUs. +# On cluster:b300-nv, dram-utilization=0.80 and DEP4 resolve to roughly the +# source recipe's 280 GiB per DP rank. TP4 remains GPU-resident. OFFLOAD_ARGS=() -if require_agentic_kv_offload_backend mooncake; then - # Mooncake embedded mode contributes one global segment per GPU rank to - # a shared distributed store. Pre-divide the aggregate host budget - # across those rank-contributed segments. +case "$KV_OFFLOAD_BACKEND" in + "") + require_agentic_kv_offload_none + ;; + vllm-simple) + require_agentic_kv_offload_backend vllm-simple + CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT )) + # Identical prefixes must hash to identical block keys across DP ranks. + export PYTHONHASHSEED=42 + OFFLOAD_CONFIG=$(cat <&2 + exit 1 + ;; +esac PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1) if [ "$DP_ATTENTION" = "true" ]; then PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP") fi -EP_ARGS=() +TP_ARGS=() +if [ "$DP_ATTENTION" = "true" ]; then + export PYTORCH_ALLOC_CONF=expandable_segments:True +else + export VLLM_ALLREDUCE_USE_FLASHINFER=1 + export VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto + TP_ARGS+=(--disable-custom-all-reduce) +fi + +MODE_ARGS=() if [ "$EP_SIZE" -gt 1 ]; then - EP_ARGS=(--enable-expert-parallel) + MODE_ARGS+=( + --enable-expert-parallel + --enable-ep-weight-filter + --moe-backend deep_gemm_amxf4_mega_moe + ) +fi +if [ "$DP_ATTENTION" = "true" ]; then + MODE_ARGS+=( + --prefill-schedule-interval 8 + --max-num-batched-tokens 8192 + ) fi -# AgentX concurrency counts live session trees, not individual requests. -# Subagent fan-out can push instantaneous request concurrency above CONC, so -# leave 2x headroom rather than clipping those bursts at the scheduler. -MAX_NUM_SEQS=$((2 * CONC)) -if [ "$MAX_NUM_SEQS" -eq 128 ]; then - MAX_NUM_SEQS=136 +if [ "$DP_ATTENTION" = "true" ]; then + # The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS. + MAX_NUM_SEQS=$((2 * CONC / TP)) +else + # Preserve the previous TP4 scheduler headroom for agentic fan-out. + MAX_NUM_SEQS=$((2 * CONC)) fi +CUDA_GRAPH_CAPTURE_SIZES="" +for ((capture_size = 1; capture_size <= MAX_NUM_SEQS; capture_size++)); do + if [ -n "$CUDA_GRAPH_CAPTURE_SIZES" ]; then + CUDA_GRAPH_CAPTURE_SIZES+="," + fi + CUDA_GRAPH_CAPTURE_SIZES+="$capture_size" +done +COMPILATION_CONFIG="{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[${CUDA_GRAPH_CAPTURE_SIZES}],\"mode\":0}" echo "Starting vllm server..." export TORCH_CUDA_ARCH_LIST="10.0" export PYTHONNOUSERSITE=1 export VLLM_FLOAT32_MATMUL_PRECISION=high -vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \ ---host 0.0.0.0 \ ---port "$VLLM_BACKEND_PORT" \ ---trust-remote-code \ ---kv-cache-dtype fp8 \ ---block-size 256 \ -"${PARALLEL_ARGS[@]}" \ -"${VLLM_CP_ARGS[@]}" \ -"${EP_ARGS[@]}" \ ---compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \ ---attention_config.use_fp4_indexer_cache=True \ ---tokenizer-mode deepseek_v4 \ ---tool-call-parser deepseek_v4 \ ---enable-auto-tool-choice \ ---reasoning-parser deepseek_v4 \ ---enable-prefix-caching \ ---no-disable-hybrid-kv-cache-manager \ ---max-num-seqs "$MAX_NUM_SEQS" \ -"${OFFLOAD_ARGS[@]}" > "$SERVER_LOG" 2>&1 & +{ set +x; } 2>/dev/null +VLLM_CMD=( + vllm serve "$MODEL_PATH" --served-model-name "$MODEL" + --host 0.0.0.0 + --port "$VLLM_BACKEND_PORT" + --gpu-memory-utilization 0.96 + --trust-remote-code + --no-enable-flashinfer-autotune + --no-disable-hybrid-kv-cache-manager + --max-num-seqs "$MAX_NUM_SEQS" + --kv-cache-dtype fp8 + --block-size 256 + --max-model-len 1048576 + --attention-config '{"use_fp4_indexer_cache":true,"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true}' + --disable-uvicorn-access-log + --tokenizer-mode deepseek_v4 + --tool-call-parser deepseek_v4 + --enable-auto-tool-choice + --reasoning-parser deepseek_v4 + --compilation-config "$COMPILATION_CONFIG" + "${PARALLEL_ARGS[@]}" + "${TP_ARGS[@]}" + "${MODE_ARGS[@]}" + "${OFFLOAD_ARGS[@]}" +) +printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" +printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" +"${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 & SERVER_PID=$! echo "Server PID: $SERVER_PID" diff --git a/configs/nvidia-master.yaml b/configs/nvidia-master.yaml index 73327351e..8f53d3712 100644 --- a/configs/nvidia-master.yaml +++ b/configs/nvidia-master.yaml @@ -1771,7 +1771,7 @@ dsv4-fp4-b200-vllm: - { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 } dsv4-fp4-b200-vllm-agentic: - image: vllm/vllm-openai:v0.23.0 + image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec model: deepseek-ai/DeepSeek-V4-Pro model-prefix: dsv4 runner: cluster:b200-dgxc @@ -1782,14 +1782,12 @@ dsv4-fp4-b200-vllm-agentic: agentic-coding: - dram-utilization: 0.80 search-space: - # Pure TP is only competitive at very low concurrency. - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5] } - # Sample the useful MooncakeStore range without repeating its collapsed - # high-concurrency tail. - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [8, 10, 16] } - - { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 38, 44, 50], router: { name: vllm-router, version: "0.1.14" } } - # Retain the external-cache transition and peak-throughput region. - - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [16, 38, 44, 56, 64, 66, 68], router: { name: vllm-router, version: "0.1.14" } } + # Pure TP is GPU-resident (cache fits) at low concurrency. + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] } + - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] } + # DEP compares SimpleCPUOffloadConnector and MooncakeStoreConnector. + - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72], router: { name: vllm-router, version: "0.1.14" } } + - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72], router: { name: vllm-router, version: "0.1.14" } } dsv4-fp4-b200-trt: image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 @@ -3003,7 +3001,7 @@ dsr1-fp8-b200-trt: osl: 1024 search-space: - { tp: 8, ep: 1, conc-start: 64, conc-end: 128 } - - { tp: 4, ep: 1, conc-start: 8, conc-end: 16 } + - { tp: 4, ep: 1, conc-start: 8, conc-end: 16 } - { tp: 8, ep: 1, conc-start: 4, conc-end: 8 } - isl: 8192 osl: 1024 @@ -3217,7 +3215,7 @@ dsv4-fp4-b300-vllm: - { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 2048 } dsv4-fp4-b300-vllm-agentic: - image: vllm/vllm-openai:v0.23.0 + image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec model: deepseek-ai/DeepSeek-V4-Pro model-prefix: dsv4 runner: cluster:b300-nv @@ -3228,16 +3226,11 @@ dsv4-fp4-b300-vllm-agentic: agentic-coding: - dram-utilization: 0.80 search-space: - # Compare native GPU-cache and MooncakeStore CPU-offload cliffs. - - { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16] } - - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [16, 18, 20, 24] } - # TP8 remains cache-resident through conc 52. - - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16, 32, 40, 48, 52] } - - { tp: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [52] } - - { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, conc-list: [8, 16], router: { name: vllm-router, version: "0.1.14" } } - - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [32], router: { name: vllm-router, version: "0.1.14" } } - # TP8 DEP retains representative low, peak, and transition points. - - { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [52, 72, 100, 128, 144], router: { name: vllm-router, version: "0.1.14" } } + # Preserve the previous GPU-resident TP4 search space on the new image. + - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4] } + - { tp: 4, kv-offloading: none, conc-list: [1, 2, 4, 6, 8, 16] } + - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 72], router: { name: vllm-router, version: "0.1.14" } } + - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 72], router: { name: vllm-router, version: "0.1.14" } } dsv4-fp4-b300-trt: image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066 diff --git a/docs/PR_REVIEW_CHECKLIST.md b/docs/PR_REVIEW_CHECKLIST.md index 01387e152..79f527258 100644 --- a/docs/PR_REVIEW_CHECKLIST.md +++ b/docs/PR_REVIEW_CHECKLIST.md @@ -24,8 +24,8 @@ As a PR reviewer and CODEOWNER, I have reviewed this and have: - [ ] Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy - [ ] If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers - [ ] If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet. -- [ ] Verified that the single-node recipes are similar to the official [vLLM recipes](https://recipes.vllm.ai/) and/or the[SGLang cookbook](https://docs.sglang.io/cookbook/intro): - - [ ] If they are not, I have verified that a PR has been opened in [vLLM recipe repo](https://github.com/vllm-project/recipes) or [SGLang repo](https://github.com/sgl-project/sglang/tree/main/docs_new) and linked it below in the additional detail section: +- [ ] Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official [vLLM recipes](https://recipes.vllm.ai/) and/or the [SGLang cookbook](https://docs.sglang.io/cookbook/intro): + - [ ] I linked the corresponding upstream PR in the [vLLM recipe repo](https://github.com/vllm-project/recipes) or [SGLang repo](https://github.com/sgl-project/sglang/tree/main/docs_new) and verified that it is **MERGED** before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below. - [ ] Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at [docs/waiver/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/docs/waiver)`.md` — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - [ ] If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below. diff --git a/perf-changelog.yaml b/perf-changelog.yaml index d77c35f1f..fa0679523 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4781,3 +4781,13 @@ - "Bump image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260708" - "Clean the export envs" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2198 + +- config-keys: + - dsv4-fp4-b200-vllm-agentic + - dsv4-fp4-b300-vllm-agentic + description: + - "Update B200 and B300 AgentX: KV offload, sparse DSV4 attention, mega-MoE, and FULL_DECODE_ONLY CUDA graphs." + - "Image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec" + - "B200: GPU-resident TP8 at conc [1,2,4,6,8] and SimpleCPU-offload TP8 at conc [8,12,16]; DEP8 at conc [8,16,24,32,40,48,56,64,68,72] with both SimpleCPU and Mooncake 0.3.11.post1." + - "B300: GPU-resident TP4 at conc [1,2,4,6,8,16]; DEP4 at conc [8,16,24,32,40,48,56,64,72] with both SimpleCPU and Mooncake 0.3.11.post1." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2202