Skip to content

[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ApostaC wants to merge 2 commits into
base: main
Choose a base branch
from
+204 −41
Open

[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b25887
feat(agentic): add LMCache arm to tuned DSV4 FP4 B200 vLLM AgentX recipe
ApostaC Jul 16, 2026
cd7b86f
chore: fill in perf-changelog pr-link with PR #2231
ApostaC Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 168 additions & 32 deletions benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
set -euo pipefail
set -eo pipefail
set -x

# Agentic trace replay benchmark for DeepSeek-V4-Pro FP4 on B200 using vLLM.
Expand All @@ -14,21 +14,26 @@ set -x
# Highest aggregate throughput at large CONC.
#
# Image is configured in nvidia-master.yaml. block_size=256,
# kv-cache-dtype=fp8, FP4 indexer cache enabled, FULL_AND_PIECEWISE cudagraph
# capture with custom_ops=all (per the vLLM blog recipe at
# https://vllm.ai/blog/deepseek-v4).
# kv-cache-dtype=fp8, FLASHINFER_MLA_SPARSE_DSV4 attention with the FP4 indexer
# cache, FULL_DECODE_ONLY cudagraph capture, and (in EP tiers) mega-MoE backend.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR
#
# KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=mooncake.
# Pure TP is GPU-resident (KV_OFFLOADING=none). DEP tiers offload KV to host
# DRAM: KV_OFFLOADING=dram requires KV_OFFLOAD_BACKEND=vllm-simple, mooncake,
# or lmcache.

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION

DCP_SIZE="${DCP_SIZE:-1}"
PCP_SIZE="${PCP_SIZE:-1}"
if [ -z "$DCP_SIZE" ]; then
DCP_SIZE=1
fi
if [ -z "$PCP_SIZE" ]; then
PCP_SIZE=1
fi
VLLM_CP_ARGS=()
if [ "$DCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--decode-context-parallel-size "$DCP_SIZE")
Expand All @@ -37,21 +42,21 @@ if [ "$PCP_SIZE" -gt 1 ]; then
VLLM_CP_ARGS+=(--prefill-context-parallel-size "$PCP_SIZE")
fi

GPU_COUNT="${GPU_COUNT:-$((TP * PCP_SIZE))}"
GPU_COUNT=$TP
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
exit 1
fi
export GPU_COUNT

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE
# Either way, MODEL_PATH is what the server is launched with.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ -n "$MODEL_PATH" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
Expand All @@ -65,12 +70,6 @@ nvidia-smi
resolve_trace_source
install_agentic_deps

# vLLM v0.22.1 can ship CUTLASS DSL 4.5.2 with stale native MLIR bindings,
# which fails DSV4 indexer compilation with mlir_global_dtors(..., data).
# Reinstall the matching native wheel until NVIDIA/cutlass#3259 is resolved.
agentic_pip_install --quiet --force-reinstall --no-deps \
'nvidia-cutlass-dsl-libs-cu13==4.5.2'

# vllm-project/router expands the one HTTP backend into one logical worker per
# DP rank and sends X-data-parallel-rank on forwarded requests. aiperf's
# X-Correlation-ID is stable for every turn of a conversation; alias it to the
Expand All @@ -94,20 +93,55 @@ export VLLM_ENGINE_READY_TIMEOUT_S=3600
# vllm-project/vllm#44774 applies the same reachability policy to Mooncake's
# store mask. 32k matches the trace-replay tuning validated for this workload.
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_USE_RUST_FRONTEND=1
export VLLM_DSV4_MEGA_FP8_COMBINE=1
export VLLM_RPC_TIMEOUT=600000

# ---- Server config ----------------------------------------------------------
SERVER_LOG="$RESULT_DIR/server.log"
ROUTER_LOG="$RESULT_DIR/router.log"
MOONCAKE_MASTER_LOG="$RESULT_DIR/mooncake_master.log"
LMCACHE_SERVER_LOG="$RESULT_DIR/lmcache_server.log"
mkdir -p "$RESULT_DIR"

SERVER_PID=""
ROUTER_PID=""
MOONCAKE_MASTER_PID=""
LMCACHE_SERVER_PID=""

OFFLOAD_ARGS=()

if require_agentic_kv_offload_backend mooncake; then
# vLLM's cuMem/VMM allocator is the recipe default; the lmcache arm clears
# this (see the lmcache case below).
CUMEM_ARGS=(--enable-cumem-allocator)
case "$KV_OFFLOAD_BACKEND" in
"")
require_agentic_kv_offload_none
;;
vllm-simple)
require_agentic_kv_offload_backend vllm-simple
CPU_BYTES_PER_RANK=$(( TOTAL_CPU_DRAM_GB * 1000 * 1000 * 1000 / GPU_COUNT ))
# Identical prefixes must hash to identical block keys across DP ranks.
export PYTHONHASHSEED=42
OFFLOAD_CONFIG=$(cat <<EOF
{
"kv_connector": "SimpleCPUOffloadConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"cpu_bytes_to_use_per_rank": ${CPU_BYTES_PER_RANK},
"lazy_offload": false,
"enable_cross_layers_blocks": "true"
}
}
EOF
)
OFFLOAD_ARGS=(
--kv-transfer-config
"$OFFLOAD_CONFIG"
)
;;
mooncake)
require_agentic_kv_offload_backend mooncake
# Embedded mode contributes one segment per GPU rank to a shared
# distributed store, so pre-divide the aggregate host-memory budget.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))
Expand All @@ -127,16 +161,14 @@ if require_agentic_kv_offload_backend mooncake; then
"global_segment_size": "${PER_RANK_GB}GB",
"local_buffer_size": "4GB",
"protocol": "rdma",
"device_name": "mlx5_0",
"device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_10,mlx5_11",
"enable_offload": false
}
EOF
export MOONCAKE_CONFIG_PATH
export MC_ENABLE_DEST_DEVICE_AFFINITY=1
# Identical prefixes must hash to identical store keys across DP ranks.
export PYTHONHASHSEED=0
# B200 GPU memory registration works through DMA-BUF, but the compute
# nodes do not expose nvidia_peermem. Force Mooncake's DMA-BUF
# GPUDirect RDMA path instead of its legacy ibv_reg_mr path.
export WITH_NVIDIA_PEERMEM=0
export MC_SLICE_SIZE=1048576
export MC_WORKERS_PER_CTX=4
Expand Down Expand Up @@ -167,16 +199,115 @@ EOF
--kv-transfer-config
'{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true}}'
)
fi
;;
lmcache)
require_agentic_kv_offload_backend lmcache
# The LMCache MP server owns the host-DRAM KV pool as one shared
# tier; vLLM ranks attach via LMCacheMPConnector, so the aggregate
# host budget is passed through undivided (unlike Mooncake's
# per-rank segments). Follows the LMCache DeepSeek-V4 recipe
# (docs.lmcache.ai/recipes/deepseek_v4_flash.html); LMCache handles
# DSV4's Sparse-MLA hybrid KV geometries automatically.
LMCACHE_VERSION=0.5.1
agentic_pip_install --quiet --no-cache-dir "lmcache==$LMCACHE_VERSION"
python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null

LMCACHE_HOST=127.0.0.1
LMCACHE_PORT=$((PORT + 12000))
LMCACHE_HTTP_PORT=$((PORT + 13000))
# LMCacheMPConnector concatenates lmcache.mp.host and port into the
# ZMQ endpoint. Bind the server to a raw host, but pass the connector
# a ZMQ-style host string.
LMCACHE_CONNECT_HOST="tcp://$LMCACHE_HOST"
# Pool target derated to 75% of the aggregate budget: pinned host
# memory is unswappable and also consumes GPU-side mapping
# resources, so leave headroom for vLLM host buffers and the OS.
# Full-budget targets OOM-killed the node (host OOM-killer or
# cudaErrorMemoryAllocation) as the cache filled past ~2 TB during
# PR #2153 bring-up.
LMCACHE_L1_SIZE_GB=$((TOTAL_CPU_DRAM_GB * 3 / 4))
# The pool grows lazily from the initial allocation, so the full
# --l1-size-gb target is not pinned at startup.
LMCACHE_L1_INIT_SIZE_GB=20
LMCACHE_MQ_TIMEOUT=300
# Identical prefixes must hash to identical cache keys across DP ranks.
export PYTHONHASHSEED=0
# LMCacheMPConnector exports the KV cache to the LMCache server
# through legacy CUDA IPC handles, and cuMem/VMM allocations cannot
# be exported that way (register_kv_caches fails with
# cudaErrorInvalidValue), so drop the allocator for this arm only.
CUMEM_ARGS=()
# Per-engine scheduler stats every 5s, to diagnose per-DP-rank KV
# cache imbalance under the session-sticky router.
export VLLM_LOG_STATS_INTERVAL=5

echo "Starting LMCache MP server on port $LMCACHE_PORT..."
# One GPU-side transfer worker avoids concurrent-GPU-transfer stalls
# under heavy async-load pressure; CPU-side workers stay at 8.
lmcache server \
--host "$LMCACHE_HOST" \
--port "$LMCACHE_PORT" \
--http-host "$LMCACHE_HOST" \
--http-port "$LMCACHE_HTTP_PORT" \
--l1-size-gb "$LMCACHE_L1_SIZE_GB" \
--l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" \
--max-gpu-workers 1 \
--max-cpu-workers 8 \
--chunk-size 1024 \
--l1-align-bytes 16384 \
--eviction-trigger-watermark 0.85 \
--eviction-ratio 0.10 \
--eviction-policy LRU \
--supported-transfer-mode lmcache_driven \
--no-separate-object-groups \
> "$LMCACHE_SERVER_LOG" 2>&1 &
LMCACHE_SERVER_PID=$!
LMCACHE_READY=0
for _ in $(seq 1 60); do
if ! kill -0 "$LMCACHE_SERVER_PID" 2>/dev/null; then
echo "LMCache server died during startup." >&2
cat "$LMCACHE_SERVER_LOG" >&2
exit 1
fi
if curl --output /dev/null --silent --fail \
"http://127.0.0.1:$LMCACHE_HTTP_PORT/healthcheck"; then
LMCACHE_READY=1
break
fi
sleep 2
done
if [ "$LMCACHE_READY" -ne 1 ]; then
echo "LMCache server did not become healthy in time." >&2
cat "$LMCACHE_SERVER_LOG" >&2
exit 1
fi

unset VLLM_USE_SIMPLE_KV_OFFLOAD
OFFLOAD_ARGS=(
--kv-transfer-config
"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}"
)
Comment on lines +285 to +289

@claude claude Bot Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new lmcache arm's --kv-transfer-config sets kv_connector":"LMCacheMPConnector" but omits kv_connector_module_path, which both sibling scripts wiring the same connector (kimik2.5_fp4_b200.sh:150, dsv4_fp4_mi355x_vllm.sh:344) include. LMCacheMPConnector is an out-of-tree class that vLLM can only resolve via kv_connector_module_path + kv_connector; without it, vllm serve will fail to construct the KV connector at startup, so every one of the 14 lmcache sweep points will crash before producing a result. Fix by adding "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector" to the JSON at line 288.

Extended reasoning...

The bug: The new lmcache case in dsv4_fp4_b200_vllm.sh (lines 285-289) builds OFFLOAD_ARGS as:

"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}"

This is missing the kv_connector_module_path key. LMCacheMPConnector is shipped by the lmcache pip package (installed a few lines earlier via agentic_pip_install ... lmcache==$LMCACHE_VERSION), not by vLLM itself. vLLM'''s built-in KVConnectorFactory only knows connector classes it ships in-tree by bare name; to resolve an out-of-tree connector class, it needs to dynamically import the module named by kv_connector_module_path and then look up kv_connector as a class name inside that module. Without the module path, vLLM falls back to its built-in name registry, which has no entry for LMCacheMPConnector, and connector construction fails at vllm serve startup.

Why the existing sanity check doesn'''t catch it: The script runs python3 -c \"import lmcache.integration.vllm.lmcache_mp_connector\" >/dev/null a few lines before launching the server. That only proves the module is importable in a throwaway Python process — it does not register the connector with vLLM'''s factory in the actual vllm serve process, so it provides no protection against this specific misconfiguration.

Cross-script evidence: Both other scripts in this repo that wire up the identical LMCacheMPConnector include the field explicitly and consistently:

  • benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:150: \"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",...
  • benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh:344: same field, same value, otherwise near byte-for-byte the same JSON shape (same lmcache.mp.host/lmcache.mp.port extra_config keys).

The new B200 arm is the only place in the codebase wiring LMCacheMPConnector without this field, strongly suggesting it was simply dropped rather than being some new supported invocation.

Impact: This is not a subtle perf regression — it'''s a hard startup crash. Every one of the 14 lmcache sweep points defined in configs/nvidia-master.yaml (dsv4-fp4-b200-vllm-agentic-lmcache: 3 TP8 points + 11 TP8-DEP8 points) launches vllm serve with this malformed --kv-transfer-config, so the server process would fail while constructing the KV connector, before any benchmark traffic is served. This defeats the entire purpose of the PR, which is to add this lmcache arm for comparison against vllm-simple and Mooncake.

Step-by-step proof:

  1. KV_OFFLOAD_BACKEND=lmcache is set for a sweep point; the script enters the lmcache) case at line ~230.
  2. The LMCache MP server starts and becomes healthy (this part is unaffected — it'''s a separate process from vLLM).
  3. OFFLOAD_ARGS is built at line 285-289 as --kv-transfer-config '\''{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",...}'\'' — no kv_connector_module_path.
  4. vllm serve is launched with this flag (line ~372, via \"${OFFLOAD_ARGS[@]}\").
  5. vLLM parses --kv-transfer-config, sees kv_connector=\"LMCacheMPConnector\" with no module path, and attempts to resolve it against its built-in registry (which contains e.g. LMCacheConnectorV1 but not LMCacheMPConnector).
  6. Resolution fails, and vllm serve exits/crashes during connector construction, before the health-check endpoint ever comes up — wait_for_server_ready at the bottom of the script will time out and the whole benchmark point fails.

Why this wasn'''t caught: The PR description explicitly says validation was only bash -n (syntax check) and generate_sweep_configs.py full-sweep (config generation), neither of which launches an actual vLLM server, so this startup failure was never exercised.

Fix: add \"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\" to the JSON at line 288, matching the sibling scripts exactly.

;;
*)
echo "Error: unsupported B200 KV_OFFLOAD_BACKEND='$KV_OFFLOAD_BACKEND'" >&2
exit 1
;;
esac

PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
fi

EP_ARGS=()
FAST_MOE_ARGS=()
if [ "$EP_SIZE" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
FAST_MOE_ARGS=(
--moe-backend deep_gemm_amxf4_mega_moe
--enable-ep-weight-filter
--prefill-schedule-interval 16
)
fi

# AgentX concurrency counts live session trees, not individual requests.
Expand All @@ -197,18 +328,23 @@ VLLM_CMD=(
--trust-remote-code
--kv-cache-dtype fp8
--block-size 256
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'
--attention_config.use_fp4_indexer_cache=True
--max-model-len 1048576
--gpu-memory-utilization 0.92
--numa-bind
"${CUMEM_ARGS[@]}"
--no-enable-flashinfer-autotune
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--enable-auto-tool-choice
--reasoning-parser deepseek_v4
--enable-prefix-caching
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}'
--no-disable-hybrid-kv-cache-manager
--disable-uvicorn-access-log
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}'
--max-num-seqs "$MAX_NUM_SEQS"
--max-cudagraph-capture-size "$MAX_NUM_SEQS"
"${PARALLEL_ARGS[@]}"
"${VLLM_CP_ARGS[@]}"
"${EP_ARGS[@]}"
"${FAST_MOE_ARGS[@]}"
"${OFFLOAD_ARGS[@]}"
)
printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt"
Comment on lines 328 to 350

@claude claude Bot Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The rewritten VLLM_CMD in this file drops --tool-call-parser deepseek_v4 and --enable-auto-tool-choice, which were present in the pre-PR version of this exact script and are set in every other agentic vLLM recipe (dsv4_fp4_b300_vllm.sh, dsv4_fp4_mi355x_vllm.sh, dsv4_fp8_h200.sh, minimax/kimi variants). This looks like an accidental drop during the #2224 recipe rewrite rather than an intentional change, and since AgentX trace replay relies on tool_choice=auto, vLLM will reject those requests without --enable-auto-tool-choice, likely failing the sweep across all offload arms (none/vllm-simple/mooncake/lmcache).

Extended reasoning...

What changed: In the VLLM_CMD array construction (around lines 328-350 of benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh), the diff removes --tool-call-parser deepseek_v4 and --enable-auto-tool-choice while keeping the adjacent --reasoning-parser deepseek_v4 flag. The pre-PR version of this exact file had both flags (visible as removed lines in the diff), so this is not a case of the recipe never having them — they were explicitly dropped during the VLLM_CMD rewrite that pulled in the #2224 tuned recipe.

Why this is inconsistent with the rest of the repo: Every other agentic-coding vLLM recipe sets both flags together: dsv4_fp4_b300_vllm.sh (lines 209-210), dsv4_fp4_mi355x_vllm.sh (lines 396/398), and dsv4_fp8_h200.sh (lines 63-64) — confirmed directly by grepping those files. The minimax and kimi vLLM agentic recipes follow the same pattern. The PR description states the b200 script 'adopts the #2224 tuned recipe verbatim,' but #2224's own b300 sibling keeps both flags, so this omission is specific to the b200 rewrite and not an intentional nightly-vLLM behavior change.

Why nothing else in the script prevents this from mattering: The workload driven by build_replay_cmd/run_agentic_replay_and_write_outputs is the inferencex-agentx-mvp coding-agent trace replay, which is fundamentally tool-calling — requests are reconstructed with a tools schema and tool_choice. vLLM does not enable auto tool-choice/parsing by default; when a request specifies tool_choice=auto and the server wasn't started with --enable-auto-tool-choice (paired with a --tool-call-parser), vLLM returns an HTTP 400 ('auto tool choice requires --enable-auto-tool-choice and --tool-call-parser'). This is a request-level rejection, independent of whether the replay's default pre-canned mode discards live assistant responses — the request still needs to succeed for aiperf to count it as a valid data point.

Step-by-step proof of impact:

  1. The lmcache-arm (and every other arm: none/vllm-simple/mooncake) all share this single VLLM_CMD — none of the offload-backend case branches add tool-call flags.
  2. vLLM starts successfully (these flags don't affect startup), so wait_for_server_ready passes and the sweep proceeds to the replay phase.
  3. build_replay_cmd issues chat-completion requests for the AgentX coding-agent trace with a tools array and tool_choice=auto (required to reproduce realistic input-token counts for tool-augmented turns).
  4. Because --enable-auto-tool-choice/--tool-call-parser are absent, vLLM rejects essentially all such requests with a 400.
  5. aiperf's --failed-request-threshold (10%, per AIPERF_FAILED_REQUEST_THRESHOLD) is exceeded almost immediately, failing the run for all 14 lmcache sweep points plus the parent vllm-simple/mooncake points that share the same script.

Fix: Re-add --tool-call-parser deepseek_v4 and --enable-auto-tool-choice to the shared VLLM_CMD array (near --reasoning-parser deepseek_v4), matching the pre-PR file and every sibling recipe.

Expand Down
35 changes: 26 additions & 9 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1771,7 +1771,30 @@ dsv4-fp4-b200-vllm:
- { tp: 8, ep: 8, dp-attn: true, conc-start: 64, conc-end: 1024 }

dsv4-fp4-b200-vllm-agentic:
image: vllm/vllm-openai:v0.23.0
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b200-dgxc
precision: fp4
framework: vllm
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Pure TP at low concurrency.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 6, 8] }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 12, 16] }
# DEP
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: vllm-simple, version: "904e4ec" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [12, 20, 28, 36, 44, 52, 60, 68, 76], router: { name: vllm-router, version: "0.1.14" } }

# LMCache CPU-offload arm of dsv4-fp4-b200-vllm-agentic, split into its own
# config so it can be triggered/tested independently of the vllm-simple and
# Mooncake curves. Points mirror the vllm-simple offload ladder above on the
# same tuned image, so the backends are directly comparable.
dsv4-fp4-b200-vllm-agentic-lmcache:
image: vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:b200-dgxc
Expand All @@ -1782,14 +1805,8 @@ dsv4-fp4-b200-vllm-agentic:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Pure TP is only competitive at very low concurrency.
- { tp: 8, kv-offloading: none, conc-list: [1, 2, 3, 4, 5] }
# Sample the useful MooncakeStore range without repeating its collapsed
# high-concurrency tail.
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [8, 10, 16] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 38, 44, 50], router: { name: vllm-router, version: "0.1.14" } }
# Retain the external-cache transition and peak-throughput region.
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: mooncake, version: "0.3.11.post1" }, conc-list: [16, 38, 44, 56, 64, 66, 68], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8, 12, 16] }
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "0.5.1" }, conc-list: [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80], router: { name: vllm-router, version: "0.1.14" } }

dsv4-fp4-b200-trt:
image: ghcr.io#semianalysisai/trtllm-deepseek-v4:feat-deepseek_v4-c185066
Expand Down
10 changes: 10 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4815,3 +4815,13 @@
- "Image: lmsysorg/sglang:v0.5.13.post1-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2123

- config-keys:
- dsv4-fp4-b200-vllm-agentic-lmcache
scenario-type:
- agentic-coding
description:
- "Add LMCache 0.5.1 DRAM KV-offload arm on the tuned B200 AgentX recipe from PR #2224/#2225 (sparse DSV4 FlashInfer attention, AMXF4 mega-MoE, FULL_DECODE_ONLY CUDA graphs) with image vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec"
- "LMCache MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; L1 pool derated to 75% of TOTAL_CPU_DRAM_GB; --enable-cumem-allocator dropped on the lmcache arm only (cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server)"
- "Points mirror the vllm-simple ladder: TP8 conc [8, 12, 16]; TP8-DEP8 conc [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80]"
- "Parent dsv4-fp4-b200-vllm-agentic updated in the same change to the PR #2224 recipe and search space (image bump, vllm-simple + Mooncake arms) so the shared script stays consistent; the parent points are re-benchmarked by PR #2224 and not re-triggered here"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2231
Loading