[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231
[LMCache] Add LMCache arm on tuned DSV4 FP4 B200 vLLM AgentX recipe #2231ApostaC wants to merge 2 commits into
Conversation
Combine the tuned B200 recipe from PR #2224/#2225 (nightly image, sparse DSV4 FlashInfer attention, FULL_DECODE_ONLY CUDA graphs, AMXF4 mega-MoE with the LMCache 0.5.1 KV-offload backend from PR #2153. The lmcache arm drops --enable-cumem-allocator (cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache MP server) and runs otherwise identical serving flags, so backends are directly comparable. Adds a standalone dsv4-fp4-b200-vllm-agentic-lmcache config section mirroring the vllm-simple concurrency ladder and a perf-changelog entry triggering it. 中文：将 PR #2224/#2225 的 B200 调优配方（nightly 镜像、稀疏 DSV4 FlashInfer 注意力、FULL_DECODE_ONLY CUDA graph、AMXF4 mega-MoE）与 PR #2153 的 LMCache 0.5.1 KV 卸载后端合并。lmcache 分支仅去掉 --enable-cumem-allocator（cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache MP server），其余 serving 参数与其他分支保持一致，便于卸载后端间直接对比。 新增独立的 dsv4-fp4-b200-vllm-agentic-lmcache 配置（测试点与 vllm-simple 阶梯对齐）及触发它的 perf-changelog 条目。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> EOF )
ApostaC
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将 perf-changelog 条目的 pr-link 填写为 PR #2231。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ApostaC
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| --trust-remote-code | ||
| --kv-cache-dtype fp8 | ||
| --block-size 256 | ||
| "${PARALLEL_ARGS[@]}" | ||
| "${VLLM_CP_ARGS[@]}" | ||
| "${EP_ARGS[@]}" | ||
| --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' | ||
| --attention_config.use_fp4_indexer_cache=True | ||
| --max-model-len 1048576 | ||
| --gpu-memory-utilization 0.92 | ||
| --numa-bind | ||
| "${CUMEM_ARGS[@]}" | ||
| --no-enable-flashinfer-autotune | ||
| --tokenizer-mode deepseek_v4 | ||
| --tool-call-parser deepseek_v4 | ||
| --enable-auto-tool-choice | ||
| --reasoning-parser deepseek_v4 | ||
| --enable-prefix-caching | ||
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"use_fp4_indexer_cache":true}' | ||
| --no-disable-hybrid-kv-cache-manager | ||
| --disable-uvicorn-access-log | ||
| --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0}' | ||
| --max-num-seqs "$MAX_NUM_SEQS" | ||
| --max-cudagraph-capture-size "$MAX_NUM_SEQS" | ||
| "${PARALLEL_ARGS[@]}" | ||
| "${VLLM_CP_ARGS[@]}" | ||
| "${EP_ARGS[@]}" | ||
| "${FAST_MOE_ARGS[@]}" | ||
| "${OFFLOAD_ARGS[@]}" | ||
| ) | ||
| printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" |
There was a problem hiding this comment.
🔴 The rewritten VLLM_CMD in this file drops --tool-call-parser deepseek_v4 and --enable-auto-tool-choice, which were present in the pre-PR version of this exact script and are set in every other agentic vLLM recipe (dsv4_fp4_b300_vllm.sh, dsv4_fp4_mi355x_vllm.sh, dsv4_fp8_h200.sh, minimax/kimi variants). This looks like an accidental drop during the #2224 recipe rewrite rather than an intentional change, and since AgentX trace replay relies on tool_choice=auto, vLLM will reject those requests without --enable-auto-tool-choice, likely failing the sweep across all offload arms (none/vllm-simple/mooncake/lmcache).
Extended reasoning...
What changed: In the VLLM_CMD array construction (around lines 328-350 of benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh), the diff removes --tool-call-parser deepseek_v4 and --enable-auto-tool-choice while keeping the adjacent --reasoning-parser deepseek_v4 flag. The pre-PR version of this exact file had both flags (visible as removed lines in the diff), so this is not a case of the recipe never having them — they were explicitly dropped during the VLLM_CMD rewrite that pulled in the #2224 tuned recipe.
Why this is inconsistent with the rest of the repo: Every other agentic-coding vLLM recipe sets both flags together: dsv4_fp4_b300_vllm.sh (lines 209-210), dsv4_fp4_mi355x_vllm.sh (lines 396/398), and dsv4_fp8_h200.sh (lines 63-64) — confirmed directly by grepping those files. The minimax and kimi vLLM agentic recipes follow the same pattern. The PR description states the b200 script 'adopts the #2224 tuned recipe verbatim,' but #2224's own b300 sibling keeps both flags, so this omission is specific to the b200 rewrite and not an intentional nightly-vLLM behavior change.
Why nothing else in the script prevents this from mattering: The workload driven by build_replay_cmd/run_agentic_replay_and_write_outputs is the inferencex-agentx-mvp coding-agent trace replay, which is fundamentally tool-calling — requests are reconstructed with a tools schema and tool_choice. vLLM does not enable auto tool-choice/parsing by default; when a request specifies tool_choice=auto and the server wasn't started with --enable-auto-tool-choice (paired with a --tool-call-parser), vLLM returns an HTTP 400 ('auto tool choice requires --enable-auto-tool-choice and --tool-call-parser'). This is a request-level rejection, independent of whether the replay's default pre-canned mode discards live assistant responses — the request still needs to succeed for aiperf to count it as a valid data point.
Step-by-step proof of impact:
- The lmcache-arm (and every other arm: none/vllm-simple/mooncake) all share this single
VLLM_CMD— none of the offload-backendcasebranches add tool-call flags. - vLLM starts successfully (these flags don't affect startup), so
wait_for_server_readypasses and the sweep proceeds to the replay phase. build_replay_cmdissues chat-completion requests for the AgentX coding-agent trace with atoolsarray andtool_choice=auto(required to reproduce realistic input-token counts for tool-augmented turns).- Because
--enable-auto-tool-choice/--tool-call-parserare absent, vLLM rejects essentially all such requests with a 400. - aiperf's
--failed-request-threshold(10%, perAIPERF_FAILED_REQUEST_THRESHOLD) is exceeded almost immediately, failing the run for all 14 lmcache sweep points plus the parent vllm-simple/mooncake points that share the same script.
Fix: Re-add --tool-call-parser deepseek_v4 and --enable-auto-tool-choice to the shared VLLM_CMD array (near --reasoning-parser deepseek_v4), matching the pre-PR file and every sibling recipe.
| unset VLLM_USE_SIMPLE_KV_OFFLOAD | ||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT,\"lmcache.mp.mq_timeout\":$LMCACHE_MQ_TIMEOUT}}" | ||
| ) |
There was a problem hiding this comment.
🔴 The new lmcache arm's --kv-transfer-config sets kv_connector":"LMCacheMPConnector" but omits kv_connector_module_path, which both sibling scripts wiring the same connector (kimik2.5_fp4_b200.sh:150, dsv4_fp4_mi355x_vllm.sh:344) include. LMCacheMPConnector is an out-of-tree class that vLLM can only resolve via kv_connector_module_path + kv_connector; without it, vllm serve will fail to construct the KV connector at startup, so every one of the 14 lmcache sweep points will crash before producing a result. Fix by adding "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector" to the JSON at line 288.
Extended reasoning...
The bug: The new lmcache case in dsv4_fp4_b200_vllm.sh (lines 285-289) builds OFFLOAD_ARGS as:
"{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{...}}"
This is missing the kv_connector_module_path key. LMCacheMPConnector is shipped by the lmcache pip package (installed a few lines earlier via agentic_pip_install ... lmcache==$LMCACHE_VERSION), not by vLLM itself. vLLM'''s built-in KVConnectorFactory only knows connector classes it ships in-tree by bare name; to resolve an out-of-tree connector class, it needs to dynamically import the module named by kv_connector_module_path and then look up kv_connector as a class name inside that module. Without the module path, vLLM falls back to its built-in name registry, which has no entry for LMCacheMPConnector, and connector construction fails at vllm serve startup.
Why the existing sanity check doesn'''t catch it: The script runs python3 -c \"import lmcache.integration.vllm.lmcache_mp_connector\" >/dev/null a few lines before launching the server. That only proves the module is importable in a throwaway Python process — it does not register the connector with vLLM'''s factory in the actual vllm serve process, so it provides no protection against this specific misconfiguration.
Cross-script evidence: Both other scripts in this repo that wire up the identical LMCacheMPConnector include the field explicitly and consistently:
benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:150:\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",...benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh:344: same field, same value, otherwise near byte-for-byte the same JSON shape (samelmcache.mp.host/lmcache.mp.portextra_config keys).
The new B200 arm is the only place in the codebase wiring LMCacheMPConnector without this field, strongly suggesting it was simply dropped rather than being some new supported invocation.
Impact: This is not a subtle perf regression — it'''s a hard startup crash. Every one of the 14 lmcache sweep points defined in configs/nvidia-master.yaml (dsv4-fp4-b200-vllm-agentic-lmcache: 3 TP8 points + 11 TP8-DEP8 points) launches vllm serve with this malformed --kv-transfer-config, so the server process would fail while constructing the KV connector, before any benchmark traffic is served. This defeats the entire purpose of the PR, which is to add this lmcache arm for comparison against vllm-simple and Mooncake.
Step-by-step proof:
KV_OFFLOAD_BACKEND=lmcacheis set for a sweep point; the script enters thelmcache)case at line ~230.- The LMCache MP server starts and becomes healthy (this part is unaffected — it'''s a separate process from vLLM).
OFFLOAD_ARGSis built at line 285-289 as--kv-transfer-config '\''{\"kv_connector\":\"LMCacheMPConnector\",\"kv_role\":\"kv_both\",...}'\''— nokv_connector_module_path.vllm serveis launched with this flag (line ~372, via\"${OFFLOAD_ARGS[@]}\").- vLLM parses
--kv-transfer-config, seeskv_connector=\"LMCacheMPConnector\"with no module path, and attempts to resolve it against its built-in registry (which contains e.g.LMCacheConnectorV1but notLMCacheMPConnector). - Resolution fails, and
vllm serveexits/crashes during connector construction, before the health-check endpoint ever comes up —wait_for_server_readyat the bottom of the script will time out and the whole benchmark point fails.
Why this wasn'''t caught: The PR description explicitly says validation was only bash -n (syntax check) and generate_sweep_configs.py full-sweep (config generation), neither of which launches an actual vLLM server, so this startup failure was never exercised.
Fix: add \"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\" to the JSON at line 288, matching the sibling scripts exactly.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29460892084 |
1 participant
Description
Combines the tuned DeepSeek-V4 FP4 B200 vLLM AgentX recipe from #2224 / #2225 with the LMCache KV-offload backend from #2153, so the LMCache arm runs on the same image and serving flags as the vllm-simple and Mooncake arms and the backends are directly comparable.
benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.shadopts the [AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224 tuned recipe verbatim (imagevllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec,FLASHINFER_MLA_SPARSE_DSV4attention,FULL_DECODE_ONLYCUDA graphs, AMXF4 mega-MoE on EP tiers,--numa-bind) and adds anlmcacheoffload-backend case: LMCache 0.5.1 MP server (lmcache_driventransfer mode) +LMCacheMPConnector, with the L1 pool derated to 75% ofTOTAL_CPU_DRAM_GB(per [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups #2153 bring-up).--enable-cumem-allocatoris dropped, becauseLMCacheMPConnectorexports the KV cache to the LMCache server through legacy CUDA IPC handles and cuMem/VMM allocations cannot be exported that way (register_kv_cachesfails withcudaErrorInvalidValue).configs/nvidia-master.yamlupdates the parentdsv4-fp4-b200-vllm-agenticto the [AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224 image/search space (required — the rewritten shared script's flags don't exist onv0.23.0) and adds a standalonedsv4-fp4-b200-vllm-agentic-lmcachesection whose points mirror the vllm-simple ladder: TP8 conc [8, 12, 16]; TP8-DEP8 conc [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80].perf-changelog.yamlentry triggers onlydsv4-fp4-b200-vllm-agentic-lmcache; the parent points are benchmarked by [AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224 itself.Validated locally:
bash -npasses;generate_sweep_configs.py full-sweepemits all 14 lmcache points (router attached on DEP points,total-cpu-dram-gbresolved fromdram-utilization: 0.80); the changelog entry passes the Pydantic schema. Not yet run on hardware — needs a sweep run.中文说明
将 #2224 / #2225 中调优后的 DeepSeek-V4 FP4 B200 vLLM AgentX 配方与 #2153 的 LMCache KV 卸载后端合并，使 LMCache 分支与 vllm-simple、Mooncake 分支在相同镜像和相同 serving 参数下运行，便于卸载后端间直接对比。
vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec、FLASHINFER_MLA_SPARSE_DSV4注意力、FULL_DECODE_ONLYCUDA graph、EP 层级的 AMXF4 mega-MoE、--numa-bind），并新增lmcache卸载后端分支：LMCache 0.5.1 MP server（lmcache_driven传输模式）+LMCacheMPConnector，L1 池按TOTAL_CPU_DRAM_GB的 75% 降额（沿用 [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups #2153 调试结论）。--enable-cumem-allocator：LMCacheMPConnector通过传统 CUDA IPC 句柄将 KV cache 导出给 LMCache server，而 cuMem/VMM 分配无法以这种方式导出（register_kv_caches报cudaErrorInvalidValue）。dsv4-fp4-b200-vllm-agentic更新为 [AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224 的镜像与搜索空间（必要改动——重写后的共享脚本参数在v0.23.0镜像上不存在），并新增独立的dsv4-fp4-b200-vllm-agentic-lmcache配置，测试点与 vllm-simple 阶梯对齐：TP8 并发 [8, 12, 16]；TP8-DEP8 并发 [8, 16, 24, 32, 40, 48, 56, 64, 68, 72, 80]。perf-changelog.yaml条目仅触发dsv4-fp4-b200-vllm-agentic-lmcache；父配置的测试点由 [AgentX] vLLM DeepSeek-V4 B200 aggregate / vLLM DeepSeek-V4 B200 聚合配置 #2224 自身进行基准测试。本地验证：
bash -n通过；generate_sweep_configs.py full-sweep生成全部 14 个 lmcache 测试点（DEP 点带 router，total-cpu-dram-gb由dram-utilization: 0.80解析）；changelog 条目通过 Pydantic schema 校验。尚未在硬件上运行——需要触发扫描。Related Issue
Builds on #2224 / #2225 (tuned B200/B300 AgentX recipes) and #2153 (LMCache backend). / 基于 #2224 / #2225（B200/B300 AgentX 调优配方）与 #2153（LMCache 后端）。
Type of Change
Checklist
perf-changelog.yamlperf-changelog.yamlentries are appended to the end of the file (the file is chronological: oldest at top, newest at bottom)OWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR — do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps (remove and re-add the label to force one)🤖 Generated with Claude Code