[AMD][AgentX] Kimi-K2.7-Code FP4 MI355X agentX vLLM#2126
Conversation
Adds the kimik2.7 fp4 agentic trace-replay recipe (kimik2.7_fp4_mi355x.sh, KV_OFFLOADING none + dram/lmcache) and the kimik2.7-fp4-mi355x-vllm-agentic config key on cluster:mi355x-amds (tp4 conc32, none + lmcache). Recipe clones LMCache to /opt/lmcache-src so the CI checkout never trips over root-owned build artifacts.
The none case previously passed --no-enable-prefix-caching, which disables the on-GPU prefix cache entirely, not just DRAM offload. That crippled the no-offload baseline (no reuse even within HBM) and made lmcache look artificially good. Leave prefix caching at vLLM's default (ON) so none is an honest GPU-only-KV baseline, apples-to-apples vs lmcache. Matches the kimik2.5 / dsv4 agentic recipes.
Set the kimik2.7-fp4-mi355x-vllm-agentic sweep to conc [1,4,8,16,32,48] for both none and dram/LMCache at TP4, dram-utilization 0.80. Add the corresponding perf-changelog entry (pr-link placeholder to fill on PR open).
Previous commit accidentally dropped perf-changelog.yaml (blob too large for inline arg). Restore it with the kimik2.7-fp4-mi355x-vllm-agentic entry appended.
hyukjlee
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4" | ||
| - "Sweep KV-offloading none vs dram/LMCache at conc [1,4,8,16,32,42]; LMCache MP server + LMCacheMPConnector, dram-utilization 0.80" | ||
| - "Recipe benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh clones LMCache to container-local /opt/lmcache-src (pinned) so CI checkout never trips over root-owned build artifacts" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126 |
There was a problem hiding this comment.
🔴 The new changelog entry's pr-link is literally .../pull/PLACEHOLDER (perf-changelog.yaml:4644). utils/validate_perf_changelog.py only accepts the canonical link (.../pull/2126 for this PR) or the XXX sentinel — PLACEHOLDER is neither, so the changelog-gate CI step in .github/workflows/run-sweep.yml will raise ChangelogValidationError and fail this PR. Fix: replace PLACEHOLDER with 2126 (or XXX) before merge.
Extended reasoning...
The new changelog entry ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER (perf-changelog.yaml:4644). Every surrounding entry uses a real PR number (2113, 2114, 2115), so this is unambiguously a leftover placeholder that was never rewritten with the real PR id.
This is not just cosmetic. utils/validate_perf_changelog.py defines the only accepted placeholder set:
PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}and validate_added_pr_link (line 144) enforces link in PR_LINK_PLACEHOLDERS or link == expected where expected == f"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}". The literal string "PLACEHOLDER" (or the full .../pull/PLACEHOLDER URL) is in neither set, so the validator will raise ChangelogValidationError on this PR.
That validator is wired into CI via .github/workflows/run-sweep.yml:102 (python3 utils/validate_perf_changelog.py), so the changelog-gate step will actively fail this PR until the link is fixed. utils/prepare_perf_changelog_merge.py performs the same check on the reuse-merge path, so even if CI were bypassed, the merge-with-reuse tooling would refuse the entry.
Step-by-step proof:
- On PR-2126 CI, the changelog gate invokes
validate_perf_changelog.pywith--pr-number 2126. - It parses head
perf-changelog.yaml, finds the newly appendedkimik2.7-fp4-mi355x-vllm-agenticentry. - It calls
validate_added_pr_link("https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER", 2126). expected = "https://github.com/SemiAnalysisAI/InferenceX/pull/2126". The link is neitherexpectednor a member ofPR_LINK_PLACEHOLDERS(only"XXX"/".../pull/XXX").raise ChangelogValidationError(...)→ the CI job exits non-zero → the PR check goes red.- If it somehow merged, the on-site changelog would contain a permanently dead hyperlink pointing at
/pull/PLACEHOLDER, breaking the pr-link traceability contract every surrounding entry follows.
Fix: change the value at perf-changelog.yaml:4644 to https://github.com/SemiAnalysisAI/InferenceX/pull/2126 (or the accepted XXX placeholder if you'd rather punt until squash-merge time). One-token change.
| # Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs: | ||
| # KV_OFFLOADING=none -> GPU KV only | ||
| # KV_OFFLOADING=dram KV_OFFLOAD_BACKEND=lmcache -> LMCache MP server + connector |
There was a problem hiding this comment.
🟡 Header comment at line 7 reads "Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs" — but this file is kimik2.7_fp4_mi355x.sh. The intended reference is almost certainly kimik2.5_fp4_mi355x.sh (the sibling this recipe is derived from — same env-var contract, install flow, and vLLM args, only the LMCache branch is new). Zero runtime impact; one-line doc fix.
Extended reasoning...
What the bug is. The recipe's header docstring at benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh:7 reads:\n\n\n# Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs:\n\n\nThat filename is the same file the comment lives in — a script cannot be a variant of itself. This is a copy-paste error from the sibling recipe.\n\nWhich file it should reference. The obvious parent is benchmarks/single_node/agentic/kimik2.5_fp4_mi355x.sh: it shares the same env-var contract (MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE), the same benchmark_lib.sh sourcing, the same hf download / amd-quark install flow, and an almost-identical vLLM launch invocation. The only structural addition in the 2.7 script is the lmcache branch of the case "$OFFLOAD_MODE" dispatch — which is exactly what the "supports TWO KV configs" clarifier is describing. So the intended comment is "Variant of kimik2.5_fp4_mi355x.sh".\n\nStep-by-step proof. (1) The path in the diff header is benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh. (2) The comment at line 7 says "Variant of kimik2.7_fp4_mi355x.sh". (3) Those two strings are byte-identical, so the comment is a self-reference. (4) The sibling kimik2.5_fp4_mi355x.sh exists on main and only supports the none offload mode (its search-space includes kv-offload-backend: native but the recipe itself pre-dates LMCache), which matches the "supports TWO KV configs" clarifier exactly (2.7 adds the lmcache branch as a second KV config).\n\nImpact. None at runtime — this is a comment. It's a minor readability defect for future maintainers landing on this file who'd otherwise trace back to the kimik2.5 recipe to understand why the code is structured this way.\n\nSuggested fix. One-line change: Variant of kimik2.7_fp4_mi355x.sh → Variant of kimik2.5_fp4_mi355x.sh. Alternately, drop the "Variant of …" preamble entirely, since the two-KV-configs description stands on its own.\n\nSeverity. Nit. Pure documentation, no behavior change, not blocking merge — worth a one-line touch-up while the author is in the file.
| - config-keys: | ||
| - kimik2.7-fp4-mi355x-vllm-agentic | ||
| description: | ||
| - "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4" |
There was a problem hiding this comment.
🟡 The perf-changelog description references cluster:gbt350docker->mi355x-amds, but gbt350docker doesn't exist anywhere else in the repo — the actual runner declared in configs/amd-master.yaml is simply cluster:mi355x-amds. Looks like stale copy-paste; suggest dropping the parenthetical or rewriting it as (runner: cluster:mi355x-amds).
Extended reasoning...
What
Line 4641 of perf-changelog.yaml reads:
Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4
The token gbt350docker is not a known cluster/runner identifier in this repo. A grep across the tree shows it appears only on this new changelog line — nowhere in configs/runners.yaml, configs/amd-master.yaml, any launcher, or any other doc.
Why it's wrong
The actual runner declared for kimik2.7-fp4-mi355x-vllm-agentic in configs/amd-master.yaml is:
runner: cluster:mi355x-amdsNo gbt350docker migration ever happened for this recipe; the AMD MI355X agentic runner is cluster:mi355x-amds (same as kimik2.5-fp4-mi355x-vllm-agentic, dsv4-fp4-mi355x-vllm-agentic, and qwen3.5-fp8-mi355x-sglang-agentic-hicache in the same file). The cluster:X->Y arrow format also appears nowhere else in perf-changelog.yaml — it isn't a project convention. This reads like stale template text carried over from an unrelated draft.
Impact
Zero runtime impact — perf-changelog.yaml is documentation, description strings are free-form, and the launcher/runner routing is driven by configs/amd-master.yaml (which is correct). The affected artifact is the public perf-changelog page: this string will render there and misrepresent the sweep provenance to any reader trying to trace where the numbers came from.
Suggested fix
Either drop the parenthetical entirely:
- "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X via vLLM v0.24.0, TP4"or make it accurate:
- "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (runner: cluster:mi355x-amds) via vLLM v0.24.0, TP4"Proof (step-by-step)
configs/amd-master.yamlkimik2.7-fp4-mi355x-vllm-agentic:block declaresrunner: cluster:mi355x-amds.grep -rn gbt350docker .returns exactly one hit —perf-changelog.yaml:4641— and no other file in the repo defines or references that cluster name.- Every peer MI355X agentic recipe in the same file uses
runner: cluster:mi355x-amds; there is no fleet, cluster, or SKU by the namegbt350dockerinconfigs/. - Therefore the changelog's
cluster:gbt350docker->mi355x-amdsdescribes routing that doesn't exist — it is a copy-paste artifact.
seungrokj
changed the title
- Frontier-only sweep: none [1,4,8] (low conc, fits GPU HBM) + lmcache [16,32] (above the KV cliff). Drops dominated/cliff points per review. - Fix recipe header: 'Variant of kimik2.5_fp4_mi355x.sh' (was self-referential) - perf-changelog: correct runner (cluster:mi355x-amds) and conc lists
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29003473063 |
… config Align the command-builder output with the InferenceX MI355X agentic-coding sweep (SemiAnalysisAI/InferenceX#2126): the validated config is TP4, so set variant tp: 4 and add VLLM_ROCM_USE_AITER_RMSNORM=0 to the mi355x extra_env (AITER RMSNorm has accuracy issues below TP8). Generated command now renders TP4 with AITER=1 / QUICK_REDUCE=INT4 / RMSNORM=0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… config Align the command-builder output with the InferenceX MI355X agentic-coding sweep (SemiAnalysisAI/InferenceX#2126): the validated config is TP4, so set variant tp: 4 and add VLLM_ROCM_USE_AITER_RMSNORM=0 to the mi355x extra_env (AITER RMSNorm has accuracy issues below TP8). Generated command now renders TP4 with AITER=1 / QUICK_REDUCE=INT4 / RMSNORM=0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: hyukjlee <hyukjlee@amd.com>
|
vllm-project/recipes#630 vllm recipe PR created. @functionstackx @seungrokj |
hi @hyukjlee you probably want to base ur PR off of this inital kvcache offloading scaffolding PR vllm-project/recipes#628 |
… config Align the command-builder output with the InferenceX MI355X agentic-coding sweep (SemiAnalysisAI/InferenceX#2126): the validated config is TP4, so set variant tp: 4 and add VLLM_ROCM_USE_AITER_RMSNORM=0 to the mi355x extra_env (AITER RMSNorm has accuracy issues below TP8). Generated command now renders TP4 with AITER=1 / QUICK_REDUCE=INT4 / RMSNORM=0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: hyukjlee <hyukjlee@amd.com>
|
Done — rebased vllm-project/recipes#630 onto the #628 kv_offload scaffolding. It now uses the first-class LMCache KV Offload row (same integration pattern as DeepSeek-V4-Pro), with only the MI355X/ROCm-specific overrides kept in the guide (HIP build from source + ROCm server flags, since the generic row installs the CUDA |
… smoke At TP4 the conc16/32 cells were GPU-KV-capacity bound: the ~121 GiB KV pool held only ~12 resident ~90k-token contexts, so throughput plateaued (~160 tok/s) and requests preempted/thrashed from conc16 up (0 preemptions at c8, 24 at c16, 35 at c32). LMCache was healthy (82% external prefix-hit) so offload quality was not the limiter -- raw resident-KV room was. Move the agentic-coding sweep to TP8 (doubles the KV pool and per-token decode compute) at conc [16,32,48] for both KV-offloading none and dram/LMCache, as a 900s trace-replay smoke. Adds an optional per-scenario `duration` override to the agentic-coding config schema (defaults to 3600s when unset) so smoke sweeps can run shorter without changing the global default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310178967 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310838023 |
Add a third KV tier to the TP8 agentic comparison alongside none and LMCache: dram/Mooncake via the embedded MooncakeStoreConnector store (built from source for ROCm, tcp protocol), modeled on the MI355X DSv4 vLLM recipe. Wire it as an additive `mooncake)` case so the existing none/lmcache paths are unchanged. configs/amd-master.yaml now carries all three backends at conc [16,32,48]. configs/smoke-kimik2.7-mooncake.yaml is a temporary overlay to dispatch only the Mooncake cells, so the in-flight none/lmcache sweep is not rerun. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310838023 |
| - "TP4 was GPU-KV-capacity bound: the ~121 GiB pool held only ~12 resident ~90k-token contexts, so throughput plateaued (~160 tok/s) and requests preempted/thrashed from conc16 up. TP8 doubles the KV pool and per-token decode compute to break the wall" | ||
| - "Add a Mooncake KV-offload backend to the recipe (embedded MooncakeStoreConnector store, built from source for ROCm, tcp protocol), mirroring the MI355X DSv4 vLLM recipe" | ||
| - "Adds optional per-scenario 'duration' override to the agentic-coding config schema (defaults to 3600s when unset)" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29314798522 |
…32,48]; fix Mooncake worker import Reshape the TP8 agentic sweep for the real 3600s comparison run: - none (GPU-only) swept only at low conc [4,8] (it collapses under load) - LMCache and Mooncake carry the high-conc [16,32,48] range - duration 3600s (the prior 900s smoke was warmup-dominated for the ~90k-token agentic inputs, so per-request metrics were unreliable) Fix Mooncake so the vLLM worker subprocesses can import it: the previous bare `make install` only laid down C++ libs, so the launcher imported `mooncake` but the workers hit "Please install mooncake ..." at KV-cache init. Build the wheel via Mooncake's scripts/build_wheel.sh (auditwheel bundles every .so) and pip install it into site-packages, and build with -DWITH_STORE=ON so store.so (MooncakeStoreConnector) is present. Drop the temporary configs/smoke-kimik2.7-mooncake.yaml overlay; all three backends now run from one canonical config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29340310850 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29343553277 |
…weep - Set VLLM_ROCM_USE_AITER_MLA=0 explicitly (64 KV heads incompatible with the ROCm AITER-MLA kernel; was only working via implicit fallback). - Add kimik2.7-fp4-mi355x-vllm-agentic-tune config key: TP8+LMCache, EP1 vs EP8 at conc[16,32], 900s. Probes whether expert parallelism improves perf (current record sweep runs EP=1 on a large MoE). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sweep Wire DP_ATTENTION into the vLLM recipe, mirroring dsv4_fp4_b300_vllm.sh: - DP_ATTENTION=true -> TP1 x DP=$TP with --enable-expert-parallel (experts EP-sharded across DP ranks), fronted by vllm-router on $PORT (consistent_hash + X-session affinity from aiperf's X-Correlation-ID). DP engine binds $PORT+1. - DP_ATTENTION=false -> unchanged pure-TP path on $PORT. - Add DP_ATTENTION to check_env_vars; clean up ROUTER_PID on exit. Retarget the -tune config key from the (non-working) EP1-vs-EP8 probe to a scaling experiment: pure-TP8 vs DEP, each with LMCache/Mooncake, conc[16,32,48], 900s. Aims for a throughput curve that grows with concurrency instead of the GPU-KV-capacity collapse the pure-TP path hits at conc>=16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_MLA)
With VLLM_ROCM_USE_AITER_MLA=0 the only ROCm MLA backend is TRITON_MLA, which
rejects block_size=1 ("No valid attention backend found ... TRITON_MLA:
block_size not supported"). --block-size=1 only ever existed to force the
now-disabled AITER-MLA path. Drop it and let vLLM pick a TRITON_MLA-valid
default, matching AMD's reference serve command (AITER_MLA=0, no --block-size).
Fixes bringup failure for every cell (pure-TP and DEP alike).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…st isolation) The DEP Mooncake c16 cell fails fast at bringup; GH matrix fail-fast then cancels the still-running LMCache/pure-TP cells before they finish. Split the LMCache scaling experiment (-tune) from Mooncake (-tune-mc) so a Mooncake bringup failure can't poison the scaling run, and so Mooncake runs alone and uploads its own server_logs for diagnosis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
4 participants
MI355X using vLLM
benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.shwithtwo KV offload backends:
none(GPU-only KV, prefix caching on) andlmcache(LMCache MP server + LMCacheMPConnector)kimik2.7-fp4-mi355x-vllm-agentic, TP4, conc sweep`[1,4,8]' for none and '[16, 32] for lmcache, dram-utilization 0.80
vllm/vllm-openai-rocm:v0.24.0amd/Kimi-K2.7-Code-MXFP4