[AMD] MiniMax-M3 MXFP8 MI355X vLLM: gate AITER sparse PA to 8k1k high-conc + native linear default / MiniMax-M3 MXFP8 MI355X vLLM：AITER 稀疏分页注意力门控至 8k1k 高并发 + 默认原生 linear#2187
Conversation
…lation linear Bump the minimaxm3-fp8-mi355x-vllm image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287). Enable it via VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4, num_kv_heads == 1 per rank) plus the recipe's quick all-reduce knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256. Drop --linear-backend emulation now that the Triton MXFP8 dense-linear GEMM is optimized. Deliberately do NOT add the #47269 --hf-overrides use_index_cache/index_topk_freq indexer skip: it reduces model-architecture FLOPs, disallowed by docs/PR_REVIEW_CHECKLIST.md. Sparse PA is a kernel-level speedup only. Verified locally on MI355X (gfx950) TP4 8k1k: conc1 10/10 and conc512 5120/5120 completed with zero failures (conc512 ~28.6k tok/s total, ~7.1k tok/s/GPU). 中文：将 minimaxm3-fp8-mi355x-vllm 镜像升级到 nightly-9e57de7197f234f9d9187715d96e07e007048c0f，该镜像已包含合并进主分支的 AITER page-16 稀疏分页注意力(sparse paged attention, vllm-project/vllm#47287)。 通过 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1（配合 VLLM_ROCM_USE_AITER=1、fp8 KV cache 及 TP4，每个 rank num_kv_heads == 1）启用，并补上 recipe 的 quick all-reduce 调优开关 VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 与 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256。由于 Triton MXFP8 稠密 linear GEMM 已优化，移除 --linear-backend emulation。刻意不添加 #47269 的 --hf-overrides use_index_cache/index_topk_freq indexer 跳层开关：它会减少模型 架构 FLOPs，违反 docs/PR_REVIEW_CHECKLIST.md。sparse PA 仅为 kernel 级加速。 已在 MI355X(gfx950) TP4 8k1k 本地验证：conc1 10/10、conc512 5120/5120 全部成功 （conc512 约 28.6k tok/s 总吞吐，约 7.1k tok/s/GPU）。 Co-authored-by: Cursor <cursoragent@cursor.com>
… for 8k1k high-conc On the sparse-PA nightly the dense-linear backend crossover reversed vs #2003: the native Triton MXFP8 GEMM now wins in the memory-bound low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) still wins in the compute-bound high-concurrency regime. Measured on gfx950 MXFP8 + sparse PA: emulation ~+3-5% at 8k1k conc 64-512 but ~-3% at conc 1-8. Gate emulation to isl>=8192 && conc>=64; native everywhere else (all 1k1k, 8k1k conc<64). Overridable via LINEAR_BACKEND. 中文：在 sparse-PA nightly 上，稠密 linear 后端的性能拐点相比 #2003 发生反转： 原生 Triton MXFP8 GEMM 现在在访存受限的低并发区间更快，而 --linear-backend emulation（bf16 hipBLASLT）仍在计算受限的高并发区间更快。gfx950 MXFP8 + sparse PA 实测：emulation 在 8k1k conc 64-512 约 +3-5%，在 conc 1-8 约 -3%。因此将 emulation 限定在 isl>=8192 且 conc>=64，其余（所有 1k1k、8k1k conc<64）使用原生 路径。可通过 LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com>
…high-conc AITER page-16 sparse PA is a long-context/high-batch optimization: it wins at 8k1k conc>=64 but adds overhead at short context (1k1k) and low concurrency. Gate both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT (sparse PA) and --linear-backend emulation behind a single MM3_HIGH_CONC_FASTPATH condition (isl>=8192 && conc>=64); everything else falls back to the #2003 path (non-shuffled Triton attention + native linear) so there is no regression outside the target regime. Net vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere. Overridable via MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND. 中文：AITER page-16 稀疏分页注意力(sparse PA)是长上下文/大批量优化：在 8k1k conc>=64 有收益，但在短上下文(1k1k)和低并发下带来额外开销。将 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT（sparse PA）与 --linear-backend emulation 统一 用 MM3_HIGH_CONC_FASTPATH 条件（isl>=8192 且 conc>=64）门控；其余情况回退到 #2003 路径（非 shuffle 的 Triton 注意力 + 原生 linear），从而在目标区间之外不产生回退。 相对 #2003：8k1k tput/gpu conc128 +3.3%、conc256 +2.5%、conc512 +4.7%（峰值）， 其余基本持平。可通过 MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor <cursoragent@cursor.com>
hongxiayang
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将 perf-changelog 的 pr-link 设置为 #2187。 Co-authored-by: Cursor <cursoragent@cursor.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I also checked two other candidates and ruled them out: the perf-changelog pr-link field is correctly filled in (points to this PR, not a placeholder), and the MM3_HIGH_CONC_FASTPATH conc>=64 threshold's apparent regression at conc64 8k1k (-0.9%) is within the same run-to-run noise band as the PR's own flagged conc8 outlier, not a real gating-boundary bug.
Extended reasoning...
Checked the gating threshold (isl>=8192 && conc>=64) against the PR's own benchmark table for a boundary regression, and checked the perf-changelog pr-link field for an unfilled placeholder. Both were consistent with intended behavior/correctly filled, so no further findings beyond the inline nit.
… concurrencies Switch VLLM_ROCM_QUICK_REDUCE_QUANTIZATION from INT6 to INT4 unconditionally (#2003 used INT6); accuracy is guarded by the 8k1k evals. The AITER quick all-reduce tuning knobs (CAST_BF16_TO_FP16=0, QUANTIZATION_MIN_SIZE_KB=256) remain in the 8k1k high-conc fast path. Aimed at lifting the low/mid-conc points (e.g. the 8k1k conc8 dip) via cheaper all-reduces. 中文：将 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION 从 INT6 无条件改为 INT4（#2003 用的是 INT6），准确性由 8k1k evals 保障。AITER quick all-reduce 调优开关 (CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256) 仍保留在 8k1k 高并发快速路径中。 目的是通过更廉价的 all-reduce 提升低/中并发点（例如 8k1k conc8 的回退）。 Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Summary
Bumps
minimaxm3-fp8-mi355x-vllmtovllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f(carries the merged AITER page-16 sparse paged-attention path, vllm-project/vllm#47287) and re-tunes the serve command for that nightly:VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1, withVLLM_ROCM_USE_AITER=1+ fp8 KV cache on TP4 wherenum_kv_heads == 1per rank). Kernel-level speedup of the same sparse-attention computation — no FLOP reduction (we deliberately do not use the #47269--hf-overrides use_index_cache/index_topk_freqindexer-skip, perdocs/PR_REVIEW_CHECKLIST.md).MM3_HIGH_CONC_FASTPATH=isl>=8192 && conc>=64). Sparse PA is a long-context/high-batch optimization and regresses short context (1k1k) and low concurrency, so bothVLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUTand--linear-backend emulationare enabled only in that regime. Everything else falls back to the prior [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 path (non-shuffled Triton attention + native linear).--linear-backend emulationfrom [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003). On this nightly the native Triton MXFP8 GEMM is faster in the memory-bound low/mid-concurrency regime; emulation (bf16 hipBLASLT) only wins at 8k1k high concurrency and stays gated there.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4; [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 used INT6) — cheaper collectives lift the whole curve and eliminate the low/mid-conc dips; accuracy is guarded by the 8k1k evals. The AITER tuning knobs (CAST_BF16_TO_FP16=0,QUANTIZATION_MIN_SIZE_KB=256) stay in the 8k1k high-conc fast path.Search space (TP4, conc 1-512 at 1k1k and 8k1k) is unchanged.
Performance vs #2003 (tput/GPU) — every concurrency is a win
e2e run (test-config, mi355x-amds): https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29342987628
No regression at any concurrency (gains +0.2% to +11.6%); peak throughput +8.5% (1k1k conc512) / +8.6% (8k1k conc512). Switching to INT4 quick all-reduce resolved the earlier 8k1k conc8 (-4.7% -> +0.2%) and conc64 (-0.9% -> +2.4%) dips seen with INT6.
Test plan
test-configgreen onmi355x-amds(run 29342987628), all 20 throughput + 2 eval jobsvllm-project/recipes) reflecting sparse PA + gated linear-backend + INT4 for MXFP8 MI355X (for CODEOWNER sign-off)中文说明
将
minimaxm3-fp8-mi355x-vllm镜像升级到nightly-9e57de7197f234f9d9187715d96e07e007048c0f（已包含合并进主分支的 AITER page-16 稀疏分页注意力 vllm-project/vllm#47287），并针对该 nightly 重新调优 serve 命令：VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1，配合VLLM_ROCM_USE_AITER=1与 fp8 KV cache，TP4 下每 ranknum_kv_heads == 1）。这是对同一稀疏注意力计算的 kernel 级加速，不减少 FLOPs（按docs/PR_REVIEW_CHECKLIST.md，刻意不使用 #47269 的--hf-overridesindexer 跳层开关）。MM3_HIGH_CONC_FASTPATH=isl>=8192 且 conc>=64）。稀疏分页注意力是长上下文/大批量优化，在短上下文(1k1k)与低并发下会回退性能，因此VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT与--linear-backend emulation仅在该区间启用，其余回退到 [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 路径（非 shuffle Triton 注意力 + 原生 linear）。--linear-backend emulation）。在该 nightly 上原生 Triton MXFP8 GEMM 在低/中并发更快；emulation 仅在 8k1k 高并发占优并被门控于此。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4；[AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003 为 INT6）——更廉价的集合通信抬升整条曲线并消除低/中并发的回退；准确性由 8k1k evals 保障。AITER 调优开关（CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256）保留在 8k1k 高并发快速路径。搜索空间（TP4，1k1k 与 8k1k 并发 1-512）保持不变。相对 #2003 每个并发点均为正收益（+0.2% ~ +11.6%），峰值吞吐 +8.5%(1k1k)/+8.6%(8k1k)；改用 INT4 quick all-reduce 后消除了此前 INT6 下 8k1k conc8(-4.7%->+0.2%)、conc64(-0.9%->+2.4%) 的回退。