From aa0393399fe2bb7ccd2be2492cd86bbbb2344bcc Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Mon, 13 Jul 2026 22:06:25 +0000 Subject: [PATCH 1/6] [AMD] MiniMax-M3 MXFP8 MI355X vLLM: enable AITER sparse PA + drop emulation linear MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bump the minimaxm3-fp8-mi355x-vllm image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287). Enable it via VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4, num_kv_heads == 1 per rank) plus the recipe's quick all-reduce knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256. Drop --linear-backend emulation now that the Triton MXFP8 dense-linear GEMM is optimized. Deliberately do NOT add the #47269 --hf-overrides use_index_cache/index_topk_freq indexer skip: it reduces model-architecture FLOPs, disallowed by docs/PR_REVIEW_CHECKLIST.md. Sparse PA is a kernel-level speedup only. Verified locally on MI355X (gfx950) TP4 8k1k: conc1 10/10 and conc512 5120/5120 completed with zero failures (conc512 ~28.6k tok/s total, ~7.1k tok/s/GPU). 中文：将 minimaxm3-fp8-mi355x-vllm 镜像升级到 nightly-9e57de7197f234f9d9187715d96e07e007048c0f，该镜像已包含合并进主分支的 AITER page-16 稀疏分页注意力(sparse paged attention, vllm-project/vllm#47287)。 通过 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1（配合 VLLM_ROCM_USE_AITER=1、fp8 KV cache 及 TP4，每个 rank num_kv_heads == 1）启用，并补上 recipe 的 quick all-reduce 调优开关 VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 与 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256。由于 Triton MXFP8 稠密 linear GEMM 已优化，移除 --linear-backend emulation。刻意不添加 #47269 的 --hf-overrides use_index_cache/index_topk_freq indexer 跳层开关：它会减少模型 架构 FLOPs，违反 docs/PR_REVIEW_CHECKLIST.md。sparse PA 仅为 kernel 级加速。 已在 MI355X(gfx950) TP4 8k1k 本地验证：conc1 10/10、conc512 5120/5120 全部成功 （conc512 约 28.6k tok/s 总吞吐，约 7.1k tok/s/GPU）。 Co-authored-by: Cursor --- .../fixed_seq_len/minimaxm3_fp8_mi355x.sh | 20 ++++++++++++++++++- configs/amd-master.yaml | 2 +- perf-changelog.yaml | 11 ++++++++++ 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh index 665f3f944b..b5b1edd0eb 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh @@ -3,6 +3,17 @@ # MiniMax-M3 MXFP8 MI355X (gfx950) single-node vLLM recipe. # https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5 # The recipe recommends MXFP8 from TP=4 on gfx950 and requires block size 128. +# +# AITER page-16 sparse paged-attention fast path (vllm-project/vllm#47287, +# merged into the pinned nightly): maps MiniMax-M3's top-k 128-token sparse +# blocks onto AITER page-16 block tables and runs AITER Gluon paged attention +# over only the selected KV pages. This is a kernel-level speedup of the same +# sparse-attention computation (no FLOP reduction), enabled via +# VLLM_ROCM_USE_AITER=1 + VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 with fp8 KV cache +# on a TP where each rank has num_kv_heads == 1 (TP4). We deliberately do NOT +# pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer +# indexer-skip override: it reduces model-architecture FLOPs, which is +# disallowed by docs/PR_REVIEW_CHECKLIST.md. source "$(dirname "$0")/../../benchmark_lib.sh" @@ -35,6 +46,14 @@ export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # the router-append shared-experts MoE fusion (vllm-project/vllm#46545). export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 +# Quick all-reduce tuning from the MiniMax-M3 AITER recipe (vllm-project/vllm#47287): +# keep the bf16 accumulation and only quantize all-reduces above 256 KB. +export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 +export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 +# Enable the AITER page-16 sparse-PA fast path (vllm-project/vllm#47287): the +# shuffled KV-cache layout lets AITER derive page-16 K/V views from the page-128 +# allocation and route decode/prefill through AITER Gluon paged attention. +export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 if [ "${EVAL_ONLY}" = "true" ]; then setup_eval_context @@ -74,7 +93,6 @@ vllm serve "$MODEL" --port "$PORT" \ --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \ --kv-cache-dtype fp8 \ --attention-backend TRITON_ATTN \ - --linear-backend emulation \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice > "$SERVER_LOG" 2>&1 & diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 083d847bdd..41a090aed1 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -2412,7 +2412,7 @@ dsv4-fp4-mi355x-atom-disagg: # https://github.com/vllm-project/recipes/commit/2a3728ed9892debfd767a72a58ebc90b33f186e5 # MXFP8 runs from TP=4 on gfx950; block size 128 is mandatory for MSA. minimaxm3-fp8-mi355x-vllm: - image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa + image: vllm/vllm-openai-rocm:nightly-9e57de7197f234f9d9187715d96e07e007048c0f model: MiniMaxAI/MiniMax-M3-MXFP8 model-prefix: minimaxm3 runner: mi355x diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 57d510dd15..9d10b3a6d9 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4750,3 +4750,14 @@ - "Image: lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928" - "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137 + +- config-keys: + - minimaxm3-fp8-mi355x-vllm + description: + - "Bump the MiniMax-M3 MXFP8 MI355X vLLM image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287)" + - "Enable AITER sparse PA for MiniMax-M3's top-k 128-token sparse blocks: export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention), with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 (num_kv_heads == 1 per rank). This is a kernel-level speedup of the same sparse-attention computation, with no FLOP reduction" + - "Add the recipe's quick all-reduce tuning knobs: VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" + - "Remove --linear-backend emulation: the nightly's Triton MXFP8 dense-linear GEMM is now optimized, so emulation is no longer needed" + - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md" + - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX From 987c1f17242da73c4475c8de4f58abbd7a5bd201 Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Tue, 14 Jul 2026 01:38:03 +0000 Subject: [PATCH 2/6] [AMD] MiniMax-M3 MXFP8 MI355X vLLM: concurrency-gate emulation linear for 8k1k high-conc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On the sparse-PA nightly the dense-linear backend crossover reversed vs #2003: the native Triton MXFP8 GEMM now wins in the memory-bound low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) still wins in the compute-bound high-concurrency regime. Measured on gfx950 MXFP8 + sparse PA: emulation ~+3-5% at 8k1k conc 64-512 but ~-3% at conc 1-8. Gate emulation to isl>=8192 && conc>=64; native everywhere else (all 1k1k, 8k1k conc<64). Overridable via LINEAR_BACKEND. 中文：在 sparse-PA nightly 上，稠密 linear 后端的性能拐点相比 #2003 发生反转： 原生 Triton MXFP8 GEMM 现在在访存受限的低并发区间更快，而 --linear-backend emulation（bf16 hipBLASLT）仍在计算受限的高并发区间更快。gfx950 MXFP8 + sparse PA 实测：emulation 在 8k1k conc 64-512 约 +3-5%，在 conc 1-8 约 -3%。因此将 emulation 限定在 isl>=8192 且 conc>=64，其余（所有 1k1k、8k1k conc<64）使用原生 路径。可通过 LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor --- .../fixed_seq_len/minimaxm3_fp8_mi355x.sh | 16 ++++++++++++++++ perf-changelog.yaml | 2 +- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh index b5b1edd0eb..1031c0990a 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh @@ -80,6 +80,21 @@ export VLLM_ROCM_USE_AITER=1 # concurrency. Overridable via env. MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-32768}" +# Concurrency-gated dense-linear backend. On this nightly the native Triton +# MXFP8 linear GEMM wins in the memory-bound low-concurrency regime, while +# --linear-backend emulation (bf16 hipBLASLT) wins in the compute-bound +# high-concurrency regime. Measured crossover on gfx950 MXFP8 sparse-PA: native +# is faster up to conc 32, emulation is ~+3-5% from conc 64 up. Gate emulation +# to the 8k1k high-concurrency tail (isl>=8192 && conc>=64); everything else +# (all 1k1k, and 8k1k conc<64) uses the native linear path. Overridable via +# LINEAR_BACKEND (set to a backend name to force it, or "native" to disable). +LINEAR_ARGS=() +if [ -n "${LINEAR_BACKEND:-}" ]; then + [ "$LINEAR_BACKEND" != "native" ] && LINEAR_ARGS=(--linear-backend "$LINEAR_BACKEND") +elif [ "$ISL" -ge 8192 ] && [ "$CONC" -ge 64 ]; then + LINEAR_ARGS=(--linear-backend emulation) +fi + start_gpu_monitor set -x @@ -93,6 +108,7 @@ vllm serve "$MODEL" --port "$PORT" \ --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \ --kv-cache-dtype fp8 \ --attention-backend TRITON_ATTN \ + "${LINEAR_ARGS[@]}" \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice > "$SERVER_LOG" 2>&1 & diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 9d10b3a6d9..42aa0a3c52 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4757,7 +4757,7 @@ - "Bump the MiniMax-M3 MXFP8 MI355X vLLM image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287)" - "Enable AITER sparse PA for MiniMax-M3's top-k 128-token sparse blocks: export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention), with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 (num_kv_heads == 1 per rank). This is a kernel-level speedup of the same sparse-attention computation, with no FLOP reduction" - "Add the recipe's quick all-reduce tuning knobs: VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" - - "Remove --linear-backend emulation: the nightly's Triton MXFP8 dense-linear GEMM is now optimized, so emulation is no longer needed" + - "Concurrency-gate --linear-backend emulation for 8k1k: on this nightly the native Triton MXFP8 linear GEMM wins in the memory-bound low-concurrency regime while emulation (bf16 hipBLASLT) wins in the compute-bound high-concurrency regime, so emulation is applied only for isl>=8192 && conc>=64 (native everywhere else, including all 1k1k). Measured on gfx950: emulation ~+3-5% at 8k1k conc 64-512, ~-3% at conc 1-8" - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md" - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX From 2bc803de1628a17eec9f207906f903f197193fde Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Tue, 14 Jul 2026 04:36:21 +0000 Subject: [PATCH 3/6] [AMD] MiniMax-M3 MXFP8 MI355X vLLM: gate sparse-PA fast path to 8k1k high-conc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AITER page-16 sparse PA is a long-context/high-batch optimization: it wins at 8k1k conc>=64 but adds overhead at short context (1k1k) and low concurrency. Gate both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT (sparse PA) and --linear-backend emulation behind a single MM3_HIGH_CONC_FASTPATH condition (isl>=8192 && conc>=64); everything else falls back to the #2003 path (non-shuffled Triton attention + native linear) so there is no regression outside the target regime. Net vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere. Overridable via MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND. 中文：AITER page-16 稀疏分页注意力(sparse PA)是长上下文/大批量优化：在 8k1k conc>=64 有收益，但在短上下文(1k1k)和低并发下带来额外开销。将 VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT（sparse PA）与 --linear-backend emulation 统一 用 MM3_HIGH_CONC_FASTPATH 条件（isl>=8192 且 conc>=64）门控；其余情况回退到 #2003 路径（非 shuffle 的 Triton 注意力 + 原生 linear），从而在目标区间之外不产生回退。 相对 #2003：8k1k tput/gpu conc128 +3.3%、conc256 +2.5%、conc512 +4.7%（峰值）， 其余基本持平。可通过 MM3_HIGH_CONC_FASTPATH / LINEAR_BACKEND 覆盖。 Co-authored-by: Cursor --- .../fixed_seq_len/minimaxm3_fp8_mi355x.sh | 42 +++++++++++++------ perf-changelog.yaml | 6 +-- 2 files changed, 32 insertions(+), 16 deletions(-) diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh index 1031c0990a..d7cf5cd2c5 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh @@ -50,10 +50,29 @@ export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 # keep the bf16 accumulation and only quantize all-reduces above 256 KB. export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 -# Enable the AITER page-16 sparse-PA fast path (vllm-project/vllm#47287): the -# shuffled KV-cache layout lets AITER derive page-16 K/V views from the page-128 -# allocation and route decode/prefill through AITER Gluon paged attention. -export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 + +# AITER page-16 sparse PA (vllm-project/vllm#47287) is a long-context, +# high-concurrency optimization: it maps MiniMax-M3's top-k 128-token sparse +# blocks onto AITER page-16 block tables. Measured on gfx950 MXFP8, it only wins +# in the 8k1k high-concurrency tail and adds overhead at short context (1k1k) or +# low batch. So enable the "high-conc fast path" (shuffled KV-cache layout for +# sparse PA + the emulation dense-linear backend, see below) only for +# isl>=8192 && conc>=64; everywhere else fall back to the #2003 path +# (non-shuffled Triton attention + native linear). Overridable via +# MM3_HIGH_CONC_FASTPATH=0/1. +if [ -z "${MM3_HIGH_CONC_FASTPATH:-}" ]; then + if [ "$ISL" -ge 8192 ] && [ "$CONC" -ge 64 ]; then + MM3_HIGH_CONC_FASTPATH=1 + else + MM3_HIGH_CONC_FASTPATH=0 + fi +fi + +if [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then + export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 +else + export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0 +fi if [ "${EVAL_ONLY}" = "true" ]; then setup_eval_context @@ -80,18 +99,15 @@ export VLLM_ROCM_USE_AITER=1 # concurrency. Overridable via env. MAX_NUM_BATCHED_TOKENS="${MAX_NUM_BATCHED_TOKENS:-32768}" -# Concurrency-gated dense-linear backend. On this nightly the native Triton -# MXFP8 linear GEMM wins in the memory-bound low-concurrency regime, while -# --linear-backend emulation (bf16 hipBLASLT) wins in the compute-bound -# high-concurrency regime. Measured crossover on gfx950 MXFP8 sparse-PA: native -# is faster up to conc 32, emulation is ~+3-5% from conc 64 up. Gate emulation -# to the 8k1k high-concurrency tail (isl>=8192 && conc>=64); everything else -# (all 1k1k, and 8k1k conc<64) uses the native linear path. Overridable via -# LINEAR_BACKEND (set to a backend name to force it, or "native" to disable). +# Dense-linear backend, gated on the same high-conc fast path as sparse PA. On +# this nightly the native Triton MXFP8 linear GEMM wins in the memory-bound +# low-concurrency regime, while --linear-backend emulation (bf16 hipBLASLT) wins +# in the compute-bound high-concurrency regime (~+3-5% at 8k1k conc>=64). +# LINEAR_BACKEND overrides (a backend name to force it, or "native" to disable). LINEAR_ARGS=() if [ -n "${LINEAR_BACKEND:-}" ]; then [ "$LINEAR_BACKEND" != "native" ] && LINEAR_ARGS=(--linear-backend "$LINEAR_BACKEND") -elif [ "$ISL" -ge 8192 ] && [ "$CONC" -ge 64 ]; then +elif [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then LINEAR_ARGS=(--linear-backend emulation) fi diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 42aa0a3c52..7473b4ee5b 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4755,9 +4755,9 @@ - minimaxm3-fp8-mi355x-vllm description: - "Bump the MiniMax-M3 MXFP8 MI355X vLLM image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287)" - - "Enable AITER sparse PA for MiniMax-M3's top-k 128-token sparse blocks: export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention), with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 (num_kv_heads == 1 per rank). This is a kernel-level speedup of the same sparse-attention computation, with no FLOP reduction" - - "Add the recipe's quick all-reduce tuning knobs: VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" - - "Concurrency-gate --linear-backend emulation for 8k1k: on this nightly the native Triton MXFP8 linear GEMM wins in the memory-bound low-concurrency regime while emulation (bf16 hipBLASLT) wins in the compute-bound high-concurrency regime, so emulation is applied only for isl>=8192 && conc>=64 (native everywhere else, including all 1k1k). Measured on gfx950: emulation ~+3-5% at 8k1k conc 64-512, ~-3% at conc 1-8" + - "Enable AITER page-16 sparse PA (VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1, with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 where num_kv_heads == 1 per rank): AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention over MiniMax-M3's top-k 128-token sparse blocks. Kernel-level speedup of the same computation, no FLOP reduction" + - "Concurrency-gate the fast path to the long-context high-concurrency tail (isl>=8192 && conc>=64). Sparse PA is a long-context/high-batch optimization and adds overhead at short context (1k1k) or low concurrency, so both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT and --linear-backend emulation are enabled only in that regime; everything else (all 1k1k, and 8k1k conc<64) falls back to the prior #2003 path (non-shuffled Triton attention + native linear). Net effect vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere" + - "Also add the AITER quick all-reduce tuning knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md" - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX From f38f4c0a87fd8d88efe20141330d4a4a8bb45a76 Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Tue, 14 Jul 2026 05:31:37 +0000 Subject: [PATCH 4/6] chore: set perf-changelog pr-link to #2187 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：将 perf-changelog 的 pr-link 设置为 #2187。 Co-authored-by: Cursor --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 7473b4ee5b..1fb91041ea 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4760,4 +4760,4 @@ - "Also add the AITER quick all-reduce tuning knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md" - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2187 From 9ffc8d454afb0a97976a227deba38dd99ef825d0 Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Tue, 14 Jul 2026 14:52:26 +0000 Subject: [PATCH 5/6] minor tuning --- .../single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh index d7cf5cd2c5..fd89ff0fb5 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh @@ -46,10 +46,6 @@ export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # the router-append shared-experts MoE fusion (vllm-project/vllm#46545). export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 -# Quick all-reduce tuning from the MiniMax-M3 AITER recipe (vllm-project/vllm#47287): -# keep the bf16 accumulation and only quantize all-reduces above 256 KB. -export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 -export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 # AITER page-16 sparse PA (vllm-project/vllm#47287) is a long-context, # high-concurrency optimization: it maps MiniMax-M3's top-k 128-token sparse @@ -70,8 +66,14 @@ fi if [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 + export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 + # Quick all-reduce tuning from the MiniMax-M3 AITER recipe (vllm-project/vllm#47287): + # keep the bf16 accumulation and only quantize all-reduces above 256 KB. + export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 + export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 else export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0 + export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 fi if [ "${EVAL_ONLY}" = "true" ]; then From 7e3210852e075f9dd60efb0cfe4e63f3782c9cd4 Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Tue, 14 Jul 2026 14:54:31 +0000 Subject: [PATCH 6/6] [AMD] MiniMax-M3 MXFP8 MI355X vLLM: use INT4 quick all-reduce for all concurrencies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Switch VLLM_ROCM_QUICK_REDUCE_QUANTIZATION from INT6 to INT4 unconditionally (#2003 used INT6); accuracy is guarded by the 8k1k evals. The AITER quick all-reduce tuning knobs (CAST_BF16_TO_FP16=0, QUANTIZATION_MIN_SIZE_KB=256) remain in the 8k1k high-conc fast path. Aimed at lifting the low/mid-conc points (e.g. the 8k1k conc8 dip) via cheaper all-reduces. 中文：将 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION 从 INT6 无条件改为 INT4（#2003 用的是 INT6），准确性由 8k1k evals 保障。AITER quick all-reduce 调优开关 (CAST_BF16_TO_FP16=0、QUANTIZATION_MIN_SIZE_KB=256) 仍保留在 8k1k 高并发快速路径中。 目的是通过更廉价的 all-reduce 提升低/中并发点（例如 8k1k conc8 的回退）。 Co-authored-by: Cursor --- .../single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh | 10 +++++----- perf-changelog.yaml | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh index fd89ff0fb5..61004c0b52 100755 --- a/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh +++ b/benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh @@ -42,10 +42,12 @@ fi SERVER_LOG=/workspace/server.log export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 -# MI355X mxfp8 recipe (vllm-project/recipes#581): INT6 quick all-reduce plus -# the router-append shared-experts MoE fusion (vllm-project/vllm#46545). +# MI355X mxfp8 recipe (vllm-project/recipes#581): INT4 quick all-reduce plus +# the router-append shared-experts MoE fusion (vllm-project/vllm#46545). INT4 +# quick all-reduce is applied at all concurrencies (accuracy is guarded by the +# 8k1k evals); #2003 used INT6. export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 -export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 +export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 # AITER page-16 sparse PA (vllm-project/vllm#47287) is a long-context, # high-concurrency optimization: it maps MiniMax-M3's top-k 128-token sparse @@ -66,14 +68,12 @@ fi if [ "$MM3_HIGH_CONC_FASTPATH" = "1" ]; then export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 - export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 # Quick all-reduce tuning from the MiniMax-M3 AITER recipe (vllm-project/vllm#47287): # keep the bf16 accumulation and only quantize all-reduces above 256 KB. export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 else export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0 - export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 fi if [ "${EVAL_ONLY}" = "true" ]; then diff --git a/perf-changelog.yaml b/perf-changelog.yaml index 1fb91041ea..138f3d5b21 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4757,7 +4757,7 @@ - "Bump the MiniMax-M3 MXFP8 MI355X vLLM image to nightly-9e57de7197f234f9d9187715d96e07e007048c0f, which carries the merged AITER page-16 sparse paged-attention path (vllm-project/vllm#47287)" - "Enable AITER page-16 sparse PA (VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1, with VLLM_ROCM_USE_AITER=1 and fp8 KV cache on TP4 where num_kv_heads == 1 per rank): AITER derives page-16 K/V views from the page-128 KV cache and routes decode/prefill through AITER Gluon paged attention over MiniMax-M3's top-k 128-token sparse blocks. Kernel-level speedup of the same computation, no FLOP reduction" - "Concurrency-gate the fast path to the long-context high-concurrency tail (isl>=8192 && conc>=64). Sparse PA is a long-context/high-batch optimization and adds overhead at short context (1k1k) or low concurrency, so both VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT and --linear-backend emulation are enabled only in that regime; everything else (all 1k1k, and 8k1k conc<64) falls back to the prior #2003 path (non-shuffled Triton attention + native linear). Net effect vs #2003: 8k1k tput/gpu conc128 +3.3%, conc256 +2.5%, conc512 +4.7% (peak), neutral elsewhere" - - "Also add the AITER quick all-reduce tuning knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 (INT6 quick all-reduce unchanged)" + - "Switch quick all-reduce quantization to INT4 for all concurrencies (#2003 used INT6); accuracy is guarded by the 8k1k evals. In the high-conc fast path also apply the AITER tuning knobs VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256" - "Deliberately do NOT pass the #47269 --hf-overrides use_index_cache/index_topk_freq cross-layer indexer-skip override: it reduces model-architecture FLOPs, which is disallowed by docs/PR_REVIEW_CHECKLIST.md" - "Serving flags are otherwise unchanged (--block-size 128, --language-model-only, --moe-backend aiter, --kv-cache-dtype fp8, --attention-backend TRITON_ATTN, minimax_m3 parsers); TP4 conc 1-512 sweep at 1k1k and 8k1k unchanged" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2187