Skip to content

[AMD] Tune Kimi-K2.5 MXFP4 MI355X vLLM toward ATOM parity / 调优 Kimi-K2.5 MXFP4 MI355X vLLM，缩小与 ATOM 的性能差距 #2213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Fangzhou-Ai wants to merge 3 commits into
base: main
Choose a base branch
from
+28 −10
Draft

[AMD] Tune Kimi-K2.5 MXFP4 MI355X vLLM toward ATOM parity / 调优 Kimi-K2.5 MXFP4 MI355X vLLM，缩小与 ATOM 的性能差距 #2213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1d24294
perf(amd): tune Kimi K2.5 MXFP4 on MI355X [skip-sweep]
Fangzhou-Ai Jul 14, 2026
601eefc
Update kimik2.5_fp4_mi355x.sh
Fangzhou-Ai Jul 14, 2026
335b39a
Update kimik2.5_fp4_mi355x.sh
Fangzhou-Ai Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions benchmarks/single_node/fixed_seq_len/kimik2.5_fp4_mi355x.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ if [ "${EVAL_ONLY}" = "true" ]; then
MAX_MODEL_LEN="$EVAL_MAX_MODEL_LEN"
fi

if [ "$MAX_MODEL_LEN" -lt 9472 ]; then
MAX_MODEL_LEN=9472
fi

# If the machine runs a MEC FW older than 177, RCCL
# cannot reclaim some memory.
# Disable that features to avoid crashes.
Expand All @@ -47,6 +51,12 @@ fi

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ROCM_USE_SKINNY_GEMM=0
export AITER_MXFP4_INTERMEDIATE=1
export AITER_BYPASS_TUNE_CONFIG=0
export AITER_MOE_SORT_BACKEND=auto
export OMP_NUM_THREADS=1

# Disable AITER RMSNorm for TP < 8 due to accuracy issues
if [ "${TP}" -lt 8 ]; then
Expand All @@ -59,9 +69,6 @@ else
EP=" "
fi

# following AMD andy luo's recipe
# https://x.com/linluo77/status/2017024513595301985

# Start GPU monitoring (power, temperature, clocks every second)
start_gpu_monitor

Expand All @@ -71,8 +78,11 @@ vllm serve $MODEL --port $PORT \
$EP \
--gpu-memory-utilization 0.90 \
--max-model-len $MAX_MODEL_LEN \
--block-size=1 \
--no-enable-prefix-caching \
--kv-cache-dtype fp8 \
--block-size 16 \
--max-num-batched-tokens 16384 \
--max-num-seqs 512 \
--async-scheduling \
--trust-remote-code \
--no-enable-prefix-caching \
--mm-encoder-tp-mode data > $SERVER_LOG 2>&1 &
Expand Down
11 changes: 6 additions & 5 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -770,7 +770,8 @@ kimik2.5-int4-mi300x-vllm:
- { tp: 8, conc-start: 4, conc-end: 64 }

kimik2.5-fp4-mi355x-vllm:
image: vllm/vllm-openai-rocm:v0.24.0
# TODO: replace with the official vLLM ROCm image containing AITER #3832.
image: TBD

@Fangzhou-Ai Fangzhou-Ai Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chunfangamd need new aiter before proceeding

model: amd/Kimi-K2.5-MXFP4
model-prefix: kimik2.5
runner: mi355x
Expand All @@ -782,13 +783,13 @@ kimik2.5-fp4-mi355x-vllm:
- isl: 1024
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 4, conc-start: 4, conc-end: 64 }
- { tp: 8, conc-start: 4, conc-end: 128 }
- { tp: 4, conc-start: 4, conc-end: 128 }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 4, conc-start: 4, conc-end: 64 }
- { tp: 8, conc-start: 4, conc-end: 128 }
- { tp: 4, conc-start: 4, conc-end: 128 }

kimik2.5-fp4-mi355x-vllm-agentic:
image: vllm/vllm-openai-rocm:v0.22.0
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4781,3 +4781,10 @@
- "Bump image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260708"
- "Clean the export envs"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2198

- config-keys:
- kimik2.5-fp4-mi355x-vllm
description:
- "Apply the accuracy-gated Kimi-K2.5 MXFP4 TP4 settings: tuned AITER MXFP4 MoE, fused shared experts, FP8 KV cache, block size 16, 16384 batched tokens, 512 sequences, async scheduling, and the AITER BF16 GEMM path"
- "Extend the existing TP4 and TP8 concurrency sweeps from 64 to 128 without changing other search dimensions; the image remains TBD until an official vLLM ROCm image contains AITER #3832"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2213