Skip to content

[AMD] Kimi-K2.5 MXFP4 ATOMESH disagg config #2172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
seungrokj wants to merge 19 commits into
base: main
Choose a base branch
from
Open

[AMD] Kimi-K2.5 MXFP4 ATOMESH disagg config #2172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
46f1caf
[AMD][ATOM] Add Kimi-K2.5-MXFP4 disagg config and scheduler-delay-fac…
seungrokj Jul 13, 2026
23f23bb
[AMD][ATOM] Add Kimi-K2.5 disagg config, scheduler-delay-factor suppo…
seungrokj Jul 13, 2026
436eb8f
[AMD][ATOM] Add pr-link to perf-changelog for Kimi-K2.5 disagg entry
seungrokj Jul 13, 2026
ff3b3bc
[AMD][ATOM] Add Kimi-K2.5 FP4 MI355X ATOM disagg benchmark script
seungrokj Jul 13, 2026
ffe3e10
[AMD][ATOM] Add atom-disagg engine support to bench.sh
seungrokj Jul 13, 2026
b6a23a8
[AMD][ATOM] Update Kimi-K2.5 disagg scenarios: 2P1D, 1P4D, 1P2D topol…
seungrokj Jul 13, 2026
f08dabf
[AMD][ATOM] Update Kimi-K2.5 disagg config per upstream ATOM recipe
seungrokj Jul 14, 2026
f8dd2d5
Merge branch 'main' into amd/k2.5_atom_pd_fp4_0713
seungrokj Jul 14, 2026
3a615a1
[AMD][ATOM] Add kv-p2p-transfer: mooncake to Kimi-K2.5 disagg config
seungrokj Jul 14, 2026
5c0bc59
[AMD][ATOM] Simplify Kimi-K2.5 disagg to 1P1D only, extend conc to 1024
seungrokj Jul 14, 2026
a359b81
[AMD][ATOM] Bump atomesh image and add 1P2D topology for Kimi-K2.5 di…
seungrokj Jul 14, 2026
999875b
Merge branch 'main' into amd/k2.5_atom_pd_fp4_0713
seungrokj Jul 14, 2026
0f551b0
[AMD][ATOM] Align Kimi-K2.5 disagg with upstream ATOM 1P1D config
seungrokj Jul 14, 2026
031b250
Merge branch 'main' into amd/k2.5_atom_pd_fp4_0713
seungrokj Jul 14, 2026
89b41fb
Merge branch 'main' into amd/k2.5_atom_pd_fp4_0713
seungrokj Jul 15, 2026
5a32ad9
[AMD][ATOM] Switch Kimi-K2.5 disagg to 1P(TP4) 2D(TP8) topology
seungrokj Jul 15, 2026
1455e32
[AMD][ATOM] Add 1P1D(TP4) topology alongside 1P2D(TP8) for Kimi-K2.5
seungrokj Jul 15, 2026
63fe435
[AMD][ATOM] Adjust conc splits: 1P1D [16-256], 1P2D [1-16]
seungrokj Jul 15, 2026
b06bfd2
Merge branch 'main' into amd/k2.5_atom_pd_fp4_0713
seungrokj Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions benchmarks/multi_node/amd_utils/bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,21 @@ for max_concurrency in "${chosen_concurrencies[@]}"; do

# Engine-specific extra flags
extra_flags=""
# vllm
if [[ "$ENGINE" == "vllm-disagg" ]]; then
extra_flags="--trust-remote-code --tokenizer $MODEL_PATH"
# atom
elif [[ "$ENGINE" == "atom-disagg" ]]; then
extra_flags="--trust-remote-code --tokenizer $MODEL_PATH"
if [ "$IS_MTP" = "true" ]; then
# just override extra_flags as dsv3 use different tokenizer path
if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then
extra_flags="--dsv4"
else
extra_flags="--use-chat-template"
fi
fi
# sglang
else
if [ "$IS_MTP" = "true" ]; then
if [[ "$MODEL_NAME" == "DeepSeek-V4-Pro" ]]; then
Expand Down
9 changes: 9 additions & 0 deletions benchmarks/multi_node/amd_utils/models_atom.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
# max_model_len: str # --max-model-len value (overrides default of unset)
# max_num_seqs: str # --max-num-seqs value (overrides default of 256)
# max_num_batched_tokens: str # --max-num-batched-tokens value (overrides default of unset)
# scheduler_delay_factor: str # --scheduler-delay-factor value (overrides default of unset)

DeepSeek-V4-Pro:
env: "ATOM_MOE_GU_ITLV=1 AITER_BF16_FP8_MOE_BOUND=0"
Expand Down Expand Up @@ -56,6 +57,14 @@ MiniMax-M3-MXFP4:
max_num_seqs: "256"
max_num_batched_tokens: "32768"

Kimi-K2.5-MXFP4:
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 AITER_MXFP4_INTERMEDIATE=1"
kv_cache_flags: "--kv_cache_dtype fp8"
mem_frac_static: "0.9"
max_model_len: "32768"
max_num_seqs: "256"
max_num_batched_tokens: "32768"

MiniMax-M3-MXFP8:
env: "AITER_QUICK_REDUCE_QUANTIZATION=INT4 ATOM_FORCE_ATTN_TRITON=1"
kv_cache_flags: "--kv_cache_dtype fp8"
Expand Down
7 changes: 6 additions & 1 deletion benchmarks/multi_node/amd_utils/server_atom.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ print(f"_YAML_MEM_FRAC_STATIC='{sh(m.get('mem_frac_static', ''))}'")
print(f"_YAML_MAX_MODEL_LEN='{sh(m.get('max_model_len', ''))}'")
print(f"_YAML_MAX_NUM_SEQS='{sh(m.get('max_num_seqs', ''))}'")
print(f"_YAML_MAX_NUM_BATCHED_TOKENS='{sh(m.get('max_num_batched_tokens', ''))}'")
print(f"_YAML_SCHEDULER_DELAY_FACTOR='{sh(m.get('scheduler_delay_factor', ''))}'")
PYEOF
# shellcheck source=/dev/null
source "$_yaml_tmp"
Expand All @@ -118,7 +119,8 @@ MEM_FRAC_STATIC="${_YAML_MEM_FRAC_STATIC:-${MEM_FRAC_STATIC:-0.85}}"
MAX_MODEL_LEN="${_YAML_MAX_MODEL_LEN:-${MAX_MODEL_LEN:-}}"
MAX_NUM_SEQS="${_YAML_MAX_NUM_SEQS:-${MAX_NUM_SEQS:-256}}"
MAX_NUM_BATCHED_TOKENS="${_YAML_MAX_NUM_BATCHED_TOKENS:-${MAX_NUM_BATCHED_TOKENS:-}}"
unset _YAML_BLOCK_SIZE _YAML_MEM_FRAC_STATIC _YAML_MAX_MODEL_LEN _YAML_MAX_NUM_SEQS _YAML_MAX_NUM_BATCHED_TOKENS
SCHEDULER_DELAY_FACTOR="${_YAML_SCHEDULER_DELAY_FACTOR:-${SCHEDULER_DELAY_FACTOR:-}}"
unset _YAML_BLOCK_SIZE _YAML_MEM_FRAC_STATIC _YAML_MAX_MODEL_LEN _YAML_MAX_NUM_SEQS _YAML_MAX_NUM_BATCHED_TOKENS _YAML_SCHEDULER_DELAY_FACTOR

# =============================================================================
# Cluster Topology Configuration
Expand Down Expand Up @@ -211,6 +213,9 @@ fi
if [[ -n "$MAX_NUM_BATCHED_TOKENS" ]]; then
MODEL_LEN_ARGS="${MODEL_LEN_ARGS} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS}"
fi
if [[ -n "$SCHEDULER_DELAY_FACTOR" ]]; then
MODEL_LEN_ARGS="${MODEL_LEN_ARGS} --scheduler-delay-factor ${SCHEDULER_DELAY_FACTOR}"
fi


cat <<INFO
Expand Down
78 changes: 78 additions & 0 deletions benchmarks/multi_node/kimik2.5_fp4_mi355x_atom-disagg.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
#!/usr/bin/env bash

source "$(dirname "$0")/../benchmark_lib.sh"

check_env_vars \
CONC_LIST \
ISL \
OSL \
IMAGE \
MODEL_PATH \
PREFILL_NUM_WORKERS \
PREFILL_TP \
PREFILL_EP \
PREFILL_DP_ATTN \
DECODE_NUM_WORKERS \
DECODE_TP \
DECODE_EP \
DECODE_DP_ATTN \
PREFILL_NODES \
DECODE_NODES \
RANDOM_RANGE_RATIO \
FRAMEWORK

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

set -x

# Use upstreamed multi_node scripts (no external clone needed)
cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1

# Set up SGL launch script-specific environment variables
export TIME_LIMIT="08:00:00"
export MODEL_PATH=$MODEL_PATH
export MODEL_NAME=$MODEL_NAME
export CONTAINER_IMAGE=$IMAGE

if [[ "${PREFILL_EP:-1}" -eq 1 ]]; then
export PREFILL_ENABLE_EP=false
else
export PREFILL_ENABLE_EP=true
fi

if [[ "$PREFILL_DP_ATTN" == "true" ]]; then
export PREFILL_ENABLE_DP=true
else
export PREFILL_ENABLE_DP=false
fi

if [[ "${DECODE_EP:-1}" -eq 1 ]]; then
export DECODE_ENABLE_EP=false
else
export DECODE_ENABLE_EP=true
fi

if [[ "$DECODE_DP_ATTN" == "true" ]]; then
export DECODE_ENABLE_DP=true
else
export DECODE_ENABLE_DP=false
fi

JOB_ID=$(bash ./submit.sh $PREFILL_NODES \
$PREFILL_NUM_WORKERS \
$DECODE_NODES \
$DECODE_NUM_WORKERS \
$ISL $OSL "${CONC_LIST// /x}" inf \
${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \
${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \
${PREFILL_TP} ${DECODE_TP} \
${RANDOM_RANGE_RATIO})

if [[ $? -ne 0 ]]; then
echo "Failed to submit job" >&2
exit 1
fi

echo "$JOB_ID"
53 changes: 53 additions & 0 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -833,6 +833,59 @@ kimik2.5-fp4-mi355x-atom:
search-space:
- { tp: 4, conc-start: 4, conc-end: 128 }

kimik2.5-fp4-mi355x-atom-disagg:
image: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atomesh_202607121715
model: amd/Kimi-K2.5-MXFP4
model-prefix: kimik2.5
runner: mi355x-disagg
precision: fp4
framework: atom-disagg
kv-p2p-transfer: mooncake
multinode: true
disagg: true
scenarios:
fixed-seq-len:
# 1P(tp4)1D(tp4)
- isl: 8192
osl: 1024
search-space:
- spec-decoding: "none"
conc-list: [ 16, 32, 64, 128, 256 ]
prefill:
num-worker: 1
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 1
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "DECODE_NODES=1"
# 1P(tp4)2D(tp8)
- isl: 8192
osl: 1024
search-space:
- spec-decoding: "none"
conc-list: [ 1, 2, 4, 8, 16 ]
prefill:
num-worker: 1
tp: 4
ep: 1
dp-attn: false
additional-settings:
- "PREFILL_NODES=1"
decode:
num-worker: 2
tp: 8
ep: 1
dp-attn: false
additional-settings:
- "DECODE_NODES=2"

dsr1-fp8-mi355x-atom:
image: rocm/atom:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511
model: deepseek-ai/DeepSeek-R1-0528
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4751,6 +4751,14 @@
- "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137

- config-keys:
- kimik2.5-fp4-mi355x-atom-disagg
description:
- "Add Kimi-K2.5-MXFP4 entry to models_atom.yaml for ATOM disaggregated inference"
- "Set env: ATOM_DISABLE_MMAP=true, AITER_QUICK_REDUCE_QUANTIZATION=INT4, AITER_MXFP4_INTERMEDIATE=1, OMP_NUM_THREADS=1"
- "Add scheduler_delay_factor support to models_atom.yaml schema and server_atom.sh; set to 1 for Kimi-K2.5"
Comment on lines +4754 to +4759

@claude claude Bot Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new perf-changelog entry at lines 4737-4742 has two blocking issues that will hard-fail this PR's own sweep CI before any GPU work runs: (1) missing the required pr-link field — every neighboring entry has one, and ChangelogEntry in utils/matrix_logic/validation.py:714 declares pr_link as required with extra="forbid", so ChangelogEntry.model_validate in utils/process_changelog.py:158 will raise a Pydantic ValidationError; (2) the referenced config-keys: [kimik2.5-fp4-mi355x-atom-disagg] does not exist in configs/amd-master.yaml — only the non-disagg kimik2.5-fp4-mi355x-atom (line 811) is defined, so get_config_keys_from_master in utils/process_changelog.py:101-102 will raise ValueError: Config key '\''kimik2.5-fp4-mi355x-atom-disagg'\'' not found in master configs. Fix by appending pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172 and either adding the kimik2.5-fp4-mi355x-atom-disagg entry to configs/amd-master.yaml or updating the changelog to reference an existing key.

Extended reasoning...

What the bug is

The new perf-changelog entry appended at perf-changelog.yaml:4737-4742 has two independent defects that both cause utils/process_changelog.py (invoked from .github/workflows/run-sweep.yml:242 on every labeled PR) to raise before the sweep does any real work.

Issue 1 — missing required pr-link

utils/matrix_logic/validation.py:708-714 defines:

class ChangelogEntry(BaseModel):
    model_config = ConfigDict(extra="forbid", populate_by_name=True)
    config_keys: List[str] = Field(alias="config-keys")
    description: List[str]
    pr_link: str = Field(alias="pr-link")   # no default → required

utils/process_changelog.py:158 runs ChangelogEntry.model_validate(entry_data) on every entry parsed out of the diff. The new entry has no pr-link: line, unlike every neighboring entry (e.g. lines 4710, 4718, 4726, 4735), so validation raises a Pydantic ValidationError: pr-link Field required.

Issue 2 — config-key does not exist in master config

configs/amd-master.yaml contains only kimik2.5-fp4-mi355x-atom (line 811, framework: atom, non-disagg). It has no kimik2.5-fp4-mi355x-atom-disagg key, even though sibling -atom-disagg keys exist for other models (dsv4-fp4-mi355x-atom-disagg L2344, minimaxm3-fp8-mi355x-atom-disagg L2651, minimaxm3-fp4-mi355x-atom-disagg L2721).

utils/process_changelog.py:87-105 get_config_keys_from_master raises ValueError('Config key "{k}" not found in master configs.') for any non-wildcard key not present in the loaded master config. The key here has no *, so it hits that branch and raises.

Step-by-step proof

  1. PR is labeled full-sweep-fail-fast (per the recipe-reminder bot on this PR).
  2. .github/workflows/run-sweep.yml:242 invokes python utils/process_changelog.py.
  3. process_changelog.py calls get_added_lines → yaml.safe_load on the added entries.
  4. For the appended entry, it calls ChangelogEntry.model_validate({'config-keys': ['kimik2.5-fp4-mi355x-atom-disagg'], 'description': [...]}) — no pr-link key.
  5. Pydantic raises ValidationError: pr-link Field required.
  6. Even if pr-link were present, the next call get_config_keys_from_master(['kimik2.5-fp4-mi355x-atom-disagg'], master_config) would then raise ValueError: Config key 'kimik2.5-fp4-mi355x-atom-disagg' not found in master configs. — the key falls into the elif branch at line 101 because it has no * wildcard and is not present in the loaded YAML.
  7. The workflow step exits non-zero and the entire sweep is skipped.

Why existing code doesn't prevent it

Both checks are strict-by-design: extra="forbid" combined with an aliased required field, and get_config_keys_from_master explicitly raises rather than warning. This is intentional — the changelog is authoritative and stale/invalid keys should fail loud. Nothing in the PR touches configs/amd-master.yaml to define the referenced -atom-disagg variant, and nothing added pr-link to the new entry.

Impact

The PR's own full-sweep CI (and any future main-branch sweep processing this changelog line) hard-fails at the changelog validation step, before any GPU allocation. The new Kimi-K2.5 disagg config never gets exercised end-to-end.

How to fix

Two small changes to the entry / master config:

  1. Add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172 under the new entry (matches every neighbor).
  2. Either add a kimik2.5-fp4-mi355x-atom-disagg entry to configs/amd-master.yaml alongside the models_atom.yaml / server_atom.sh changes (framework: atom-disagg, model: amd/Kimi-K2.5-MXFP4, multinode scenarios like the sibling MiniMax entries), or update the changelog line to reference an existing key such as kimik2.5-fp4-mi355x-atom if the disagg config is intentionally deferred.

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2172

- config-keys:
- dsv4-fp8-h200-vllm
- dsv4-fp8-h200-vllm-mtp
Expand Down