Fix throughput-test: real root cause (context-length mismatch), switch to agentic-replay recipe#25
Open
aistackdev wants to merge 51 commits into
Open
Fix throughput-test: real root cause (context-length mismatch), switch to agentic-replay recipe#25aistackdev wants to merge 51 commits into
aistackdev wants to merge 51 commits into
Conversation
- runners.yaml: register `h100-1x` runner type with our GreenNode H100 node - nvidia-master.yaml: add `gptoss-fp4-h100-1x-vllm` (openai/gpt-oss-20b at FP4, TP=1, conc 4-16) — sized to fit a single 80GB H100, used as a smoke test - runners/launch_h100-greennode.sh: plain-Docker launch script for single-VM GPU boxes (existing scripts assume Slurm + enroot, which we don't have on GreenNode) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
callanjfox/kv-cache-tester → vngcloud/kv-cache-tester cquil11/aiperf → vngcloud/aiperf Lets a single fine-grained PAT scoped to vngcloud org authorize both the main repo and submodule clones during actions/checkout, without needing a two-step anonymous-checkout pattern. Pinned commit SHAs already exist in the forks, so no submodule re-pin required.
- runners.yaml: register rtx5090-1x runner type pointing at our 5090 node - nvidia-master.yaml: dsr1qwen3-bf16-rtx5090-1x-vllm config (deepseek-ai/DeepSeek-R1-0528-Qwen3-8B, BF16, TP=1, conc 4-16) - runners/launch_rtx5090-greennode.sh: plain-Docker launcher - benchmarks/single_node/dsr1qwen3_bf16_rtx5090.sh: vLLM serve + bench script for the 8B distill on a single 5090
- nvidia-master.yaml: 8k1k fixed-seq-len + agentic-coding scenarios
added to gptoss-fp4-h100-1x and both dsr1qwen3-rtx5090 configs.
8k1k single-node entries auto-trigger lm-eval at median/highest conc
via InferenceX's mark_eval_entries logic — so this also enables eval.
- New dsr1qwen3-fp8-rtx5090-1x-vllm config (vLLM on-the-fly FP8 quant).
- Agentic scripts for dsr1qwen3 on RTX 5090 (bf16 + fp8 variants).
TORCH_CUDA_ARCH_LIST=12.0 for sm_120 Blackwell consumer.
- dsr1qwen3_bf16_rtx5090.sh: MAX_MODEL_LEN now respects the env var
(${MAX_MODEL_LEN:-9472}) so 8k1k fits.
Matrix: 30 entries (gptoss h100-1x: 10, dsr1qwen3-bf16 5090: 10,
dsr1qwen3-fp8 5090: 10 — each = 6 fixed-seq-len + 4 agentic).
Mirrors the 1x config (same gpt-oss-20b model, image, scenarios) but TP=2 on h100-greennode_01. Gives direct apples-to-apples scaling comparison: 1x H100 (TP=1) vs 2x H100 (TP=2) on identical workload.
The single character at EOF made the script exit 127 after every successful gpt-oss benchmark, so every gptoss-fp4-h100-1x/-2x job in run 26285383625 ran to completion (full TTFT/TPOT emitted) then was marked failed. dsr1qwen3 jobs on rtx5090 were unaffected (different launch script).
Native Gemma 4 MTP via vLLM v1 --speculative-config (PR vllm-project/vllm#41745, merged 2026-05-08). Targets the 2x H100 GreenNode box (h100-greennode_01), uses google/gemma-4-31B-it target + google/gemma-4-31B-it-assistant drafter (Q-only decoder sharing target's KV cache, ~few hundred MB extra weights, no separate process). Image pinned to v0.22.0 as the first vLLM release post-PR. Drafter HF slug is the presumed 31B name following the E2B/E4B/26B pattern from the PR test plan — may need adjusting if Google's actual upload differs. Sweep: 1k1k + 8k1k * conc {4,8,16}, all spec-decoding: mtp. Comparable to the MEP-0002 prior-art Gemma 4 baseline.
v0.22.0 doesn't exist on Docker Hub. v0.21.0 (tagged 2026-05-15) is the first stable release after PR #41745 was merged (2026-05-06), so it contains the Gemma 4 MTP code path. After v0.21.0 vLLM only ships nightly builds.
aiperf SystemController times out at 'Configure Profiling' on all GreenNode runners. Commented out the agentic-coding blocks for: gptoss-fp4-h100-1x-vllm gptoss-fp4-h100-2x-vllm dsr1qwen3-bf16-rtx5090-1x-vllm dsr1qwen3-fp8-rtx5090-1x-vllm Restore once #2 is resolved. The fixed-seq-len scenarios still run.
Gemma 4 is multimodal; vLLM startup fails with:
ValueError: max_tokens_per_mm_item (2496) is larger than
max_num_batched_tokens (2048)
because the vision encoder emits 2496 tokens per image and chunked
MM input is disabled by default. 8192 covers MM items and also
matches our 8k1k workload upper bound.
Rename gemma4-fp8-h100-2x-vllm-mtp → gemma4-fp8-h100-2x-vllm-bench and add 'spec-decoding: none' rows alongside the existing 'spec-decoding: mtp' rows. Matrix now emits 12 entries (6 mtp + 6 none) per dispatch, so MTP-vs-baseline rows share a wall-clock window — no between-run drift in HF mirror speed, runner thermal, or vLLM patch level. Also fix unrelated YAML typo on line 17 (prefill:wr → prefill:) that was blocking matrix validation.
Validator forbids extra fields on single-node search-space entries, so num_speculative_tokens can't ride the matrix entry directly. Workaround: distinct model-prefix per N, the script derives N from $MODEL_PREFIX. Script filename variants (gemma4n4_fp8_h100.sh, gemma4n6_fp8_h100.sh) symlink to gemma4_fp8_h100.sh. Also drop the broken /workspace/spec_metrics_*.txt curl — server.log (already uploaded by the workflow's Upload server logs step) already captures vLLM's 'SpecDecoding metrics:' lines every ~10s during inference, so the curl was redundant. It was also broken: workspace gets wiped by next job's actions/checkout 'clean: true', so only the last job's file survived (empty).
…s search-space fields
Both knobs were previously inexpressible in the matrix schema:
- N was smuggled into model-prefix (gemma4n4/n6) and decoded by a case
statement in gemma4_fp8_h100.sh — this polluted infmax_model_prefix in
agg_bmk.json with fake "models" and forced any consumer to string-parse
the suffix to recover N.
- max-num-batched-tokens was hardcoded to 8192 in the bench script.
Adds them as typed Optional[int] fields on SingleNodeSearchSpaceEntry +
SingleNodeMatrixEntry, plumbs through generate_sweep_configs.py, the
workflow templates (benchmark-tmpl.yml inputs + env + RESULT_FILENAME
slug so concurrent jobs at different knob values don't collide),
launch_h100-greennode.sh RUN_ENV, the bench script, and
process_result.py (emits flat columns in agg_bmk.json) +
compare_results.py (DB lookup key).
Collapses gemma4-fp8-h100-2x-vllm-bench-{n4,n6} back into the single
gemma4-fp8-h100-2x-vllm-bench key with N=2/4/6 as search-space rows,
and deletes the n4/n6 script symlinks. Adds new
gemma4-fp8-h100-1x-vllm-bench (TP=1, mnbt sweep 4k/8k/16k) as the
first config exercising the new max-num-batched-tokens knob.
Dashboard handoff: agg_bmk.json rows gain two optional int columns
(num_speculative_tokens, max_num_batched_tokens). RESULT_FILENAME slug
gains _n<value>_mnbt<value> segments. compare_results.build_config_params
passes both fields as DB lookup params for the comparator to consume
once the schema is migrated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vLLM's APIServer / EngineCore are launched via multiprocessing.spawn, which starts fresh Python interpreters. Those re-import huggingface_hub and call get_token(); if HF Hub was imported before the env var was read on a given code path, the worker emits "You are sending unauthenticated requests to the HF Hub ..." even though the parent shell has HF_TOKEN set (confirmed via the `docker run -e HF_TOKEN` plumbing in launch_*.sh + benchmark-tmpl.yml). Writing the token to ~/.cache/huggingface/token sidesteps env-var propagation entirely — every HF Hub code path falls back to that file. Also exports HUGGING_FACE_HUB_TOKEN for legacy library paths. Auto-invoked at source time so every bench script that does source "$(dirname "$0")/../benchmark_lib.sh" gets the fix without having to add a call. No-op when HF_TOKEN is unset (local smoke runs without GHA secrets). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Qwen3.5-27B (dense) FP8 on the single H100 GreenNode box (h100-1x ->
h100-greennode_00), vLLM v0.21.0. Mirrors the gemma4-fp8-h100-1x-vllm-bench
sweep: isl {1k,8k} x osl 1k x conc {4,8,16}, sweeping max-num-batched-tokens
4k/8k/16k (18 jobs total).
Distinct model-prefix (qwen3.5-27b) so the inferencex.com dashboard does not
conflate this dense model with the existing qwen3.5-397B-A17B MoE configs.
HF slug Qwen/Qwen3.5-27B-FP8 is the presumed name following the
<repo>-<size>-FP8 pattern; confirm against Qwen's actual upload before dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The config qwen3.5-27b-fp8-h100-1x-vllm-bench (d5e5dc1) had no matching launch script, so the runner died at "Launch job script" with exit 127 (benchmarks/single_node/qwen3.5-27b_fp8_h100.sh: not found). Add it as a vLLM single-H100 script modeled on gemma4_fp8_h100.sh: - vLLM auto-detects fp8 from the Qwen/Qwen3.5-27B-FP8 checkpoint's quantization_config (native fp8, dynamic activation scheme), so no --quantization flag; --dtype bfloat16 sets the compute dtype. - Serve flags per the production config: --reasoning-parser qwen3, --enable-auto-tool-choice, --tool-call-parser qwen3_xml, --gpu-memory-utilization 0.90, --trust-remote-code, TP=1. - max-model-len + max-num-batched-tokens threaded from the matrix via config.yaml so the mnbt 4k/8k/16k sweep works; --max-num-seqs=$CONC. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SGLang counterpart to gemma4-fp8-h100-2x-vllm-bench so Gemma 4 31B can be
compared vLLM-vs-SGLang on the same 2x H100 / TP=2 baseline (spec-none).
No SGLang gemma4 config existed before; gemma4 had only ever run on vLLM.
Three pieces:
- runners/launch_h100-greennode.sh: prefer a framework-tagged single_node
script (<prefix>_<prec>_h100_<framework>.sh) with a fallback to the
historical engine-less name. Lets two engines coexist for one model on
this box (b200/b300 launchers already do this). NO spec suffix is added:
scripts like gemma4_fp8_h100.sh branch internally on $SPEC_DECODING, and
no *_h100*_vllm.sh scripts exist, so every current vLLM config (gemma4,
qwen3.5-27b, gptoss) keeps falling back to its legacy name unchanged.
- benchmarks/single_node/gemma4_fp8_h100_sglang.sh: SGLang launch, on-the-fly
fp8 (--quantization fp8) + fp8 KV to mirror the vLLM bench, TP from matrix.
- gemma4-fp8-h100-2x-sglang config: 6 jobs (isl{1k,8k} x conc{4,8,16}).
MTP omitted on purpose — SGLang's gemma4 MTP path differs from vLLM's
--speculative-config drafter; this is baseline-vs-baseline. First dispatch
will also confirm whether SGLang v0.5.12 supports the Gemma 4 arch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Documents the 7-layer path from a config entry in .github/configs/*-master.yaml through generate_sweep_configs.py, e2e-tests.yml, benchmark-tmpl.yml, runners/launch_<family>.sh, to the benchmarks/single_node/*.sh launch script and the dashboard. Captures the edit recipes for the common changes (engine, sweep params, card count, new model), the derived launch-script naming rule, the two silent-failure rules (a new config needs a matching launch script or it dies at "Launch job script" with exit 127; runner != card count), the HF-slug/arch pre-checks, local validation commands, the vngcloud dispatch command, and run-watching/debugging tips. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First SGLang dispatch confirmed SGLang v0.5.12 DOES support Gemma 4 (gemma4_mm.py/gemma4_vision.py load, server starts), but on-the-fly --quantization fp8 crashes in the vision tower: gemma4_vision.py forward -> fp8 apply -> triton_scaled_mm assert (scale_b.shape mismatch), so the scheduler dies and the server never becomes healthy. Fix: serve the pre-quantized RedHatAI/gemma-4-31B-it-FP8-dynamic checkpoint (compressed-tensors, ignore=['re:.*vision.*','lm_head', 're:.*embed_tokens.*']) and drop --quantization. SGLang auto-detects compressed-tensors and honours the ignore list, keeping the vision encoder in bf16 while running the LLM in fp8 — exactly "fp8 LLM, vision unquantized". Slightly different fp8 recipe than the vLLM bench's on-the-fly fp8 (noted in the config comment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cut the prose walkthrough, HF curl recipes, verbose flow diagram and Notes section; keep the load-bearing facts (flow, field reference, derived script-name rule, edit table, the two silent-failure rules, validate + dispatch commands, engine gotchas). ~196 -> ~60 lines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nch)
SGLang baseline for Qwen3.5-27B on a single H100, mirroring the vLLM
qwen3.5-27b-fp8-h100-1x-vllm-bench on the SAME checkpoint
(Qwen/Qwen3.5-27B-FP8) for an engine-to-engine comparison. 6 jobs
(isl {1k,8k} x conc {4,8,16}); no max-num-batched-tokens sweep since
that's a vLLM knob (SGLang uses --chunked-prefill-size, fixed 8192).
Launch script qwen3.5-27b_fp8_h100_sglang.sh adapted from the 397B-A17B
qwen3.5_fp8_h100.sh: dense single-GPU (dropped expert-parallel-size and
the multi-GPU flashinfer allreduce fusion), and — crucially — NO
--quantization. Qwen/Qwen3.5-27B-FP8 is a pre-quantized block-fp8
checkpoint whose quantization_config.modules_to_not_convert already
excludes the whole vision tower (model.visual.*), lm_head, embeddings and
the linear-attn conv1d/in_proj layers; SGLang auto-detects that and keeps
the vision encoder in bf16. Forcing on-the-fly fp8 would quantize the
vision tower and crash in triton_scaled_mm (the gemma4 failure mode).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-ingest) Dashboard ingests only runs you mark ready. Document the two paths: [ingest] commit-prefix convention for the 15-min auto-scan (with the current startswith caveat), and the force-ingest workflow_dispatch by run id that works today. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The auto-ingest concurrency group (cancel-in-progress: false) keeps only one running + one pending, so dispatching several run_urls at once gets the middle ones cancelled. Dispatch sequentially. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rd or rows skip Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add dev to the push and pull_request branch filters in run-sweep.yml so the perf sweep fires on dev as well as main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run sweep on dev branch too
* Add AIPerf benchmark client integration Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: route gemma4 h100 sweep to 2x runner * test: narrow gemma4 h100 sweep * test: benchmark gemma4 with aiperf only * docs: add aiperf integration guide * chore: remove local h100 launcher docs * Trigger Run Sweep when a PR is opened Add the 'opened' pull_request type so the sweep fires as soon as a non-draft PR is opened (still gated by the perf-changelog.yaml path filter), not only on ready_for_review/synchronize/label. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: AIPerf agentic-replay for 3 trace datasets (+ optional --tokenizer) (#9) ## What Integrates the **agentic-replay** scenario type into the InferenceX AIPerf path and wires up three trace datasets, so a recorded `mooncake_trace` JSONL can be replayed once (closed-loop, with think-time) or driven back-to-back as a capacity sweep — all through official AIPerf v0.9.0. ## Datasets added (`benchmarks/single_node/agentic/datasets/`) - `minimax_claude_code_prod_v3.jsonl` — Claude-Code MiniMax production trace (think-time) - `agentic_coding_1variant_64k_150s.jsonl` — synthetic agentic-coding 64k tier (think-time) - `gemma_blend_prod.jsonl` — Gemma blend_prod, single-turn back-to-back (`strip-trace-delays`) ## Plumbing - **Adapter** (`utils/bench_serving/aiperf_adapter.py`): duration-mode stop condition, `--no-fixed-schedule`, `--inter-turn-delay-cap-seconds`, `--dataset-sampling-strategy`, `--benchmark-grace-period`, goodput, and an optional **`--tokenizer`** passthrough (defaults to the served model when unset). - **Matrix/validation** (`generate_sweep_configs.py`, `validation.py`): `agentic-replay` scenario + `tokenizer` field, both flowing through `e2e-tests.yml` → `benchmark-tmpl.yml` → launcher → adapter. - **Launcher** (`qwen3-4b-2507_bf16_h100_vllm.sh`, `runners/launch_h100-greennode.sh`): trace-subset (`#N`), `STRIP_TRACE_DELAYS`, duration-based replay, canonical `REPLAY_ARGS`. - **Configs**: three `*-smoke*` keys on the qwen3-4b-2507 path + matching `perf-changelog.yaml` entries. ## Docs / tests - ADR-0002 (keep both agentic paths: think-time vs back-to-back) + MVP-vs-Mode1 note. - `agentic-replay-run` Claude skill (runbook; inherits config mechanics from `bench-config`). - Unit tests: adapter (tokenizer set/omit), validation, sweep-config, process_result — green. ## Validation 3 datasets validated live at conc=4 on Qwen3-4B (h100-greennode_00): prefix-cache hit MiniMax prod **51.5%**, 64k 1-variant **95.7%**, Gemma blend b2b **3.5%** — spanning the structural spectrum. * feat(aiperf): add utils/aiperf-mooncake submodule (clean v0.9.0 fork) Second aiperf submodule (utils/aiperf-mooncake) pinned to a clean v0.9.0 fork (thangquang09/aiperf) wired via AIPERF_SOURCE_DIR so the mooncake agentic-replay path installs our fork instead of stock PyPI 0.9.0. The weka path (utils/aiperf) is unaffected. See ADR-0003. Also update the agentic-replay-run skill to require the AIPERF_SOURCE_DIR export in the launch script for all three datasets, and CONTEXT.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(skill): add optional DCGM toggle to agentic-replay-run Ask the user (Q6) whether to enable DCGM; when yes, the agent edits the runner launch script to start a dcgm-exporter sidecar (--network host) so AIPerf reaches localhost:9400/metrics for richer GPU telemetry. gpu_metrics.csv unchanged. Includes the copy-paste bash block and first-run port-9400 check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(agentx-weka): Qwen3-4B vLLM launcher + agentx-weka-run skill Adds the agentx-weka (inferencex-agentx-mvp / cc-traces-weka corpus) path for vLLM on h100-greennode, validated by a 90s smoke at conc 2/4 (0 errors). - benchmarks/single_node/agentic/qwen3-4b-weka_bf16_h100_vllm.sh: launcher that isolates aiperf in a clean venv so its anyio/starlette upgrade never reaches vLLM's system python (v0.21.0 crashes otherwise). - benchmark_lib.sh: WEKA_NUM_DATASET_ENTRIES caps the 949-trace corpus (default 949, backward-compatible). - nvidia-master.yaml: qwen3-4b-weka smoke config (no perf-changelog entry, so it does not auto-trigger a sweep; dispatch manually). - .claude/skills/agentx-weka-run: skill documenting the path + the venv / ordering / trace-load / disk gotchas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(aiperf): bump aiperf-mooncake submodule to include non-finite metrics fix Bumps utils/aiperf-mooncake 524fac8b -> 1d2377f5, which merges a backport of upstream ai-dynamo/aiperf#1025 onto the pinned benchtool/aiperf-0.9.0 line. Without the fix, SGLang's NaN sglang:fwd_occupancy gauge (uninitialized before the first forward pass) slips past the old `== float("inf")` filter, gets orjson-encoded as JSON null, fails ServerMetricsRecordMessage validation, and silently drops the ENTIRE scrape — taking cache_hit_rate / cached_tokens with it. The agentic-replay path (/agentic-replay-run, mooncake_trace) scrapes SGLang /metrics with --enable-metrics, so it was affected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(skill): agentic-replay-run smoke uses 2-request warmup Default 20-request warmup runs before profiling and can eat a short smoke window (observed ~750s on 31B/131072-ctx). For duration 90, set --warmup-request-count to 2 in the launch script (not env-plumbed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [dataset] add minimax CC v4 weka-format dataset (234 traces, hackathon week) Production-grounded agentic-coding dataset from vMonitor gateway logs: - Date range: June 10-17, 2026 (hackathon week) - Full capture: 260 sessions pulled, 234 traces (>=2 turns), 21,449 requests - Keyed by x-claude-code-session-id (real CC conversations) - Weka trace format: block_size=64, LCP-based hash_ids - Compaction modeled via reset_context (>10% context change, 6.2% reset rate) - Concurrent bursts linearized - think_time stored RAW (no cap) — runtime capping via scenario config - Latency: latencies.request (full wall-clock) - Validated: pydantic WekaTrace 234/234 pass, cache 86.5% Stats: p50=37 turns, ISL p50=63K, OSL p50=121, think_time p50=0.8s p90=47.6s * feat(agentic): add minimax v4 weka smoke * chore(runner+skill): add DCGM sidecar to h200-greennode launcher; update agentic-replay-run skill - runners/launch_h200-greennode.sh: DCGM exporter sidecar always-on before docker run - .claude/skills/agentic-replay-run/SKILL.md: centralize flow, DCGM default-on, flexible yyyy/mm/dd run-naming pattern Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * submodule+skill(aiperf-mooncake): switch to benchtool/agentx-weka; weka via aiperf-mooncake utils/aiperf-mooncake now tracks benchtool/agentx-weka (forked from ajc/agentx in the thangquang09 repo), which carries the weka_trace loader AND the data_collector math.isfinite NaN filter. This lets weka_trace run through the agentic-replay path on the same submodule as mooncake_trace, with no SGLang runtime patch needed. - .gitmodules: aiperf-mooncake branch benchtool/aiperf-0.9.0 -> benchtool/agentx-weka - pointer: 1d2377f5 -> a75c4612 - agentic-replay-run skill: weka_trace AIPerf source utils/aiperf -> utils/aiperf-mooncake; fork-pin section + confirm-fork log note updated - datasets README: same source update + rationale Do not use utils/aiperf (vngcloud fork) for weka any more: its data_collector.py uses '== float("inf")' which never catches NaN, so SGLang's fwd_occupancy=NaN drops the entire /metrics scrape and would require the runtime patch (patches/aiperf-skip-nonfinite-server-metrics.patch). --------- Co-authored-by: Thắng. Lý Quang (5) <thanglq5@vng.com.vn> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
This reverts commit f0072cc.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…put) (#15) * vendor: bring in aiperf_adapter.py from dev for smoke-test throughput probe Cherry-picking the smoke-test matrix onto a branch off main (rather than merging all of dev, which carries ~30 unrelated commits) needs this one dependency: utils/bench_serving/aiperf_adapter.py, the team's standard aiperf wrapper the throughput probe shells out to. It's stdlib-only (argparse/json/subprocess/pathlib), no other dev-only dependencies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: post-deploy smoke-test matrix (input/process/discovery) Documents the plan for a new correctness+throughput check against live inference-cicd deployments, keyed off its /discover self-report endpoint rather than a hand-maintained catalog. Confirmed live: only sglang-vanilla is currently registered/discoverable; sglang-mooncake-store and sglang-pd-disaggregation exist in-cluster but have no public Ingress yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: use aiperf (not benchmark_serving.py) for smoke-test throughput Corrects the throughput probe design to reuse utils/bench_serving/ aiperf_adapter.py + benchmark_lib.sh's ensure_aiperf (both on dev), per the team's standing decision to standardize throughput checks on aiperf. Confirmed the adapter's synthetic isl/osl mode works standalone against any --url with a plain PyPI install -- no aiperf-mooncake submodule, no self-hosted benchmark-client runner, no Docker required for this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: all 3 stacks now registered in /discover sglang-mooncake-store and sglang-pd-disaggregation are publicly reachable and self-reporting now, closing the gap flagged in the original design. Smoke-test matrix should cover all three from the start. Noted sglang-pd-disaggregation's schema quirk (extra disaggregation field, flat tp that doesn't split prefill/decode) as a non-blocking follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: rewrite smoke-test design docs as current-state only Drop the past-vs-now narration (revision notes, strikethrough, "updated from an earlier version of this doc") that accumulated across iterative edits -- keep only what's true now, for readability. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: note tokens/watt needs a not-yet-exposed DCGM endpoint aiperf already supports --gpu-telemetry-url for power-normalized metrics (aiperf_adapter.py threads it through), and a DCGM exporter does run in the cluster (gpu-operator namespace, port 9400) -- but it's ClusterIP-only and not in /discover's payload for any of the 3 stacks. Tokens/GPU-count doesn't need this (just throughput / discover.tp); tokens/watt does. Documented as a pass-through-if-present, skip-if-absent field so wiring it up later is additive, not a redesign. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: /discover now exposes gpu_metrics_url -- but nodes are multi-tenant inference-cicd added a top-level gpu_metrics_url with a per-node DCGM index. Verified live: sglang-pd-disaggregation has a dedicated node (safe to wire up now), but sglang-vanilla and sglang-mooncake-store share a 4-GPU node -- aiperf's --gpu-telemetry has no pod-label filter, so pointing it at that shared node's URL would mix in the other stack's (and an idle GPU's) power draw. Proposed fix: ask inference-cicd for a per-stack gpu_metrics_url (same convention as version_url), pre-filtered, so aiperf's native ingestion still works unchanged rather than hand-filtering DCGM lines ourselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * design: inference-cicd added per-stack gpu_metrics_url -- gap closed Verified live: each /discover stack entry now has its own gpu_metrics_url (/gpu-metrics/by-stack/<name>), pre-filtered server-side to that stack's own pod's GPU lines -- confirmed for sglang-vanilla and sglang-mooncake-store (which share a physical 4-GPU node) that the other stack's GPUs and an idle GPU are correctly excluded. Wires straight into aiperf --gpu-telemetry-url for all three stacks now, no InferenceX-side filtering needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: add smoke-test workflow (metadata + tool-calling probes) New post-deploy correctness check against inference-cicd's live stacks, per design/smoke-test-matrix.md. Triggered by repository_dispatch (stack-deployed) or manual workflow_dispatch. The matrix is built at run time from inference-cicd's live /discover endpoint, cross-referenced against .github/configs/smoke-tests.yaml (which only declares which probes to run and expected-metadata for drift checks -- never base_url/model/framework/tp, which come from /discover live). Two probes wired up so far, both verified against the live cluster: - metadata: diffs version_url's live report against smoke-tests.yaml's `expect` block. - tool-calling: sends a real tool-enabled chat completion, asserts a tool_calls response. Currently fails against all 3 live stacks (no tool-call parser configured server-side for this model) -- a genuine finding, kept as a hard failure per team decision, not papered over. throughput probe (aiperf-based) intentionally not wired in yet -- next commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: add aiperf-based throughput probe to smoke test Wires the throughput probe into the smoke-test workflow, per design/throughput-test.md: a short aiperf concurrency sweep against the live endpoint via utils/bench_serving/aiperf_adapter.py (the team's standard aiperf wrapper), with everything about where to send requests (--url/--endpoint/--endpoint-type/--model/--gpu-telemetry-url) derived from the live /discover entry -- only isl/osl/concurrency/duration come from smoke-tests.yaml. Includes a redeploy guard: snapshots version_url before and after the sweep and fails the probe if they differ, rather than silently reporting throughput numbers that mixed two deployments. Verified end-to-end against the live cluster (sglang-vanilla, conc=1, 8s duration): aiperf_adapter.py runs correctly, GPU telemetry pass-through works, result JSON parses, redeploy guard reports false as expected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The throughput probe computed real aiperf numbers (tokens/sec, TTFT, ITL per concurrency) but nothing surfaced them anywhere -- confirmed by inspecting the first live CI run (29191249678): the step summary only showed a one-line "completed sweep at conc=[...]" and the underlying data was silently discarded. - run_smoke_test.py now renders a per-concurrency numbers table in the Markdown summary alongside the pass/fail table. - Adds --results-file to write the full raw probe results (all data, not just pass/fail) as JSON, uploaded as a build artifact per stack. Verified locally end-to-end against the live cluster (sglang-vanilla, conc=[1,4]): summary table renders correctly, results JSON has full per-concurrency aiperf output. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Per sync with InferenceX-app (2026-07-12): they'll pull/ingest run_type: live-check results into their own tab, so this repo doesn't need to call trigger-ingest itself. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…coding-trace dataset (#18) Smoke test (metadata + tool-calling) and throughput test are two unrelated checks -- throughput was a bundled probe using tiny synthetic isl/osl padding, undersized and prone to intermittent HF Hub rate-limit crashes under smoke-test.yml's parallel matrix. It's now a standalone throughput-test.yml workflow using aiperf's semianalysis_cc_traces_weka public dataset (real Claude Code coding-session traces) for a richer signal, with HF_TOKEN wired in from the start. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
… submodule, not PyPI (#19) First live run (29196983273) failed at conc=1: pip's aiperf==0.9.0 is upstream NVIDIA's package and rejects the dataset name outright -- semianalysis_cc_traces_weka is a vngcloud/aiperf fork addition, only available via an editable install of the utils/aiperf submodule (same pattern as benchmark_lib.sh's install_agentic_deps()). Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
… AIPerf metrics (#20) * Add a subprocess timeout to the throughput-test aiperf call A cancelled GH Actions run (29197167756) got stuck ~30min with logs never persisted -- GH's cancel signal doesn't reliably interrupt this synchronous subprocess call. Add a bounded timeout (benchmark-duration + 600s setup/ drain buffer) so a stuck aiperf invocation raises a clear error instead of hanging silently and unobservably. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * TEMP diagnostic: shrink sglang-vanilla throughput test to isolate hang (not for main) * Handle AIPerf metric blocks that are legitimately absent (not null) AIPerf's export schema marks every latency/throughput metric block as optional and omits it from the JSON (rather than serializing null) when a run doesn't produce enough samples to compute it -- e.g. inter_token_latency needs multi-token completions, which a short-duration/low-concurrency/ low-dataset-count window may not produce. aiperf_adapter.py's build_result() assumed every block was always present and crashed with a bare KeyError, which silently killed the whole concurrency sweep on the first sparse point instead of just omitting that stat. * Revert "TEMP diagnostic: shrink sglang-vanilla throughput test to isolate hang (not for main)" This reverts commit 95e5c4f. --------- Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…unner (#21) * Run throughput-test's sweep job on our self-hosted benchmark-client runner The last two workflow_dispatch attempts on ubuntu-latest both hit "The runner has received a shutdown signal" mid-sweep -- confirmed via the GitHub API this is GitHub's own hosted-runner fleet (runner_group_name: "GitHub Actions"), not something we can inspect or fix from our side. Switch to our own bench-client_00 self-hosted runner (labels: self-hosted, benchmark-client), which is already online and used elsewhere for aiperf-based client benchmarking, so a long-running sweep against a live endpoint isn't at the mercy of hosted-runner churn. get-jobs stays on ubuntu-latest -- it's a quick /discover call with no long-running step to lose. * Fix pip invocation for the self-hosted runner (no bare pip on PATH) bench-client_00 is a bare host, not ubuntu-latest's preconfigured Python env -- 'pip: command not found'. Use python3 -m pip --break-system-packages, matching benchmarks/benchmark_lib.sh's existing install helpers on the same runner. * Bootstrap pip on the self-hosted runner (python3 has no pip module at all) bench-client_00 failed with 'No module named pip' -- python3 -m pip isn't just missing from PATH, the pip module itself isn't installed. Bootstrap via ensurepip first, falling back to get-pip.py if ensurepip itself was stripped from this Python build. * Add ~/.local/bin to PATH for the aiperf console script Same non-root user constraint as the pip bootstrap: pip installs console scripts under ~/.local/bin on this runner since it can't write to the system bin dir, but that's not on PATH by default -- 'aiperf' (invoked directly as a subprocess command, not a module) was FileNotFoundError. * TEMP: add runner health diagnostic step (not for main) * Revert "TEMP: add runner health diagnostic step (not for main)" This reverts commit be225f2. * Cap aiperf worker count to avoid OOM-killing the benchmark-client runner Confirmed via SSH + journalctl/dmesg: aiperf's default worker auto-scaling (min(concurrency, cpu_count*0.75-1), capped at 32) spawned enough worker processes on our 16-core bench-client host that the sweep grew to ~31.6GB RSS and got OOM-killed by the kernel -- twice, on two different runner registrations on the same physical host, taking the whole runner service down each time (systemd: 'Failed with result oom-kill'). Add --max-workers passthrough to aiperf_adapter.py and cap it at 4 for this workflow's sweep -- it's a lightweight live-check, not a from-scratch sweep, and doesn't need aiperf's full auto-scaled worker pool to sustain concurrency up to 32. * TEMP diagnostic: shrink sglang-vanilla to num-dataset-entries=10 to isolate memory scaling (not for main) * Lower num-dataset-entries default to reduce OOM risk on the self-hosted runner Replaces the single-stack TEMP diagnostic override. Applies num-dataset-entries: 20 to all 3 stacks (down from the 100 default) as a conservative stopgap -- this workflow also fires automatically via repository_dispatch on every deploy, so leaving it at a value known to OOM-kill the runner would crash it on every future deploy, not just manual test dispatches. This does not fix the underlying memory-scaling issue (aiperf's semianalysis_cc_traces_weka loader appears to reconstruct the full 949-trace corpus regardless of num-dataset-entries) -- it only reduces blast radius. Tuning the real fix is left as a follow-up; see design/throughput-test.md and session memory for the full investigation. * Lower max concurrency in throughput sweep from 32 to 16 --------- Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
InferenceX-app's throughput-test ingest needs a GPU model to slot a sweep point into their configs table (part of their uniqueness key), and neither /discover, /version, nor either live-check artifact currently reports it. The data exists in gpu_metrics_url's DCGM Prometheus feed (modelName label per GPU), but querying it at ingest time would be wrong/missing for any run where the pod has since moved, rescheduled, or been torn down -- ingest can lag a run by more than a few minutes, and definitely can for backfill. Add a shared utils/gpu_metrics.py helper that scrapes gpu_metrics_url once at test time and bakes gpu_model into the artifact itself (same philosophy as metadata.data already snapshotting /version verbatim). Wired into both smoke-test (top-level gpu_model, alongside stack/run_type) and throughput-test (data.gpu_model, alongside dataset/sweep). Raises loudly on a pod reporting more than one distinct modelName across GPUs rather than silently picking one; treated as best-effort enrichment otherwise (a lookup failure logs a warning and sets gpu_model: null, it does not fail the smoke/throughput check itself). Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
Confirmed live (see session memory project_toolcalling_limitation.md): at temperature=0.0, DeepSeek-Coder-V2-Lite-Instruct deterministically never attempts a tool call under tool_choice=auto -- a stable decoding-time property of a model that was never tool-call-instruction-tuned, not a parser bug. The probe was testing model initiative, which no server-side fix can change, instead of what we actually care about: whether the server correctly parses and returns a well-formed call once the model is forced to attempt one. Switch to tool_choice=required so the probe tests something this repo can actually detect regressions in. Co-authored-by: Ngô Quang Hòa <hoanq3@vng.com.vn>
InferenceX-app's second ask on the gpu-metrics thread: their configs table's natural key also needs framework/precision/tp (and disaggregation when applicable), and throughput-test/smoke-test are two fully independent workflows with no shared run ID or guaranteed-same timestamp -- they can't safely join them by (stack, latest-date) since a redeploy between the two runs would silently attribute throughput numbers to the wrong config. framework/precision/tp come straight off the matrix entry (already sourced from /discover, no extra call). disaggregation comes from the /version payload run() already fetches for redeploy detection (version_before) -- also no extra call, only included when present, mirroring metadata.data's convention of only pd-disaggregation stacks reporting it.
Confirmed recurring on sglang-vanilla (hit on the very first main validation run and twice more validating the config-snapshot feature): a heavy sweep appears to make the stack's own /version endpoint transiently unresponsive right after the sweep finishes. The redeploy-detection re-check wasn't guarded, so this crashed the whole script and discarded an already-completed sweep. Now best-effort: a failure here sets redeployed_mid_run to None (unconfirmed, distinct from confirmed-false) instead of losing the run.
…test for real stats (not for main)
…ostic (not for main)
… stdout (not for main)
…mary tail (not for main)
Found via live investigation (curl reproduction + aiperf's own error_summary export): the semianalysis_cc_traces_weka corpus's real Claude-Code coding sessions routinely exceed 32K tokens, but this deployment serves DeepSeek-Coder-V2-Lite-Instruct-FP8 with a 32768-token max context (a serving-config choice -- the model itself supports up to 128K). Every oversized request was rejected outright with HTTP 400, silently producing all-empty sweep-point stats no matter how num-dataset-entries/conc-list/benchmark-duration-s were tuned -- none of that tuning was ever going to fix a request-rejection problem. Switch the utils/aiperf submodule to thangquang09/aiperf's benchtool/agentx-weka branch, which adds --max-context-length filtering to the weka loader (drops oversized conversations before they hit the endpoint, mirrors WekaTraceLoader._filter_traces_by_max_context). Wire it through aiperf_adapter.py and set MAX_CONTEXT_LENGTH=30000 in run_throughput_test.py (30000, not 32768, to leave headroom for completion tokens). Also: turn the ad-hoc error_summary diagnostic used to find this into a permanent, low-noise one (only logs when aiperf actually reports request errors), and restore production config for sglang-vanilla (the tiny single-conc/120s-duration values were diagnostic-only, not needed now that the real cause is fixed).
Fixes the real root cause found via live investigation: the lightweight concurrency-ping approach (plain --public-dataset sweep, no --scenario) was a fundamental dataset/model-context mismatch, not something num-dataset- entries/conc-list/benchmark-duration-s tuning could ever fix. Deployment context has since been raised to 128K (external change), and this switches to the same recipe benchmark-tmpl.yml/run-sweep.yml use for real agentic coding workloads: --public-dataset semianalysis_cc_traces_weka_with_subagents_060826 --scenario inferencex-agentx-mvp --benchmark-duration 600 --use-server-token-count --num-dataset-entries 64 --slice-duration 1.0 --max-context-length 120000 --unsafe-override All new as code defaults in run_throughput_test.py, overridable per-stack via throughput-tests.yaml (now much shorter -- every field has a sensible default, stacks only need to declare conc-list).
… new recipe (not for main)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes bundled (grew out of the same investigation):
1. Config snapshot for InferenceX-app's ingest (
framework/precision/tp/disaggregationinto throughput artifacts, plus a 503-guard on the post-sweep redeploy re-check) -- see PR history for details, already validated live earlier in this branch's life.2. The real root cause of "throughput-test never produces real numbers": it was never a duration/entries/worker-count tuning problem. Live investigation (curl reproduction against the endpoint + aiperf's own
error_summaryexport) found the actual cause:semianalysis_cc_traces_weka's real Claude-Code coding-session traces (every trace ≥20 turns, cumulative context grows turn over turn) routinely exceeded the deployment's 32768-token max context, so nearly every request was rejected outright with HTTP 400 ("input longer than the model's context length") -- no amount ofnum-dataset-entries/conc-list/benchmark-duration-stuning could ever fix a request-rejection problem.Fixed via two changes together:
utils/aiperfsubmodule tothangquang09/aiperf(benchtool/agentx-wekabranch) and adopted the internal agentic-replay recipe (matchesbenchmark-tmpl.yml/run-sweep.yml) instead of the lightweight concurrency-ping:--public-dataset semianalysis_cc_traces_weka_with_subagents_060826,--scenario inferencex-agentx-mvp,--benchmark-duration 600,--use-server-token-count,--num-dataset-entries 64,--slice-duration 1.0,--max-context-length 120000,--unsafe-override. All wired as code defaults inrun_throughput_test.py, overridable per-stack viathroughput-tests.yaml(now much shorter -- everything has a sensible default).Also added: a permanent (not diagnostic-only)
error_summarysurfacing from aiperf's raw export whenever it's non-empty, so a future silent-failure mode doesn't require re-deriving this from scratch.Confirmed live, all 3 stacks, full production config (conc-list [1,8,16])
Real, comparable throughput data at conc=1 and conc=8 for all three:
sglang-vanilla: 1363 / 696 tok/ssglang-mooncake-store: 1355 / 4789 tok/s (scales notably better at conc=8)sglang-pd-disaggregation: 1346 / 2001 tok/sconc=16consistently disconnects/errors on all three (ServerDisconnectedError/ClientPayloadError) -- a real capacity ceiling finding for these deployments under sustained heavy agentic load, not a bug to chase further.Test plan