Skip to content

Add SPEED-Bench reference synthetic AL values for DeepSeek-V4-Pro MTP 1-8#1592

Open
qiching wants to merge 1 commit into
SemiAnalysisAI:mainfrom
qiching:albecheng/add-dsv4-reference-al
Open

Add SPEED-Bench reference synthetic AL values for DeepSeek-V4-Pro MTP 1-8#1592
qiching wants to merge 1 commit into
SemiAnalysisAI:mainfrom
qiching:albecheng/add-dsv4-reference-al

Conversation

@qiching
Copy link
Copy Markdown

@qiching qiching commented May 30, 2026

Measured with SPEED-Bench coding dataset, temperature=1.0, thinking=true. Values used for synthetic acceptance rate configuration in MTP benchmarks.

AL Values

MTP AL
1 1.90
2 2.60
3 2.97
4 3.04
5 3.13
6 3.08
7 3.13
8 3.12

Note

Low Risk
Documentation-style benchmark reference data only; no runtime or production code paths change.

Overview
Adds benchmarks/speedbench-reference-al.yaml with measured acceptance length (AL) targets for DeepSeek-V4-Pro at MTP levels 1–8, keyed by num_speculative_tokens.

Values come from SPEED-Bench (coding dataset, temperature 1.0, thinking on) and are intended as golden AL inputs for synthetic acceptance rate setup in MTP benchmarks.

Reviewed by Cursor Bugbot for commit 1de285d. Bugbot is set up for automated code reviews on this repo. Configure here.

Measured with SPEED-Bench coding dataset, temperature=1.0, thinking=true.
Values used for synthetic acceptance rate configuration in MTP benchmarks.
@qiching qiching requested a review from a team May 30, 2026 07:11
Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@xinli-sw
Copy link
Copy Markdown
Collaborator

@qiching , a few recommendations

  1. title [1/N] Synthetic MTP - Add SPEED-Bench reference synthetic AL values for DeepSeek-V4-Pro MTP 1-8

  2. In the PR description, mention that we will also have speedbench as part of github workflows, however, as the first iteration, we'd like to get alignment to make sure partners all feel confident and equally about these AL values

  3. attach full repro for the numbers (serve command, installation, speedbench command, etc)

  4. attach full results of the runs you had (the jsonl file) for audibility

Great work so far, cc @benchislett @functionstackx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants