Skip to content

MoE prefill bf16 perf improvement for qwen-3.5-35B-A3B #18829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
digantdesai wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

MoE prefill bf16 perf improvement for qwen-3.5-35B-A3B #18829

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bf0eb3a
Add CUDA sort shim for AOTI export (thrust-based sort_stable fallback)
digantdesai Apr 9, 2026
31e3f79
Add MoE benchmark for Triton fused_moe kernel vs eager/compile baselines
digantdesai Apr 10, 2026
f352f3f
Optimize MoE scale loading: broadcast when BLOCK_SIZE_K <= group_size
digantdesai Apr 10, 2026
707797b
Add batched tensor-core MoE kernel for prefill
digantdesai Apr 10, 2026
5055971
Route prefill MoE to batched tensor-core kernel in Qwen3.5 export
digantdesai Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion backends/cuda/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,9 @@ set(_aoti_cuda_shim_sources runtime/shims/memory.cpp

# Only build int4mm shim when CUDA language/toolchain is available.
if(CMAKE_CUDA_COMPILER)
list(APPEND _aoti_cuda_shim_sources runtime/shims/int4mm.cu)
list(APPEND _aoti_cuda_shim_sources runtime/shims/int4mm.cu
runtime/shims/sort.cu
)
endif()

add_library(aoti_cuda_shims SHARED ${_aoti_cuda_shim_sources})
Expand Down
Loading
Loading