pulley/cranelift: opcode fusion at call_indirect lazy-init dispatch tail#13447
pulley/cranelift: opcode fusion at call_indirect lazy-init dispatch tail#13447matthargett wants to merge 12 commits into
Conversation
matthargett
requested review from
cfallin and
fitzgen
and removed request for
a team
CI status13/14 jobs are green. Remaining failure is
Our changes that touch instance/allocation are guarded on Same |
matthargett
force-pushed
the
pulley-fusion-dispatch-tail-upstream
branch
from
dce728e to
9528946
Compare
github-actions
Bot
added
cranelift
Subscribe to Label ActionDetailsThis issue or pull request has been labeled: "cranelift", "cranelift:area:machinst", "cranelift:meta", "isle", "pulley", "wasmtime:api"Thus the following users have been cc'd because of the following labels:
To subscribe or unsubscribe from this label, edit the |
|
Thanks for the PR, and like the previous PR this is stacked on we'd appreciate a bit more care taken when communicating here. Extensively documenting all these numbers is fine, but for example I've no idea what any of these benchmarks are or where their source is. I also don't really know how to reason about how the speedups seem to be balanced by slowdowns, especially in the middle column for the iPhone XS. For "PMU", "Icestorm", "Processing", "Discarded", etc, could you explain what those words all mean? I'm not entirely sure myself... Procedurally it's fine to stack PRs on one another, but given the quantity of commits here this'll probably not get a review until after #13445 has landed and this is rebased. Alternatively, if you'd like, feel free to split out things from this PR (for example the pulley opcodes) and have them land separately. |
PMU is a reference to xcode/xctrace's instruments for profiling, and Processing vs Discarded is CPU pipeline frontend vs backend efficiency. (Discarded is when branch prediction is wrong, so speculative execution results are pure waste.) Icestorm is the name for the effiicency cores (E-cores) of the device, which I'm focusing on to try and get the best performance-per-watt out of the decisions/choices rather than high-wattage and/or performance core uplift to consider the work "done". I fully admit I wasn't up to speed on all the jargon and code/marketing names for the separate silicon IP and profiler tooling in these devices until I started this work. I've tried to use the higher-level terms efficiency cores where possible, but also want to make it easy for people to do their own web/AI searches so they have an easier time than I did on the research and execution. I've been deep into x86, x64, Qualcomm XR2, and Zen 2 profiling and optimziation before, but not on Apple hardware. the benchmarks in in the repo I've referenced before: https://github.com/rebeckerspecialties/wasm-benchmark/ . this shoudl allow anyone to reproduce the WASM runtime performance "bakeoff" on their own Apple devices (iPhone, iPad, Apple TV 4K, Apple Watch, etc). I'm trying to show my work as much as I can here so this can have the best user-visible uplift possible for the extra complexity. IME, performance profiling and optimization can often be a bit hyperbolic and I'm doing my best to show that I'm being data-driven on the real devices I have available to me. |
matthargett
force-pushed
the
pulley-fusion-dispatch-tail-upstream
branch
from
9528946 to
45c7f7e
Compare
matthargett
force-pushed
the
pulley-fusion-dispatch-tail-upstream
branch
from
45c7f7e to
ec03209
Compare
Add `ModuleTranslation::tables_mutated`, a `SecondaryMap<TableIndex, bool>` populated during `ModuleEnvironment::translate` recording whether any function in the module mutates a given table at runtime via `table.set` / `table.fill` / `table.copy` (as dest) / `table.grow` / `table.init`. Imported tables are conservatively marked mutated. Active `elem` segments at instantiation time are part of initial state, not mutations. O(total opcodes) extra pass over each function body. Groundwork for follow-on call_indirect optimizations gated on the predicate; nothing consumes the bit in this commit.
When `call_indirect` resolves to a constant index into a provably immutable funcref table whose contents are statically known from `elem` segments, rewrite the call to a direct `call F` at lowering time. Skips all per-dispatch checks (bounds, null, sig) and replaces the indirect jump with a direct branch. Gated on `is_immutable_funcref_table(table_idx)` (= predicate from the previous commit + statically-known table contents).
…ables When a funcref table is provably immutable AND every entry in its elem segments has the same function signature as the call_indirect's type annotation, the runtime signature check is statically redundant and is elided in `translate_call_indirect`.
When a funcref table is provably immutable AND none of its precomputed elem-segment entries are null, the runtime null check after the funcref load is statically redundant and is elided. Distinct from the sig-check elision: this targets tables that mix sigs but never contain null.
For provably non-growable funcref tables (`!tables_mutated` excludes `table.grow`), the table size is fixed at instantiation and the per-call_indirect bounds-check load can be replaced with a constant fold using `precomputed_funcref_table_contents.len()`.
`crates/environ/tests/table_mutability.rs`: 12 cases covering the mutation-tracking predicate across `table.set`/`fill`/`copy`/`grow`/ `init`, imported tables, multi-table modules, and active-elem-segment behavior.
Three soundness corrections to the call_indirect elision chain: 1. `is_immutable_funcref_table` previously returned true when the table had no per-function `table.set` etc. uses but had a passive elem segment whose `elem.init` could land at runtime. Track the passive-segment dest tables and treat them as potentially mutated. 2. The constant-index direct-call rewrite assumed the resolved funcref's vmctx matched the caller's; correct it to load the callee's `vmctx` from the precomputed `VMFuncRef`. 3. Null-check elision must NOT fire when the precomputed table contains the tagged-null pattern (slot value `1`); add that case. Disas filetests cover each scenario.
Upstream bytecodealliance#13487 moved the precomputed funcref image to Module::table_initialization (TryPrimaryMap<DefinedTableIndex, TryVec<FuncIndex>>, reserved_value() = null) and dropped the TableInitialValue::Null { precomputed } shape. Adapt the three elision predicates to read the new map directly.
Exercise the table shapes the elisions apply to through the *.wast runtime suite: in-bounds calls, signature mismatches, null slots, and out-of-bounds indices all behave identically whether or not the checks were elided at compile time. Covers mixed-signature and uniform-signature tables plus a declared-growable table that is never grown.
matthargett
force-pushed
the
pulley-fusion-dispatch-tail-upstream
branch
from
ec03209 to
1b9013a
Compare
Mirror of the direct-call `call{1,2,3,4}` family: each new op
combines `xmov xN, argN` ABI fixups with the indirect call. Reads
arg values before writing the ABI registers so the sequence is safe
when an argN aliases the corresponding ABI register.
`call_indirect1 dst, arg1`:
x0 = state[arg1]
lr = pc
pc = state[dst]
Saves up to N Pulley dispatches per call_indirect site (one per
moved arg). In practice at least one — the callee vmctx ABI fixup.
Cranelift wiring in the next commit.
Extend `Inst::IndirectCall`'s `info.dest` from `XReg` to
`PulleyCallIndirect { target, args: SmallVec<[XReg; 4]> }`, parallel
to `PulleyCall`. `gen_call_ind_info` pulls the first 0-4 integer
args from `uses` (where they were going through regalloc's
`reg_fixed_use`, synthesising an `xmov` each) into `args`, where
they flow as free reg uses and the emitted `call_indirect{1,2,3,4}`
opcode moves them at call time.
The emit side picks the narrowest op after the same "drop args
already in their ABI register" loop used by direct calls. Net effect
is one fewer Pulley dispatch per moved arg at each call_indirect
site: the ABI argument-move fixups regalloc used to emit as
standalone `xmov` dispatches fold into the call op itself.
Filetest snapshots updated for the new `dest` shape.
Exercises call_indirect at arg counts 0/1/4/6 with interleaved int
and float args over an immutable funcref table, so the emitted
call_indirect{1,2,3,4} ops (and the >4-arg fall-through to the
normal ABI path) are checked for correct argument delivery on all
four engine configs.
matthargett
force-pushed
the
pulley-fusion-dispatch-tail-upstream
branch
from
1b9013a to
1a7b790
Compare
2 participants
TL;DR
Stacks on #13445 (per-table mutability tracking). Adds a fused dispatch op family at the
call_indirectlazy-init brif site (similar in shape to WAMR's preprocessed-bytecode register IR) plus acall_indirect{1,2,3,4}family mirroring direct-callcall{1,2,3,4}. Consistent 4–8 % wallclock wins on polymorphic vtable workloads across iPhone 12 E-core, Apple Watch SE2 , and M4 E-core. iPhone XS e-core is mixed and the reason, is visible in hand-rolled aarch64 assembly microbenchmarks: branch-prediction pressure is the cross-microarch variable.Closes ~10 % of the Pulley/WAMR wallclock gap on the iPhone 12 vtable suite (vtable_poly4 1.73× → 1.58×) and pushes WAMR within 1.16× on
xmrsplayeron Apple Watch SE2 — the closest cross-device result in our matrix.Dependency
Depends on #13445 landing first. Both PRs are stacked from the same fork branch; this PR's diff against
mainincludes #13445's 11 commits at the bottom + 13 fusion commits on top. The fusion is gated onis_eagerly_initialized_funcref_table(the predicate added in #13445), so it only fires when the table-mutability proof holds.Stack
13 commits on top of #13445:
band + brif + 2 xloadsat the call_indirect lazy-init tail (5 Pulley dispatches → 2 per call_indirect site).call_indirect{1,2,3,4}opcodes mirror direct-callcall{1,2,3,4}.Inst::IndirectCallbundles first 4 integer ABI args into the call opcode instead of synthesisingxmovs via regallocreg_fixed_use.sink_pure_instof the continuation-block loads broke the lazy-init slow path's rejoin; trapping fails closed under the predicate).Wallclock medians, N=10, phase-4 vs
table-mutability-trackingbaselineBENCH_TARGET_MS=2000;.utilityQoS on iOS /taskpolicy -bon M4.PMU buckets (single 12 s xctrace
CPU Countersper workload)Common bottleneck across A14 + M4 is Processing (back-end-bound). I can't measure these advacned CPU counters on iPhone XS (A12) or Apple Watch, I can only get wall clock time and power draw.
Verification
craneliftpulley_call_*integration testscargo fuzz run differential --no-default-featureswithALLOWED_ENGINES=pulley,wasmtime— 0 crashes / 0 divergencesExtra credit
Cross-device measurement harness I built for this bake-off across WASM runtimes: rebeckerspecialties/wasm-benchmark#1
I submitted two PRs to WAMR to support exceptions and loose SIMD, so that it can run more of the benchmarks and generally have functional partity without losing its performance edge.