Skip zero-fill of hash buffer by writing null sentinels#21347
Draft
Dandandan wants to merge 1 commit intoapache:mainfrom
Draft
Skip zero-fill of hash buffer by writing null sentinels#21347Dandandan wants to merge 1 commit intoapache:mainfrom
Dandandan wants to merge 1 commit intoapache:mainfrom
Conversation
Hash functions now write all positions including nulls (using a consistent null sentinel hash) when rehash=false (first column). This allows with_hashes to skip the buffer zero-fill, saving ~0.5µs per 8192-element batch on the no-nulls hot path. Changes: - with_hashes: use unsafe set_len instead of resize(n, 0) - hash_array_primitive/hash_array: fill with null sentinel then overwrite valid positions via valid_indices() - hash_string_view_array_inner: write null sentinel instead of skip - hash_dictionary_inner: write null sentinel for null keys/values - hash_run_array_inner: fill null run ranges with sentinel - create_hashes: zero-fill only for complex types (struct, list, map, union) whose hash functions always combine with existing values Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
run benchmark with_hashes |
|
🤖 Criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing skip-zero-fill-hash-buffer (7056098) to 1e93a67 (merge-base) diff File an issue against this benchmark runner |
|
🤖 Criterion benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A - performance optimization
Rationale for this change
Profiling ClickBench queries showed
__bzero(frombuffer.resize(n, 0)) as ~1-2% of CPU time inwith_hashes/create_hashes. The zero-fill is unnecessary when hash functions write all positions including nulls.What changes are included in this PR?
Hash functions now write all buffer positions when
rehash=false(first column), using a consistent null sentinel hash (random_state.hash_one(1u8)) for null positions. This allowswith_hashesto skip the zero-fill entirely.with_hashes: useset_leninstead ofresize(n, 0)— avoids memsethash_array_primitive/hash_array: fill with null sentinel, then overwrite valid positions viavalid_indices()hash_string_view_array_inner: write null sentinel instead ofcontinuefor null positionshash_dictionary_inner: write null sentinel for null keys/valueshash_run_array_inner: fill null run ranges with sentinelcreate_hashes: zero-fill only for complex types (struct, list, map, union) whose hash functions always combine with existing valuesBenchmark results (
with_hashesbench, int64 single column):Are these changes tested?
Existing tests updated to expect non-zero null sentinel hash values.
Are there any user-facing changes?
No. Null positions now get a consistent non-zero hash instead of 0, but this is an internal implementation detail.
🤖 Generated with Claude Code