Skip to content

perf: optimize might_contain_longs in datafusion-comet-spark-expr#4921

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/might_contain_longs-datafusion-comet-20260714-102146
Draft

perf: optimize might_contain_longs in datafusion-comet-spark-expr#4921
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/might_contain_longs-datafusion-comet-20260714-102146

Conversation

@andygrove

Copy link
Copy Markdown
Member

This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.

Which issue does this PR close?

N/A — autonomous exploratory PR.

Rationale for this change

Probe the bloom filter with an i64-specialized murmur3, a multiply-based remainder replacing the per-hash integer division, and a single hoisted version/modulus dispatch writing straight into a BooleanBuffer instead of collecting per-row Options.

What changes are included in this PR?

Probe the bloom filter with an i64-specialized murmur3, a multiply-based remainder replacing the per-hash integer division, and a single hoisted version/modulus dispatch writing straight into a BooleanBuffer instead of collecting per-row Options.

How are these changes tested?

Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).

Benchmark (criterion):

  • saturated: 17.362% faster (base 122667ns -> cand 101370ns)
  • sparse: 23.629% faster (base 78306ns -> cand 59803ns)

Full criterion output:

bloom_filter_might_contain/sparse
                        time:   [59.552 µs 59.748 µs 59.990 µs]
                        change: [−23.958% −23.629% −23.297%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 20 outliers among 100 measurements (20.00%)
  2 (2.00%) high mild
  18 (18.00%) high severe
bloom_filter_might_contain/saturated
                        time:   [101.25 µs 101.28 µs 101.32 µs]
                        change: [−17.440% −17.362% −17.279%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  8 (8.00%) high severe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant