feat: Add direct native shuffle execution optimization [experimental]

[andygrove]

Summary

This PR introduces an experimental optimization that allows the native shuffle writer to directly execute the child native plan instead of reading intermediate batches via JNI. This avoids the JNI round-trip for single-source native plans.

Current flow:

Native Plan → ColumnarBatch → JNI → ScanExec → ShuffleWriterExec

Optimized flow:

Native Plan → (directly in native) → ShuffleWriterExec

Configuration

The optimization is controlled by a new config option:

• spark.comet.exec.shuffle.directNative.enabled (default: false)

Scope

The optimization currently applies when:

• Native shuffle mode is enabled (spark.comet.shuffle.mode=native)
• The shuffle's child is a single-source native plan with CometNativeScanExec
• The partitioning is not RangePartitioning (which requires sampling)

Changes

File	Change
CometShuffleDependency.scala	Added childNativePlan field to pass native plan to writer
CometShuffleExchangeExec.scala	Added detection logic for single-source native plans
CometShuffleManager.scala	Pass native plan to shuffle writer
CometNativeShuffleWriter.scala	Use child native plan directly when available
CometConf.scala	Added COMET_SHUFFLE_DIRECT_NATIVE_ENABLED config option
CometDirectNativeShuffleSuite.scala	Comprehensive test suite with 15 tests

Test plan

• Added CometDirectNativeShuffleSuite with 15 tests covering:
  • Basic optimization with single scan source
  • Filter/project operators
  • Single partition and multi-partition cases
  • Config enable/disable behavior
  • Range partitioning fallback
  • JVM shuffle mode fallback
  • Various data types
  • Edge cases (empty tables, filtered results)
  • Correctness comparison between optimized and non-optimized paths
• Verified existing CometNativeShuffleSuite tests still pass (16/16)

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 63.43284% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.96%. Comparing base (f09f8af) to head (6132098).
⚠️ Report is 863 commits behind head on main.

Files with missing lines	Patch %	Lines
...t/execution/shuffle/CometShuffleExchangeExec.scala	20.51%	27 Missing and 4 partials ⚠️
...t/execution/shuffle/CometNativeShuffleWriter.scala	79.06%	9 Missing and 9 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3230      +/-   ##
============================================
+ Coverage     56.12%   59.96%   +3.83%     
- Complexity      976     1433     +457     
============================================
  Files           119      170      +51     
  Lines         11743    15819    +4076     
  Branches       2251     2616     +365     
============================================
+ Hits           6591     9486    +2895     
- Misses         4012     5008     +996     
- Partials       1140     1325     +185     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

@viirya I'm curious what you think about this idea. Does it make sense?

[viirya]

Looks like CometNativeShuffleWriter was added after, I am not familiar with its design. I took a quick look at the code. So it is basically a ShuffleWriter and executes a native plan to write inputs from an iterator of ColumnarBatch into shuffle file. Although it executes a native plan to write, its input is still coming from JVM even its input plan maybe a native plan too.

So this idea is to avoid the JVM/JNI overhead on the input. It sounds good.

[andygrove]

@sqlbenchmark run tpch --iterations 3

[andygrove]

Looks like CometNativeShuffleWriter was added after, I am not familiar with its design. I took a quick look at the code. So it is basically a ShuffleWriter and executes a native plan to write inputs from an iterator of ColumnarBatch into shuffle file. Although it executes a native plan to write, its input is still coming from JVM even its input plan maybe a native plan too.

So this idea is to avoid the JVM/JNI overhead on the input. It sounds good.

Thanks for taking a look! I will keep experimenting with this.

[sqlbenchmark]

Comet TPC-H Benchmark Results

Baseline: main (b0477a72)
PR: 191e5194 - Merge apache/main into direct-native-shuffle-execution
Scale Factor: SF100
Iterations: 3

Query Times

Query	Baseline Avg (s)	Baseline Best (s)	PR Avg (s)	PR Best (s)	Change (Avg)	Change (Best)
Q1	9.92	9.38	9.78	9.19	⚪ -1.5%	⚪ -2.1%
Q2	5.20	4.82	5.25	4.96	⚪ +0.9%	⚪ +2.7%
Q3	9.77	9.58	9.88	9.75	⚪ +1.1%	⚪ +1.8%
Q4	10.95	10.78	11.10	10.86	⚪ +1.4%	⚪ +0.7%
Q5	18.81	18.67	18.91	18.72	⚪ +0.5%	⚪ +0.2%
Q6	2.40	2.37	2.44	2.36	⚪ +1.5%	⚪ -0.1%
Q7	12.22	11.92	12.37	12.09	⚪ +1.2%	⚪ +1.4%
Q8	24.70	24.30	24.88	24.34	⚪ +0.7%	⚪ +0.2%
Q9	39.62	38.06	39.49	37.92	⚪ -0.3%	⚪ -0.4%
Q10	10.62	10.51	10.58	10.27	⚪ -0.3%	⚪ -2.3%
Q11	4.39	4.30	4.41	4.32	⚪ +0.5%	⚪ +0.4%
Q12	7.04	6.98	7.13	7.06	⚪ +1.3%	⚪ +1.1%
Q13	7.21	6.81	7.29	6.90	⚪ +1.0%	⚪ +1.5%
Q14	3.56	3.50	3.70	3.58	⚪ +4.0%	⚪ +2.2%
Q15	7.14	7.07	7.29	7.28	⚪ +2.1%	⚪ +2.9%
Q16	4.28	4.17	4.42	4.12	⚪ +3.2%	⚪ -1.1%
Q17	32.56	32.15	32.74	32.28	⚪ +0.6%	⚪ +0.4%
Q18	33.68	33.33	33.79	33.53	⚪ +0.3%	⚪ +0.6%
Q19	6.94	6.72	6.80	6.55	⚪ -2.0%	⚪ -2.5%
Q20	6.65	6.58	6.70	6.66	⚪ +0.8%	⚪ +1.3%
Q21	45.62	45.22	46.19	45.70	⚪ +1.3%	⚪ +1.1%
Q22	4.90	4.85	4.98	4.94	⚪ +1.6%	⚪ +2.0%
Total	308.17	302.05	310.11	303.38	⚪ +0.6%	⚪ +0.4%

Spark Configuration

Setting	Value
Spark Master	local[*]
Driver Memory	32G
Driver Cores	8
Executor Memory	32G
Executor Cores	8
Off-Heap Enabled	true
Off-Heap Size	24g
Shuffle Manager	CometShuffleManager
Comet Replace SMJ	true

---

Automated benchmark run by dfbench