feat: CometNativeScan per-partition plan data, add DPP [iceberg]

[mbutrovich]

Which issue does this PR close?

Closes #3442

Rationale for this change

Dynamic Partition Pruning (DPP) is essential for optimizing star schema queries. Previously, native_datafusion scans would fall back to Spark when DPP was present. This PR enables full DPP support by deferring partition serialization until execution time, after DPP subqueries are resolved.

Additionally, this approach reduces serialization overhead when scanning large Parquet datasets, as partition metadata is no longer replicated across all Spark partitions.

What changes are included in this PR?

Architecture:

• CometNativeScanExec now uses lazy serializedPartitionData to defer serialization to execution time
• CometNativeScan.convert() creates a placeholder with only a scan_id at planning time
• serializePartitions() resolves DPP subqueries and serializes filtered partitions at execution
• Uses originalPlan.partitionFilters instead of partitionFilters because AQE's PlanDynamicPruningFilters converts subqueries to literals via makeCopy

SubqueryBroadcast Transformation:

• Transform SubqueryBroadcastExec children to use CometBroadcastExchangeExec wrapped in CometColumnarToRowExec for row-based output
• This enables broadcast exchange reuse via canonicalization - the same CometBroadcastExchangeExec used in the join can be reused by the DPP subquery filter
• Uses transformUp instead of transformUpWithSubqueries to preserve ReusedSubqueryExec object identity for scalar subqueries

Configuration:

• New flag spark.comet.scan.dpp.enabled (default: true) replaces spark.comet.dppFallback.enabled

Shims:

• Added getDppFilteredFilePartitions() and getDppFilteredBucketedFilePartitions() to ShimCometScanExec for Spark 3.4/3.5/4.0
• Added resolveSubqueryAdaptiveBroadcast() to ShimSubqueryBroadcast

Spark Diff Updates:

• Updated DynamicPartitionPruningSuite.checkPartitionPruningPredicate to recognize CometColumnarToRowExec → CometBroadcastExchangeExec as a valid SubqueryBroadcast child structure

Other:

• Removed custom equals/hashCode from CometNativeScanExec to prevent incorrect AQE exchange reuse between scans with different projections

How are these changes tested?

• New CometExecSuite tests for DPP with native_datafusion scans, multiple partition columns, non-broadcast subqueries (SubqueryExec), and subquery reuse (ReusedSubqueryExec)
• New test for DPP broadcast exchange reuse validation
• New test for SubqueryBroadcast transformation preserving ReusedSubqueryExec references
• New CometIcebergNativeSuite test for Iceberg DPP with non-broadcast joins
• Updated Spark 3.5.8 diff to handle Comet's SubqueryBroadcast structure in DPP validation and CometNativeScan in DPP validation

[andygrove]

Awesome work @mbutrovich! LGTM pending CI. Let's get this merged and keep iterating/testing.

[mbutrovich]

Closing. See #3510. I will open a smaller follow-up for the first task in a few minutes.