[SPARK-57982][SQL] Make exact percentile monotonic for large values

[Ma77Ball]

What changes were proposed in this pull request?

Rewrite the exact percentile linear interpolation in PercentileBase.getPercentile from (higher - position) * lower + (position - lower) * higher to the equivalent lower + (position - lower) * (higher - lower). This method is shared by percentile, percentile_cont, median, and their WITHIN GROUP forms.

Why are the changes needed?

The two forms are equal in exact arithmetic, but the old one scales both (potentially large) endpoint values by the weights independently, so rounding error on the order of the gap between adjacent doubles can make the result non-monotonic in the percentage:

val df = (0 until 20).map(i => 1e18 + i * 128.0).toDF("x")
df.selectExpr("percentile(x, 0.04)", "percentile(x, 0.05)").show(false)
// before: 1.00000000000000013E18, 1.0E18                    (p04 > p05, not monotonic)
// after:  1.00000000000000013E18, 1.00000000000000013E18

The new form scales only the small (higher - lower) delta, so the result stays monotonic.

Does this PR introduce any user-facing change?

Yes. Exact percentile / percentile_cont / median on large-magnitude inputs can change in the last ULP and are now monotonic in the percentage.

How was this patch tested?

Added UT in PercentileQuerySuite; existing percentile suites and the percentiles.sql golden file pass unchanged.

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored with Claude Opus 4.8

[Ma77Ball]

Hi @MaxGekk / @peter-toth, PTAL at this small percentile monotonicity fix (SPARK-57982) when you get a chance.

[Yicong-Huang]

cc @cloud-fan

[yadavay-amzn]

Nice fix, and the monotonicity argument holds up

I checked the numeric claim and it looks solid: the old (1-t)*lo + t*hi form is bounded but not monotonic, and switching to lo + t*(hi-lo) fixes that. One optional non-blocking thought inline. Thanks for tracking this down @Ma77Ball.

What I verified:

• Reproduced the described break: on the 1e18 + i*128.0 data the old form gives percentile(0.04)=1.0000000000000001e18 which is greater than percentile(0.05)=1e18, and across the 0..100 sweep the old form has 6 monotonicity violations while the new form has 0.
• The common path isn't regressed: 2012.3 is the exactly-rounded result for the courseSales case, so the golden update looks like an improvement rather than a regression.
• The new test does fail on master and pass with the change.
• percentile_cont / median / WITHIN GROUP and the frequency variant all route through this same getPercentile, so they're covered too.

[yadavay-amzn]

Optional, non-blocking. Computing higher - lower here can overflow to Infinity when the two adjacent keys straddle zero with magnitudes near Double.MaxValue (e.g. lower=-9e307, higher=9e307 makes the delta overflow, so p50 comes back as Infinity where the old form returned a finite value). It's an extremely unlikely input and the old form had its own accuracy issues in that range, so I don't think it should block anything. Might be worth a one-line comment noting the tradeoff, or nothing at all. Your call.

[Ma77Ball]

Hello @yadavay-amzn, thank you for the review. I've added a comment noting the tradeoff so it's flagged if the overflow ever surfaces for values near Double.MaxValue.

[peter-toth]

The fix looks correct and the shape is right -- keeping it in the shared getPercentile covers percentile / percentile_cont / median and both WITHIN GROUP forms in one line, and percentile_disc / percentile_approx are untouched since they return observed values rather than interpolating.

One thing the DataFrameAggregateSuite change surfaces: percentile(year, 0.3) on the Java group goes 2012.2999999999997 -> 2012.3, i.e. this isn't only a large-value change -- ordinary inputs shift in the last ULP too. Since exact percentile / percentile_cont / median can now return a value that differs from earlier releases, it might be worth a docs/sql-migration-guide.md entry to call out the behavior change. It's only ~1 ULP and the new value is the more-correct one, so IMO no legacy config is needed -- just a note.

IMO this reads more as a numerical-robustness improvement than a correctness bug fix: the old form was a standard, valid interpolation, results only move in the last ULP, and no documented monotonicity or exact-rounding guarantee is being broken. So shipping it in the next feature release without backporting to maintenance branches seems right.

Suggested migration-guide bullet, under ## Upgrading from Spark SQL 4.2 to 4.3:

• Since Spark 4.3, the exact percentile, percentile_cont, and median aggregate functions (including their WITHIN GROUP (ORDER BY ...) forms) compute the linear interpolation between two neighboring values as lower + fraction * (higher - lower) instead of (1 - fraction) * lower + fraction * higher. The two are equal in exact arithmetic, but the new form is monotonically non-decreasing in the requested percentage and avoids a rounding error the old form could introduce. As a result these functions may return a value that differs from earlier releases in the last ULP. percentile_disc and percentile_approx are unaffected.

[Ma77Ball]

Hello @peter-toth, thank you for the review. I've added the migration-guide bullet point as suggested under 4.2 to 4.3.

[peter-toth]

Hey @Ma77Ball, thanks for the fix and quick iteration.
Let me leave this PR open for one more day in case anyone else has any suggestions and then merge it tomorrow.

[yadavay-amzn]

LGTM, thanks for making this change

[peter-toth]

Merge Summary:

• merged into master 7353508
• merged into branch-4.x d2d3a75

Posted by merge_spark_pr.py

[peter-toth]

Thank you @Ma77Ball for the fix and congratulations to your first Apache Spark contribution!
Please let me know your JIRA id so that I can assign https://issues.apache.org/jira/browse/SPARK-57982 to you.

Thank you @yadavay-amzn for the review!

[Ma77Ball]

Thank you, @peter-toth and @yadavay-amzn, for reviewing and helping me finish my first contribution. I greatly appreciate your help.

JIRA ID: ma77ball