[SPARK-43413][SQL] Support QUALIFY clause#55074
Open
azmatsiddique wants to merge 11 commits intoapache:masterfrom
Open
[SPARK-43413][SQL] Support QUALIFY clause#55074azmatsiddique wants to merge 11 commits intoapache:masterfrom
azmatsiddique wants to merge 11 commits intoapache:masterfrom
Conversation
…n last CSV column ### What changes were proposed in this pull request? This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (`""`) when the `escape` option is set to an empty string (`""`). Previously, mid-line empty quoted strings correctly resolved to null/empty, but the last column resolved to a literal `"` character due to univocity parser behavior. ### Why are the changes needed? To ensure consistent parsing of CSV data regardless of column position. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations. ### How was this patch tested? Added a new regression test in `CSVSuite` that verifies consistent parsing of both mid-line and end-of-line empty quoted fields.
…INTERNAL_ERROR_BROADCAST
…or-conditions.json
# Conflicts: # sql/connect/client/jdbc/src/test/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaDataSuite.scala
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What: Implements QUALIFY clause for filtering window functions in Spark SQL.
Why: Eliminates the need for subqueries when filtering by window results, bringing Spark SQL closer to parity with engines like BigQuery and Snowflake.
User-Facing: Adds the QUALIFY clause (and |> QUALIFY for pipe operators).
Testing: Added qualify.sql to SQLQueryTestSuite with full coverage of positive and negative cases.