[SPARK-57583][SQL] Support lazy dictionary decoding for TIME in the vectorized Parquet reader#57239
Open
yadavay-amzn wants to merge 1 commit into
Open
[SPARK-57583][SQL] Support lazy dictionary decoding for TIME in the vectorized Parquet reader#57239yadavay-amzn wants to merge 1 commit into
yadavay-amzn wants to merge 1 commit into
Conversation
…ectorized Parquet reader
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is the "option B" follow-up to SPARK-57551, under the TIME data type epic (SPARK-57550). It re-enables lazy dictionary decoding for TIME columns in the vectorized Parquet reader by making the lazy path precision/unit-aware, rather than disabling it.
ParquetDictionary.decodeToLongnow applies the TIME transform on the lazy path:micros -> nanos(when the column is stored as micros) followed byDateTimeUtils.truncateTimeToPrecision, producing values byte-for-byte identical to the eagerTimeVectorUpdater.ParquetDictionarycarries the on-disk unit and the requested precision (set at construction inVectorizedColumnReaderwhen the column is aTimeType).TimeTypeParquetOps.supportsLazyDictionaryDecodingnow returnstrue, so TIME columns use the lazy optimization again through the types framework's dispatch.The change is scoped to TIME: non-TIME dictionary reinterpretations and the other INT64 transforms (
TIMESTAMP_MILLISupcast, timestamp rebase) remain eager, as before.Why are the changes needed?
SPARK-57551 added read-time precision truncation for TIME columns and, to guarantee truncation applies, disabled lazy dictionary decoding for
TIME(NANOS)(option A) so dictionary-encoded TIME columns eager-decoded through the truncating updater. This PR restores the lazy dictionary-decoding optimization (option B) without losing that correctness, by teaching the lazy path the same unit conversion and precision truncation.Does this PR introduce any user-facing change?
No result change. It is a performance improvement: dictionary-encoded TIME columns can again be read via the lazy dictionary path.
How was this patch tested?
New
ParquetIOSuitetests exercising the lazy path (dictionary-encoded + vectorized reader + lazy decoding enabled):TIME(NANOS)andTIME(MICROS)columns read at a lower precision produce truncated values identical to the eager path; a lazy-vs-eager parity test; a nullable multi-value case; and a full-precision (no-truncation) case.TimeTypeParquetOpsSuiteupdated for the re-enabled lazy support. checkstyle + scalastyle clean.Was this patch authored or co-authored using generative AI tooling?
Authored with assistance by Claude Opus 4.8.