[SPARK-39660][SQL] Support v2 DESCRIBE TABLE .. PARTITION#55064
Open
viirya wants to merge 5 commits intoapache:masterfrom
Open
[SPARK-39660][SQL] Support v2 DESCRIBE TABLE .. PARTITION#55064viirya wants to merge 5 commits intoapache:masterfrom
viirya wants to merge 5 commits intoapache:masterfrom
Conversation
viirya
force-pushed
the
SPARK-39660-describe-table-partition-v2
branch
2 times, most recently
from
e2f7bc6 to
9152840
Compare
## What changes were proposed in this pull request? Enable `DESCRIBE [EXTENDED] TABLE <v2_table> PARTITION (col=val, ...)` for V2 tables that implement `SupportsPartitionManagement`, bringing parity with the V1/Hive behavior. Previously any partition spec on a V2 table threw unconditionally: "DESCRIBE does not support partition for v2 tables." ### Changes: - `DataSourceV2Strategy`: replaced blanket error throw with a capability check (`SupportsPartitionManagement`); passes `partitionSpec` through to `DescribeTableExec`. - `DescribeTableExec`: added `partitionSpec` parameter and new `addPartitionDetails()` method that normalizes the spec, converts it to a typed `InternalRow`, validates partition existence via `loadPartitionMetadata()`, and (for EXTENDED) appends a `# Detailed Partition Information` section. - `QueryCompilationErrors` / `error-conditions.json`: replaced legacy `_LEGACY_ERROR_TEMP_1111` with named error `DESCRIBE_PARTITION_NOT_SUPPORTED_FOR_V2_TABLE`. ## How was this patch tested? Replaced the single "not supported" test in `v2/DescribeTableSuite` with four targeted tests: partition validation, extended partition info, non- existent partition error, and capability error for tables without partition management. All 29 V2 + 66 V1 `DescribeTableSuite` tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…BE TABLE PARTITION Use Option[PartitionSpec] in DescribeRelation and extend V2PartitionCommand so ResolvePartitionSpec handles UnresolvedPartitionSpec -> ResolvedPartitionSpec resolution automatically during analysis, consistent with other partition commands. Move the non-partition-management error check from DataSourceV2Strategy into CheckAnalysis so it is raised as an AnalysisException at analysis time. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
viirya
force-pushed
the
SPARK-39660-describe-table-partition-v2
branch
from
9152840 to
01a7d46
Compare
Fix alphabetical ordering of DESCRIBE_PARTITION_NOT_SUPPORTED_FOR_V2_TABLE in error-conditions.json (must come after DESCRIBE_JSON_NOT_EXTENDED). Update describe.sql analyzer golden file to reflect new UnresolvedPartitionSpec toString format after DescribeRelation.partitionSpec changed from TablePartitionSpec to Option[PartitionSpec]. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…] renders with double brackets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s from DescribeRelation arg Option[PartitionSpec] renders without surrounding brackets in the plan string, unlike the previous Map[String, String] which rendered as [k=v, ...]. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| EXPLAIN DESCRIBE t PARTITION (c='Us', d=2) | ||
| -- !query analysis | ||
| ExplainCommand 'DescribeRelation [c=Us, d=2], false, [col_name#x, data_type#x, comment#x], SimpleMode | ||
| ExplainCommand 'DescribeRelation unresolvedpartitionspec((c,Us), (d,2), None), false, [col_name#x, data_type#x, comment#x], SimpleMode |
Member
There was a problem hiding this comment.
At the first glance, this looks a little strange. However, since this is EXPLAIN on DESCRIBE statement, this unresolvedpartitionspec seems fine to me, too.
dongjoon-hyun
approved these changes
Mar 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Enable
DESCRIBE [EXTENDED] TABLE <v2_table> PARTITION (col=val, ...)for V2 tables that implementSupportsPartitionManagement, bringing parity with the V1/Hive behavior.Previously any partition spec on a V2 table threw unconditionally:
"DESCRIBE does not support partition for v2 tables."
Changes:
DataSourceV2Strategy: replaced blanket error throw with a capability check (SupportsPartitionManagement); passespartitionSpecthrough toDescribeTableExec.DescribeTableExec: addedpartitionSpecparameter and newaddPartitionDetails()method that normalizes the spec, converts it to a typedInternalRow, validates partition existence vialoadPartitionMetadata(), and (for EXTENDED) appends a# Detailed Partition Informationsection.QueryCompilationErrors/error-conditions.json: replaced legacy_LEGACY_ERROR_TEMP_1111with named errorDESCRIBE_PARTITION_NOT_SUPPORTED_FOR_V2_TABLE.DESCRIBE TABLE PARTITION: UseOption[PartitionSpec]inDescribeRelationand extendV2PartitionCommandsoResolvePartitionSpechandlesUnresolvedPartitionSpec->ResolvedPartitionSpecresolution automatically during analysis, consistent with other partition commands.Why are the changes needed?
DESCRIBE TABLE .. PARTITIONwas not supported for V2 tables. This adds the missing support by integrating with the existing V2 partition infrastructure:DescribeRelationnow extendsV2PartitionCommandso partition specs are resolved at analysis time throughResolvePartitionSpec, consistent with how other partition commands (e.g., SHOW PARTITIONS, ALTER TABLE .. DROP PARTITION) work.Does this PR introduce any user-facing change?
Yes.
DESCRIBE TABLE .. PARTITIONsupports V2 table after this PR.How was this patch tested?
Replaced the single "not supported" test in
v2/DescribeTableSuitewith four targeted tests: partition validation, extended partition info, non-existent partition error, and capability error for tables without partition management. All 29 V2 + 66 V1DescribeTableSuitetests pass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6