Skip to content

[core][spark] Support history-preserving replace table#7860

Open
Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:dev/replace-table
Open

[core][spark] Support history-preserving replace table#7860
Zouxxyy wants to merge 2 commits into
apache:masterfrom
Zouxxyy:dev/replace-table

Conversation

@Zouxxyy
Copy link
Copy Markdown
Contributor

@Zouxxyy Zouxxyy commented May 15, 2026

Purpose

Support Spark replace-table flows for Paimon tables while preserving table history where possible.

This change adds staged replace support for REPLACE TABLE and CREATE OR REPLACE TABLE ... AS SELECT, using truncate plus schema replacement for compatible FileStore tables so old snapshots remain available for time travel. It also falls back to drop-and-create when the source or target is not a Paimon table, handles SparkGenericCatalog provider checks, and documents the Spark SQL behavior.

Follow-up TODOs:

  • Support replace-table flows with RESTCatalog.
  • Support replacing a table with different partitioning. This is tied to the underlying truncate mechanism and likely needs a separate PR.

Tests

Added coverage for catalog-level replace table, Spark SQL replace table, create-or-replace-as-select, incompatible schema replacement, table existence errors, provider/table type fallback, and time-travel reads after replace.

@Zouxxyy Zouxxyy changed the title [spark] Support history-preserving replace table [core][spark] Support history-preserving replace table May 15, 2026
@Zouxxyy Zouxxyy force-pushed the dev/replace-table branch from f94eae7 to 1470f31 Compare May 15, 2026 05:05
@JingsongLi
Copy link
Copy Markdown
Contributor

JingsongLi commented May 15, 2026

We can implement REST Catalog in this PR too. This way we can have a global perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants