Skip to content

[Draft][Discussion] Support implicit partition#3475

Draft
litiliu wants to merge 2 commits into
apache:mainfrom
litiliu:codex/fluss-3435-implicit-partition
Draft

[Draft][Discussion] Support implicit partition#3475
litiliu wants to merge 2 commits into
apache:mainfrom
litiliu:codex/fluss-3435-implicit-partition

Conversation

@litiliu

@litiliu litiliu commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

This is a draft/discussion PR for #3435, based on the implicit partition FIP:
https://docs.google.com/document/d/1wDOkB-MuLrn3QYqL0fMlr1rj-lq891UbBcqXJSzCNcU/edit?usp=sharing

The FIP is still under community review, so this PR is intended to make the proposed implementation concrete and easier to discuss before finalizing the API/behavior.

What Changed

  • Introduce implicit partition metadata with partition keys backed by expressions, initially supporting date-trunc based partition transforms.
  • Support implicit partition routing for client writes, lookups, partition computation, and metadata refresh paths.
  • Preserve physical partition semantics while separating virtual partition spec keys from table schema fields.
  • Extend descriptor/table registration JSON serde and server-side validation for persisted partition expression metadata.
  • Add connector/lake conversion coverage where implicit partition metadata must be accepted or explicitly rejected.
  • Add unit and integration coverage for descriptor validation, serde, partition computation, client write/lookup, metadata refresh, and connector conversion paths.

Validation

  • git diff --check upstream-tracking...HEAD
  • mvn -pl fluss-common,fluss-client,fluss-server,fluss-flink/fluss-flink-common,fluss-lake/fluss-lake-iceberg,fluss-lake/fluss-lake-hudi,fluss-lake/fluss-lake-lance,fluss-lake/fluss-lake-paimon -DskipTests test-compile

Manual verification was also done with a debug program that created an implicit daily-partitioned primary-key table, wrote five rows for yesterday and five rows for today, queried the rows, and checked that the partition directories were created with the expected date-based names.

Notes

  • This PR intentionally remains draft while the FIP is being discussed.
  • Maintainers and reviewers should treat this as implementation material for API/behavior discussion, not as a final ready-to-merge change yet.

@litiliu

litiliu commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author
截屏2026-06-11 17 33 36 截屏2026-06-11 17 22 03

@litiliu litiliu force-pushed the codex/fluss-3435-implicit-partition branch from 895666d to 33d3ad5 Compare June 12, 2026 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant