Commit 69ca97b
authored
Fix DELETED manifest entry snapshot_id in OverwriteFiles (#3237)
# Rationale for this change
When _OverwriteFiles._deleted_entries() creates DELETED manifest
entries, it now sets snapshot_id to the current (deleting) snapshot's ID
instead of retaining the original INSERT snapshot's ID.
Closes #3236
According to the [Iceberg spec (Manifest Entry
Fields)](https://iceberg.apache.org/spec/#manifest-entry-fields),
`snapshot_id` for a DELETED entry (status=2) should be the snapshot ID
in which the file was deleted. However,
`_OverwriteFiles._deleted_entries()` was copying the original entry's
`snapshot_id` (from the INSERT snapshot) into the new DELETED entry.
This caused downstream consumers that filter manifest entries by
`snapshot_id` (e.g. Iceberg Java's `IncrementalChangelogScan`) to
silently miss DELETED files, breaking CDC pipelines.
## Are these changes tested?
Added `test_manifest_entry_snapshot_id_after_partial_deletes` in
`tests/integration/test_deletes.py`.
## Are there any user-facing changes?
N/A
---------
Signed-off-by: Sotaro Hikita <bering1814@gmail.com>1 parent 2558dcd commit 69ca97b
2 files changed
Lines changed: 50 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
652 | 652 | | |
653 | 653 | | |
654 | 654 | | |
655 | | - | |
| 655 | + | |
656 | 656 | | |
657 | 657 | | |
658 | 658 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
975 | 975 | | |
976 | 976 | | |
977 | 977 | | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
0 commit comments