How to add_files for BucketTransform partitioned data?

[dhallam]

Question

Versions:

pyiceberg == 0.11.0
Glue 5.1
Spark 3.5.6
Python 3.13
Amazon S3 (not S3Tables)

Question:

I have an iceberg table that is partitioned as

partitions=[  
    Partition("meta_received", DayTransform(), "received_day"),  
    Partition("meta_id", BucketTransform(5), "id_bucket"),  
    Partition("meta_digest", BucketTransform(5), "digest_bucket"),  
]

When writing to iceberg, if there is a failed commit, multiple parquet files are left in S3 under prefixes like

my_table/data/received_day=2026-01-22/id_bucket=2/digest_bucket=2/00000-19-9e0ba586-49c3-4f1e-b273-3db0c7fd0bda-0-00002.parquet

I want to add these files to the table to incorporate the data. I'd rather add the files in place to remove the need to move them, load them and insert them "manually" into the table.

Using

with table.transaction() as tx:  
    tx.add_files(  
        [            
	        "s3://my_bucket/my_table/data/received_day=2026-01-22/id_bucket=2/digest_bucket=2/00000-19-9e0ba586-49c3-4f1e-b273-3db0c7fd0bda-0-00002.parquet",
	        ...
        ],  
        check_duplicate_files=True,  
    )

I get

Cannot infer partition value from parquet metadata for a non-linear Partition Field: id_bucket with transform bucket[5]

because BucketTransform's preserve_order is False.

The partition info is present in the prefix.

Is there a way (or what is the best way) to commit these parquet files into the table?

Many thanks.

[jayceslesar]

Until concurrency is properly supported you are going to need to retry the commit. If the commit initially fails then those files are considered orphans as they exist in the bucket but are not referenced by the table.

[dhallam]

@jayceslesar A scenario we are looking to cover is where AWS Firehose (which commits to Iceberg via AWS Glue) fails to commit after its retries (for some reason, e.g. a blip in IAM or maybe an OCC conflict with a housekeeping task) - so ends up with failed commits at rest. So it's effectively that commit that we want to retry - but all we have are the files in parquet in partitioned prefixes, just not commited to the table - and we're trying to figure out the most efficient and effective way to do that.