[Python] Scanning batch size is limited to 65536 (2**16).

[asfimport]

Scanning batches is documented to default to a batch size of 1,000,000. But the behavior is that batch size defaults to - and is limited to - 65536.

In []: dataset.count_rows()
Out[]: 538038292

In []: next(dataset.to_batches()).num_rows
Out[]: 65536

In []: next(dataset.to_batches(batch_size=10**6)).num_rows
Out[]: 65536

In []: next(dataset.to_batches(batch_size=10**4)).num_rows
Out[]: 10000

 

 

Environment: macOS
Reporter: A. Coady / @coady

_{Note: This issue was originally created as ARROW-16015. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
The batch size can split up larger row groups to fit small batch sizes but it won't, at the moment, merge together small row groups to fit large batch sizes. Performance-wise this tends to be expensive (you'd need to allocate space big enough for both and then copy the data) but I can see how it might be useful is some scenarios.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
We can maybe clarify in the documentation of batch_size that the actual upper limit might depend on your files (such as row group size in Parquet).

I suppose this is also the same for the record batch size in IPC / Feather files? (or stripe size in ORC files)

[asfimport]

Weston Pace / @westonpace:
Yes. Clarification would be good. We could even change the parameter name to max_batch_size. That is what we did in the table source node. Also, a min_batch_size is still useful I think, though we would just want to document that it has performance implications.

You are correct that this would apply to IPC & ORC. This even applies to CSV because the batch size is independent of the block size (which is specified in the CSV read options).

[thisisnic]

Closing this as the batch size is no longer capped at 65536. With batch_size=1M it now returns all 200K rows in one batch, and the default batch size is 131072 (2^17) instead of the old 65536.