Skip to content

perf: optimize to_csv_inner in datafusion-comet-spark-expr#4925

Draft
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/to_csv_inner-datafusion-comet-20260714-123452
Draft

perf: optimize to_csv_inner in datafusion-comet-spark-expr#4925
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/to_csv_inner-datafusion-comet-20260714-123452

Conversation

@andygrove

Copy link
Copy Markdown
Member

This PR was created by an LLM as a draft PR. I will mark it as ready for review after human review.

Which issue does this PR close?

N/A — autonomous exploratory PR.

Rationale for this change

Replaced up to six per-field full-string pattern searches and char-by-char escaping with one precomputed ASCII byte scan and bulk run copies, keeping the original searches as a fallback for non-ASCII/multi-char delimiters.

What changes are included in this PR?

Replaced up to six per-field full-string pattern searches and char-by-char escaping with one precomputed ASCII byte scan and bulk run copies, keeping the original searches as a fallback for non-ASCII/multi-char delimiters.

How are these changes tested?

Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).

Benchmark (criterion):

  • to_csv: 7.281% faster (base 858808ns -> cand 796276ns)

Full criterion output:

to_csv                  time:   [791.73 µs 793.59 µs 795.87 µs]
                        change: [−7.5817% −7.2812% −6.9754%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant