feat: write-path optimizations, parallel ingest, schema cache + benchmarks#7
Open
adubovikov wants to merge 9 commits into
Open
feat: write-path optimizations, parallel ingest, schema cache + benchmarks#7adubovikov wants to merge 9 commits into
adubovikov wants to merge 9 commits into
Conversation
Split schema coordination from parallel append, add write-path settings, layout caching, and micro-batch coalescing for raw_ingest_file. Includes benchmark harnesses and A/B script vs official release (~16% faster write on 50k rows vs v0.0.2).
run_vs_release_stats.sh runs paired rounds with median/p95 write and read deltas vs v0.0.2; shared helpers live in ab_bench_lib.sh.
raw_ingest_file now uses RawStreamIngestor with auto-scaled consumer threads (rawduck_pipeline_consumers) for parallel pool submit while schema evolution stays serial. Adds PrValidation workflow with make test and a 3-round A/B benchmark gate (median write must not regress >10% vs release).
feat: optimize native write path with tunable pool and pipeline
Skip InferSchema+FlattenSchema on absorbed shapes by caching the schema tree (RawCachedSchema) in ObjectCache keyed by shape hash. Parser threads in raw_ingest_file reuse the cached schema after the first batch is absorbed, eliminating per-batch inference overhead. Extraction hot-path improvements: - Dense column_idx on RawNode replaces unordered_map<RawNode*,idx_t> lookup in RawExtractor::Traverse - Zero-alloc FindChild (memcmp linear scan) replaces child_lookup.find(string(key,len)) — removes one std::string allocation per JSON key per row - MergeValueInternal uses FindChild for existing children, only allocating a string when creating a new child - memset in RawExtractor::Reset replaces vector::assign Benchmark results (50k rows, 6 cols, 10-round best-of-N): - Write: -10% vs v0.0.2 release (median 110→105 ms) - Read: -13% vs v0.0.2 release (median 20.5→17.5 ms) At scale (500k rows), RawDuck writes 17% faster than opaque JSON/VARCHAR storage while reading 2.3-3.4x faster and using 2.3x less disk.
feat: schema cache + extraction hot-path optimizations
Reformat ProcessWithSchema signatures and the ingest parser lambda per duckdb clang-format 11 rules so the upstream Format Check passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the fork's
mainup to upstream, bundling several merged feature lines:raw_ingest.cpp,raw_records.cpp).RawParsedPayload::ProcessWithSchemafast path that skips re-inference across batches).raw_json.cpp/.hpp,raw_functions.hpp.PrValidation.yml,scripts/(A/B vs release, stats, CI benchmark gate) andagent_planning/bench/helpers.Notes
*.cpp/*.hppfiles verified clean against duckdb's clang-format 11 config.agent_planning/contains research/benchmark artifacts (not shipped in the extension build).Test plan
clang-format --style=fileclean on every changed C/C++ filemake test