Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed

- Per-zone stats from current Rust writers (`vortex.zoned`, vortex-jni 0.76.0) decode again. The 0.76 zoned layout replaced the legacy `vortex.stats` bit-set metadata with an aggregate-function spec list and dropped the per-stat truncation flags; the reader now reconstructs the stats table from that spec list (min/max/sum/null_count), so `columnZoneStats` and aggregate push-down work against those files instead of throwing `ClassCastException`. ([#197](https://github.com/dfa1/vortex-java/pull/197))
- Scans of files whose columns use different chunk grids no longer fail with `mixed per-column chunking beyond 1-vs-N is not supported`. The scan planner now splits at the merged boundary grid — the sorted union of every column's chunk boundaries, matching the Rust reference — and decodes each column's covering chunk once, slicing it zero-copy to each window. This handles both nested grids (Raincloud `emotions-dataset-for-nlp`, where `label`'s coarse chunks nest inside `text`'s finer grid) and disjoint grids (`uci-beijing-multi-site-air-quality`, where numeric and `station` boundaries do not nest). Aligned N-vs-N and 1-vs-N scans keep their existing slice-free fast path. ([#221](https://github.com/dfa1/vortex-java/issues/221))
- CSV export renders nested struct columns as JSON object cells (`{"field":value,...}`, strings JSON-escaped, null fields as JSON `null`, nested structs recursed) instead of throwing `unsupported array type: StructArray`. ([#217](https://github.com/dfa1/vortex-java/issues/217))
- Scanning a struct whose columns include a nested struct no longer fails chunk planning. A nested `vortex.struct` layout column (e.g. Raincloud `countries-of-the-world`'s `data` column) is now treated as a single full-range chunk source and decoded through the layout registry's new struct decoder into a `StructArray`, matching the Rust reference (a nested struct spans its parent's row range, not an independent chunking). `schema`/`count`/`inspect` already worked; plain scans and `select` of sibling columns now work too. CSV export of the struct column itself remains unsupported (a separate rendering limitation). ([#207](https://github.com/dfa1/vortex-java/issues/207))
- CSV export renders unsigned integer columns (U8–U64) with their unsigned values — high-half values previously printed as two's-complement negatives (uci-wine `magnesium` U8 132 exported as -124), silent corruption found by the Raincloud conformance suite. ([#208](https://github.com/dfa1/vortex-java/issues/208))
Expand Down
8 changes: 5 additions & 3 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,11 @@ Per-slug status lives in `integration/src/test/resources/raincloud/expected-stat
(`ok` must pass; `gap:<issue>` must still fail, so a fix flips the entry in the same change;
`untriaged` runs and reports without failing the build). A scheduled workflow
(`raincloud-conformance.yml`) hydrates a size-capped subset weekly. Current triage —
28 `ok`, 2 known gaps (both misaligned per-column chunk grids,
[#221](https://github.com/dfa1/vortex-java/issues/221)); 217 slugs untriaged. Fixed so far
by this suite: lazy dict U8/U16 values ([#206](https://github.com/dfa1/vortex-java/issues/206)),
30 `ok`, 0 known gaps; 217 slugs untriaged. Fixed so far
by this suite: misaligned per-column chunk grids
([#221](https://github.com/dfa1/vortex-java/issues/221) — scan now splits at the merged boundary
grid, slicing each column's covering chunk per window), lazy dict U8/U16 values
([#206](https://github.com/dfa1/vortex-java/issues/206)),
nested struct columns in scan ([#207](https://github.com/dfa1/vortex-java/issues/207)),
unsigned integers rendered signed ([#208](https://github.com/dfa1/vortex-java/issues/208) /
[#216](https://github.com/dfa1/vortex-java/issues/216) — silent corruption, incl. wrong filter
Expand Down
4 changes: 2 additions & 2 deletions integration/src/test/resources/raincloud/expected-status.csv
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ diabetes-health-indicators-dataset,untriaged
disease-symptom-description-dataset,untriaged
docmatix-zero-shot,untriaged
electric-motor-temperature,untriaged
emotions-dataset-for-nlp,gap:221
emotions-dataset-for-nlp,ok
fhv_tripdata_2025,untriaged
fhvhv_tripdata_2025,untriaged
finemath-4plus,untriaged
Expand Down Expand Up @@ -193,7 +193,7 @@ uci-air-quality,ok
uci-auto-mpg,ok
uci-automobile,untriaged
uci-bank-marketing,ok
uci-beijing-multi-site-air-quality,gap:221
uci-beijing-multi-site-air-quality,ok
uci-bike-sharing-dataset,ok
uci-breast-cancer,untriaged
uci-breast-cancer-wisconsin-diagnostic,untriaged
Expand Down
Loading
Loading