Skip to content

Sharded dataset generation#73

Draft
PatrickRMiles wants to merge 6 commits into
LBANN:mainfrom
PatrickRMiles:miles30/sharded_sample_generation
Draft

Sharded dataset generation#73
PatrickRMiles wants to merge 6 commits into
LBANN:mainfrom
PatrickRMiles:miles30/sharded_sample_generation

Conversation

@PatrickRMiles
Copy link
Copy Markdown
Collaborator

Summary

Adds v3 physical sharded dataset generation and loading, with shard files written to match the configured DistConv layout. This lets large samples be generated directly as per-rank physical shards instead of always writing full-volume samples, and should allow us to generate scale 11 data.

Changes

  • Generate one set of voxelized fractal indices per logical volume, then render all requested physical shards from that shared data.
  • Write v3 dataset metadata containing the physical shard layout and shard-suffixed volume/mask files.
  • Load v3 physical shard files directly when the dataset layout matches the requested DistConv layout.
  • Preserve v2/full-volume dataset loading for unsharded cache reuse and runtime slicing.
  • Canonicalize v3 cache IDs for equivalent shard layouts so reordered dc_num_shards / dc_shard_dims pairs reuse the same dataset cache.

Validation

  • Ran sharded dataset equivalence comparisons across problem scales 6, 7, and 8 for layouts 1x1x1, 1x1x2, 1x2x2, and 1x1x4; all 18 pairwise comparisons passed.

@PatrickRMiles PatrickRMiles marked this pull request as draft May 28, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant