Sharded dataset generation#73

Open

PatrickRMiles wants to merge 12 commits into

PatrickRMiles:miles30/sharded_sample_generation

Commits on Jun 5, 2026

fix fractalgen bug: computing acceptance criteria variance across wrong axis
Patrick Miles
committed
pass absolute fractal dir path to scaffold calls to ensure existing fractals found
Patrick Miles
committed
revert more complex changes, focus on simpler implementation: dataset sharding matches distconv spec
Patrick Miles
committed
sharded volgen WIP
Patrick Miles
committed
normalize shard layout in dataset config hashing so equivalent shard layouts hash the same way
Patrick Miles
committed
small tweaks to shrink diff
Patrick Miles
committed
ruff
Patrick Miles
committed
add docstrings
Patrick Miles
committed

Commits on Jun 10, 2026

update README
Patrick Miles
committed
volumegen: every rank ensures needed dirs exist
Patrick Miles
committed

Commits on Jun 11, 2026

rework sharded volgen
Patrick Miles
committed

Commits on Jun 25, 2026

ruff
Patrick Miles
committed