Skip to content

Commit 7e18e1a

Browse files
committed
fix: force test set to full data in load_test_data
--sample_ratio is a train-only starvation knob; the test set is the fixed generalization ruler and must never be subsampled. Previously load_test_data inherited self.sample_ratio, which subsampled test too and made test losses incomparable across a sample_ratio sweep. Force 1.0 unconditionally. Standard runs use 1.0, so only sub-1.0 train-starved runs are affected.
1 parent bc52a15 commit 7e18e1a

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

strategies/tFL.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -510,8 +510,12 @@ def load_train_data(
510510
def load_test_data(
511511
self, sample_ratio: float = None, shuffle: bool = False
512512
) -> DataLoader:
513-
if sample_ratio is None:
514-
sample_ratio = getattr(self, "sample_ratio", 1.0)
513+
# The test set is the fixed generalization ruler and must ALWAYS be full.
514+
# --sample_ratio is a train-side starvation knob (e.g. the grokking probe);
515+
# subsampling test would make test losses incomparable across a sample_ratio
516+
# sweep. Force 1.0 regardless of the passed/instance value. Standard runs use
517+
# 1.0 anyway, so this only changes sub-1.0 train-starved runs.
518+
sample_ratio = 1.0
515519
testloader = self.load_data(
516520
file=self.test_file,
517521
sample_ratio=sample_ratio,

0 commit comments

Comments
 (0)