fix(generate): fail loudly on a headerless .wgt.csv source, don't misparse it (#194)

## Summary
- Reported ask was ".wgt.csv weighted source at the attribute level, not
just whole-entity" - turned out inverted: `<key source="x.wgt.csv">`
(attribute level) already applies weights correctly (verified
statistically: 79% vs 80% expected on a 3000-row sample). The real,
confirmed bug is the opposite end: `<generate source="x.wgt.csv">`
(entity level) silently misparses the headerless value|weight format as
a headered CSV, producing nonsense column names and fewer rows than the
file has.
- Full fix (wiring `WeightedEntityDataSource` into `<generate>`'s
chunked pagination) is bigger and riskier than it first looks:
`ChunkSourceReader`'s `loads_all` path (what a weighted source would
need) re-permutes its pool via `get_distributed_data`/`get_unique_data`
- reusing it would double-apply selection on top of already-weighted
draws, and a naive per-page rebuild risks silently duplicating rows
across multiprocess workers. That's real surgery on shared pagination
code, scoped as follow-up work, not this PR.
- This PR only fails loudly instead of silently producing garbage for
`.wgt.csv` at `<generate>`-level.
- `.wgt.ent.csv` (a normal headered CSV with an extra "weight" column)
is explicitly NOT touched - reading it plainly (no resampling, weight
column as literal data) is coherent and an existing test
(`test_source_script/test_iterate_source_scripted.xml`) already relies
on exactly that for an unrelated feature. Confirmed it still passes.

## Test plan
- [x] TDD: failing tests for both `.wgt.csv`-must-raise and
`.wgt.ent.csv`-must-still-read-plainly, confirmed RED/GREEN.
- [x] Existing `test_iterate_source_script` (which reads a
`.wgt.ent.csv`-named file for an unrelated reason) still passes.
- [x] Full suite: 1428 passed, 15 skipped.
- [x] mypy/ruff clean.

Author: ake2l

datamimic_ce/tasks/task_util.py

@@ -344,6 +344,20 @@ def gen_task_load_data_from_source_or_script(
             else:
                 # Evaluate script in source
                 source_data = context.evaluate_python_expression(stmt.script)
+        elif source_str.endswith(".wgt.csv"):
+            # A ".wgt.csv" file is headerless (value|weight, no column names) - the plain CSV
+            # reader below would treat its first data row as a header, producing nonsense column
+            # names and one fewer row than the file has. Unlike ".wgt.ent.csv" (a normal headered
+            # CSV that merely has an extra "weight" column - reading it plainly is coherent, just
+            # unweighted), there is no coherent plain-CSV reading of this format at all. <key
+            # source="...wgt.csv"> already applies its weights correctly; <generate>-level
+            # weighted-entity sourcing is real work, not yet done - fail loudly instead of
+            # silently generating garbage.
+            raise ValueError(
+                f"<generate> '{stmt.full_name}': source '{source_str}' is a headerless weighted "
+                f"value|weight file - not supported at <generate>-level (only <key source=...> "
+                f"applies '.wgt.csv' weights today)"
+            )
         # Load data from CSV
         elif source_str.endswith(".csv"):
             source_data = DataSourceRegistry.load_csv_file(

tests_ce/integration_tests/test_generate_wgt_source_guard/data/active.wgt.csv

@@ -0,0 +1,2 @@
+true|80
+false|20

tests_ce/integration_tests/test_generate_wgt_source_guard/data/products.wgt.ent.csv

@@ -0,0 +1,3 @@
+id,name,weight
+1,Cheap,80
+2,Expensive,20

tests_ce/integration_tests/test_generate_wgt_source_guard/generate_wgt_ent.xml

@@ -0,0 +1,3 @@
+<setup>
+    <generate name="rows" source="data/products.wgt.ent.csv" separator="," count="5" target="JSON"/>
+</setup>

tests_ce/integration_tests/test_generate_wgt_source_guard/generate_wgt_plain.xml

@@ -0,0 +1,3 @@
+<setup>
+    <generate name="rows" source="data/active.wgt.csv" count="5" target="JSON"/>
+</setup>

tests_ce/integration_tests/test_generate_wgt_source_guard/test_generate_wgt_source_guard.py

@@ -0,0 +1,40 @@
+# DATAMIMIC
+# Copyright (c) 2023-2025 Rapiddweller Asia Co., Ltd.
+# This software is licensed under the MIT License.
+# See LICENSE file for the full text of the license.
+# For questions and support, contact: info@rapiddweller.com
+
+"""<generate source="x.wgt.csv"> (the headerless value|weight format) falls through to the plain
+CSV reader, which has no coherent way to read it - it would treat the first data row as a header,
+producing nonsense column names and one fewer row than the file has. Full weighted-entity support
+at <generate>-level needs pagination-core work (see PR #194 discussion); until that lands, fail
+loudly instead of silently producing garbage.
+
+".wgt.ent.csv" is different: it's a normal headered CSV that happens to carry an extra "weight"
+column, so reading it plainly (ignoring the weight, no resampling) is coherent - just unweighted.
+An existing test (test_source_script/test_iterate_source_scripted.xml) already relies on exactly
+that, so this must keep working."""
+
+from pathlib import Path
+
+import pytest
+
+from datamimic_ce.data_mimic_test import DataMimicTest
+
+_dir = Path(__file__).resolve().parent
+
+
+def test_generate_source_wgt_csv_raises():
+    engine = DataMimicTest(_dir, "generate_wgt_plain.xml", capture_test_result=True)
+    with pytest.raises(ValueError, match=r"\.wgt\.csv"):
+        engine.test_with_timer()
+
+
+def test_generate_source_wgt_ent_csv_reads_as_plain_csv():
+    """Not weighted (no resampling, no bias toward the higher-weight row) - just an ordinary
+    headered CSV read, weight column included as literal data. Must NOT raise."""
+    engine = DataMimicTest(_dir, "generate_wgt_ent.xml", capture_test_result=True)
+    engine.test_with_timer()
+    rows = engine.capture_result()["rows"]
+    assert {row["name"] for row in rows} <= {"Cheap", "Expensive"}
+    assert all("weight" in row for row in rows)