Enhancement to generators selection layer and new evaluation layer

[yhong123]

Objective

Improve the existing rule-based synthetic data generation system by introducing:

• keyword-based pruning and lightweight uncertainty-aware ranking for generator selection
• minimal quantitative evaluation layer

without modifying the core generator implementations.

──────────────────────────────────────────────
│ 4. Governance & Human Review (IG / Clinicians)│
──────────────────────────────────────────────
│ 3. Evaluation Layer (Fidelity / Utility / Privacy metrics)│ - evaluate
──────────────────────────────────────────────
│ 2. Synthetic Data Generation Layer │
──────────────────────────────────────────────
│ 1. Schema creating + Generator Selection Layer │ - configure-generators
──────────────────────────────────────────────

Enhancement to generators selection layer

To improve the propose command by

• adding keyword-based pruning method (as suggested by Tim in sqlsynthgen alan-turing-institute/DataMatryoshka#37)
• normalising heterogeneous raw fit scores into a probabilistic ranking and reducing output length using entropy-based truncation

without modifying generator logic or fit computation.

New evaluation layer

To add a new evaluation layer that measures how closely the synthetic data matches the real data, how useful it is for analysis, and whether there are any privacy risks. To support this, we will add a new command, evaluate.
Minimal metrics:

• fidelity metrics (how well the distributions match the original data, e.g. KL divergence, correlation matrix distance);
  • column-level fidelity evaluates each column independently
  • table-level fidelity examines relationships and dependencies within a table
  • dataset-level fidelity evaluates relationships across tables within the dataset
• dataset-level utility metric (whether the data preserves analytical usefulness for downstream tasks, e.g. downstream ML performance, accuracy drop, calibration shift)
• dataset-level privacy metric (whether there is any risk of revealing sensitive or unique records, e.g. nearest-neighbour distance ratio)

The goal is not to replace human oversight, but to make it more informed, consistent, and auditable. The user can use the fidelity metrics to tune generators and regenerate data, while the utility and privacy metrics are likely to be of greater interest to IG people.

[yhong123]

Following the discussion with Tim, the post-generation evaluation layer will be deferred to future work. For now, I will focus on improving the propose mechanism within configure-generators.

[yhong123]

Enhancement to generators selection layer

Discussion: #129

Tasks:

• Minor bug in class Bucket. In /proposers/base.py, class Buckets::__init__(), ((func.floor(column) - bottom) / width).label("b") throws away every non-integer result, calculates incorrect bucket index, ignores many rows and produces a different histogram. Modify to func.floor((column - bottom) / width).label("b") to calculate the bucket index and group rows into histogram buckets.
• New /evaluators/ module. The propose command recommends candidate generators for individual columns. Each candidate generator is assigned a fit score that quantifies how closely the distribution of the generated synthetic data matches the distribution of the corresponding real data. In the current implementation, each generator computes its fit score independently, often using a different feature representation, probability space, or evaluation metric. Consequently, the resulting scores are not directly comparable, making it difficult to rank generators consistently. The new /evaluators/ module introduces a unified evaluation framework that standardises the evaluation pipeline while remaining extensible. The initial implementation includes the following components:
  • FeatureExtractor – extracts comparable features from real and synthetic column data.
  • DistributionBuilder – constructs probability distributions from the extracted features.
  • Metric – defines the distance or similarity metric used to compare distributions.
  • ColumnEvaluator – orchestrates the evaluation pipeline and computes a standardised fit score for a single column.
• /evaluators/ module. Additional implementation:
  • ColumnStatistics
  • EvaluationPipeline
  • NoveltyMetric, DiversityMetric
  • EvaluationPipeline for special column type such as date interval
  • weighted-sum ranking
• /evaluators/ module. Add:
  • semantic fidelity evaluation dimension alongside the statistical fidelity, novelty and diversity dimensions (this item is currently deferred due to its complexity but may be reconsidered in the future)
  • multi-objective Pareto ranking in addition to weighted-sum ranking
  • uniqueness-scaled formula to penalise generators that just copy real values
  • type-safety filter to exclude continuous generators for integer column
  • keyword-based ranking boost: a small curated list of column-name keywords (e.g. first_name, email, country) that softly boost a matching generator's score
  • propose vs propose all: propose now shows only Pareto front-1 candidates (with original index numbers preserved for set <number>) after excluding type-incompatible generators; propose all bypasses both filters to show everything.
• /evaluators/ module. Add:
  • unit tests
• Extend /evaluators/ module. The architecture is designed to be modular and extensible. Additional evaluators, such as TableEvaluator, can be introduced in the future to support evaluation at the table or dataset level while reusing the same feature extraction, distribution building, and metric components. (A new issue will be created for this)

[MonikaSvata]

Meeting had with Paul and Shipra.
Shipra will summarise feedback.

[MonikaSvata]

Unit test to be added, and comments needed from @tim-band

[MonikaSvata]

Review in progress.