Repository navigation
eval: one RQE per schema query, the strictest template's - #216
milindsrivastava1997 merged 3 commits into
Conversation
Templates 14, 15 and 17 share some groupings, so one query string carried several RQEs with different accuracy scopes. The MILP could serve them from different deployments, which the engine can't do for one query string. By default only the strictest template's RQE is kept (every group, then 1%, then 5%), and an infeasible strictest copy is an error. --keep-duplicate-queries keeps every copy; the §6.3 tables reproduce with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e2e-query-map-to-different-deployments # Conflicts: # scripts/export_autosketch_eval_table.py
milindsrivastava1997
left a comment
There was a problem hiding this comment.
Two stale docs outside the diff:
rqe-optimizer/README.md:36still saystemplatesmultigrouphas 62 RQEs (andtemplatesall104). The default export now writes 52 / 94; only--keep-duplicate-queriesgives 62 / 104.rqe-optimizer/results/e2e-rate10/README.md:91Reproduce runs--synthetic --samples-per-sec 10without--keep-duplicate-queries, so it no longer rebuilds the committedinputs/tables(multigroup 50 RQEs instead of 60). Add the flag, like the other READMEs in this PR.
| if SKIP_BELOW_CURVES and covered_min_n < SCHEMA_MIN_COVERED_N: | ||
| # A weaker template's copy might fit, but serving the | ||
| # query at a weaker target would change the workload. | ||
| if not KEEP_DUPLICATE_QUERIES and copies > 1: |
There was a problem hiding this comment.
This turns a leave-out into a hard failure for the whole export. With --synthetic --samples-per-sec 2 (or 1) it raises on t17_distinct_users_by_service,endpoint/keys/5m (the strictest of 3 templates, covered group of 449 items), and no tables get written, not even classic. Before this PR only that RQE was dropped. Should this keep leaving the query out (all its copies) with a stderr line, instead of raising?
There was a problem hiding this comment.
Keeping the error; that was the decision in #214. A query several templates ask must keep its strictest requirement, so the export stops rather than drop or weaken it. At --samples-per-sec 2 the fix is to change the templates (or pass --keep-duplicate-queries), not to plan a workload without that query.
| @@ -834,21 +873,29 @@ def main(argv=None): | |||
| help=f"--synthetic: samples/s per series (default " | |||
| f"{SYNTHETIC_SAMPLES_PER_SEC}); the schema metrics' rates scale " | |||
| "with it, and schema RQEs below the curves' first N are left out") | |||
There was a problem hiding this comment.
Stale: without --keep-duplicate-queries, a shared query's strictest copy below the curves' first N now raises rather than being left out. The help here (and on --keep-duplicate-queries) should say so.
There was a problem hiding this comment.
Done in 5445c5c: both helps now say the strictest copy below the curves' first N fails the export without --keep-duplicate-queries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review in 5445c5c:
|
…ry (#217) Regenerates rqe-optimizer/results/e2e-rate10 (classic, all, multigroup at shared = 1, 10 samples/s) on main: the denser grid's cost table and curves (#213), instant RQEs reading one scrape (#215), and one RQE per schema query (#216). Tables, --runs 3 results and --runs 1 plans; the README gives the RQE counts, costs, classic deployments and ASAPQuery's --data-ingestion-interval-ms 100. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Closes #214.
Summary
Templates 14, 15 and 17 share some groupings, so one query string (e.g. distinct
user_idbyservice, endpoint) carried several RQEs with different accuracy scopes. The MILP could serve them from different deployments. The engine answers a query string from one aggregation (ProjectASAP/ASAPQuery#778), so ASAPQuery's--planner savedrefuses those plans: 13 of the 16 asap SQL conversions of the e2e 10/smultigroupandallplans.service, endpoint, t14 forregion,serviceandregion, service, and t15 elsewhere: 10 fewer RQEs (5 queries × 2 windows).--samples-per-sec), the export raises instead of falling back to a weaker copy.--keep-duplicate-queries: keeps every copy. The §6.3 commands and descriptions in the READMEs pass it.The e2e 10/s tables and plans are not regenerated here. That's one re-export and re-plan after this and #215 merge (decisions in #214).
Testing
python3 scripts/test_export_autosketch_eval_table.py: 31 pass, 2 new (the strictest copy is kept; an infeasible strictest copy raises). Reversing the strictness order fails both.--keep-duplicate-queries, all 12 committed tables inautosketch-vs-asap-hydra/inputs/tablesregenerate exactly. Without it, the 8 with schema templates change.--samples-per-sec 10exports without error; every e2e SQL query reads one RQE (multigroup: 50 RQEs, was 60).🤖 Generated with Claude Code