Skip to content

eval: one RQE per schema query, the strictest template's - #216

Merged
milindsrivastava1997 merged 3 commits into
mainfrom
214-eval-rqes-of-one-e2e-query-map-to-different-deployments
Oct 14, 2026
Merged

milindsrivastava1997 merged 3 commits into
mainfrom
214-eval-rqes-of-one-e2e-query-map-to-different-deployments

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Closes #214.

Summary

Templates 14, 15 and 17 share some groupings, so one query string (e.g. distinct user_id by service, endpoint) carried several RQEs with different accuracy scopes. The MILP could serve them from different deployments. The engine answers a query string from one aggregation (ProjectASAP/ASAPQuery#778), so ASAPQuery's --planner saved refuses those plans: 13 of the 16 asap SQL conversions of the e2e 10/s multigroup and all plans.

  • Default: a query several templates ask is only its strictest template's RQE: every group (t17), then 1% (t14), then 5% (t15). That's t17 for service, endpoint, t14 for region, service and region, service, and t15 elsewhere: 10 fewer RQEs (5 queries × 2 windows).
  • Infeasible strictest copy: if it is left out at the table's rate (--samples-per-sec), the export raises instead of falling back to a weaker copy.
  • --keep-duplicate-queries: keeps every copy. The §6.3 commands and descriptions in the READMEs pass it.

The e2e 10/s tables and plans are not regenerated here. That's one re-export and re-plan after this and #215 merge (decisions in #214).

Testing

  • python3 scripts/test_export_autosketch_eval_table.py: 31 pass, 2 new (the strictest copy is kept; an infeasible strictest copy raises). Reversing the strictness order fails both.
  • With --keep-duplicate-queries, all 12 committed tables in autosketch-vs-asap-hydra/inputs/tables regenerate exactly. Without it, the 8 with schema templates change.
  • --samples-per-sec 10 exports without error; every e2e SQL query reads one RQE (multigroup: 50 RQEs, was 60).

🤖 Generated with Claude Code

Templates 14, 15 and 17 share some groupings, so one query string carried
several RQEs with different accuracy scopes. The MILP could serve them
from different deployments, which the engine can't do for one query
string. By default only the strictest template's RQE is kept (every group,
then 1%, then 5%), and an infeasible strictest copy is an error.
--keep-duplicate-queries keeps every copy; the §6.3 tables reproduce with
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997 milindsrivastava1997 linked an issue Oct 14, 2026 that may be closed by this pull request
…e2e-query-map-to-different-deployments

# Conflicts:
#	scripts/export_autosketch_eval_table.py

@milindsrivastava1997 milindsrivastava1997 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two stale docs outside the diff:

  • rqe-optimizer/README.md:36 still says templatesmultigroup has 62 RQEs (and templatesall 104). The default export now writes 52 / 94; only --keep-duplicate-queries gives 62 / 104.
  • rqe-optimizer/results/e2e-rate10/README.md:91 Reproduce runs --synthetic --samples-per-sec 10 without --keep-duplicate-queries, so it no longer rebuilds the committed inputs/tables (multigroup 50 RQEs instead of 60). Add the flag, like the other READMEs in this PR.

if SKIP_BELOW_CURVES and covered_min_n < SCHEMA_MIN_COVERED_N:
# A weaker template's copy might fit, but serving the
# query at a weaker target would change the workload.
if not KEEP_DUPLICATE_QUERIES and copies > 1:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This turns a leave-out into a hard failure for the whole export. With --synthetic --samples-per-sec 2 (or 1) it raises on t17_distinct_users_by_service,endpoint/keys/5m (the strictest of 3 templates, covered group of 449 items), and no tables get written, not even classic. Before this PR only that RQE was dropped. Should this keep leaving the query out (all its copies) with a stderr line, instead of raising?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the error; that was the decision in #214. A query several templates ask must keep its strictest requirement, so the export stops rather than drop or weaken it. At --samples-per-sec 2 the fix is to change the templates (or pass --keep-duplicate-queries), not to plan a workload without that query.

Comment thread scripts/export_autosketch_eval_table.py Outdated
@@ -834,21 +873,29 @@ def main(argv=None):
help=f"--synthetic: samples/s per series (default "
f"{SYNTHETIC_SAMPLES_PER_SEC}); the schema metrics' rates scale "
"with it, and schema RQEs below the curves' first N are left out")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale: without --keep-duplicate-queries, a shared query's strictest copy below the curves' first N now raises rather than being left out. The help here (and on --keep-duplicate-queries) should say so.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 5445c5c: both helps now say the strictest copy below the curves' first N fails the export without --keep-duplicate-queries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997

Copy link
Copy Markdown
Contributor Author

Review in 5445c5c:

  • rqe-optimizer/README.md: those rows describe the committed §6.3 results, which were exported with every copy, so 62 / 104 stay; they now say so, and give 52 / 94 without the flag.
  • e2e-rate10/README.md: the flag is on both export commands and in the inputs description. With it, multigroup rebuilds byte for byte. classic and all differ only in the 6 instant RQEs that eval: instant RQEs read one scrape, not 1 s #215 changed; the regeneration after this PR replaces those tables and drops the flag.
  • Help text: done. The leave-out → error behaviour stays (inline reply).

@milindsrivastava1997
milindsrivastava1997 merged commit acc2db1 into main Oct 14, 2026
2 checks passed
milindsrivastava1997 added a commit that referenced this pull request Oct 14, 2026
…ry (#217)

Regenerates rqe-optimizer/results/e2e-rate10 (classic, all, multigroup at
shared = 1, 10 samples/s) on main: the denser grid's cost table and curves
(#213), instant RQEs reading one scrape (#215), and one RQE per schema query
(#216). Tables, --runs 3 results and --runs 1 plans; the README gives the
RQE counts, costs, classic deployments and ASAPQuery's
--data-ingestion-interval-ms 100.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eval: RQEs of one e2e query map to different deployments

1 participant