docs(asap-tools): evaluation plan for data plane vs ClickHouse

[zzylol]

Summary

Evaluation plan for the paper subsection on ASAP's data plane: the ASAPQuery precompute and query engine against ClickHouse with materialized views. The file is asap-tools/experiments/DATA_PLANE_CLICKHOUSE_EVAL_PLAN.md. The design is settled; nothing is implemented yet.

• Same plan, two data planes. asap-planner --planner milp (Optimizer: move sketch-bench #129 MILP into asap-planner-rs (--planner milp) #753) produces the plan as streaming_config.yaml. A translator turns it into ClickHouse MV DDL, so both systems deploy the same plan.
• First pass: quantiles only, DDSketch on both sides. ClickHouse uses quantilesDD(α), and ASAPQuery uses DDSketch(α), which is added via Add data-plane and query-engine support for new aggregation families #787. Same algorithm and same α, so only the data plane differs. KLL goes in an appendix.
• Arms. Each arm runs separately: plain ClickHouse as the accuracy reference, ClickHouse with exact MVs, ClickHouse with DD-state MVs, and ASAPQuery.
• Workload. label_0 × instance series at 1 sample/s, with Pareto a = 1.5 values and a 95% accuracy target. 8 workloads sweep C (groups), s (series per group) and W (query window). Queries are per-group quantiles, with per-series quantiles in the appendix.
• Runs. Ingest is paced, with queries running during ingest, over 15-minute event-time spans plus one 1-hour default run. Each system is capped at 16 cores / 64 GiB on CloudLab c6320 nodes. That gives 132 runs, about 22 h on 3 nodes.
• Cost uses model A from docs/evaluation/autosketch-vs-planner.md, so the planner and data-plane sections share units.

Dependencies

• DDSketch in ASAPQuery: the DD slice of Add optimizer aggregation families and capability/SLA metadata #762 + Add data-plane and query-engine support for new aggregation families #787.
• --planner milp: Optimizer: move sketch-bench #129 MILP into asap-planner-rs (--planner milp) #753.

The pilot can run before either lands, using today's asap-planner with KLL, to debug the plumbing.

🤖 Generated with Claude Code

[zzylol]

Note from the #792 (DDSketch) review: the two DDSketch implementations pick a different rank for the same quantile, so accuracy has to be scored against one shared reference.

• Rank convention. asap_sketchlib's DDSketch returns the value at rank ceil(q·n). ClickHouse's quantilesDD uses q·(n−1), and quantilesExact has its own convention again. Each is within α of its own definition of the true quantile, so they can disagree by up to one rank. To keep that out of the comparison, compute rank error directly from the reference rows, as the fraction of the window's raw values ≤ the estimate, compared with q. Don't compare against the value some ClickHouse quantile function returns. Both arms are then scored on identical ground truth.
• Window boundaries. ASAPQuery windows are (start, end]: a sample exactly on a boundary goes to the window that ends there (window_manager.rs, pane_start_for). ClickHouse toStartOfInterval buckets are [start, end). At 1 sample/s on whole seconds, every boundary sample lands in a different window on the two sides. The reference query for each ASAPQuery window should select ts > start AND ts <= end, and the ClickHouse MV arms need the same shift (e.g. bucket on ts - 1ms). Otherwise exact-MV vs reference and ASAPQuery vs reference are measured over different rows.

No change to #792 for either point; this is about how §6 "Accuracy" and the MV DDL are specified.