diff --git a/docs/README.md b/docs/README.md index c9abe1177..e01911037 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,7 +11,7 @@ Choose the path that matches what you need to do. ## Run or inspect a query -Follow [Run and inspect a query](user_guide_docs/run-a-query.md). Tool-specific setup stays with the tool, including the [DAG viewer instructions](../tools/dag-viewer/RUNNING.md). +Follow [Run and inspect a query](user_guide_docs/run-a-query.md). Tool-specific setup stays with the tool, including the [Stage Viewer instructions](../tools/dag-viewer/RUNNING.md). ## Embed the library diff --git a/docs/user_guide_docs/run-a-query.md b/docs/user_guide_docs/run-a-query.md index c6f52aa61..6b883dd0f 100644 --- a/docs/user_guide_docs/run-a-query.md +++ b/docs/user_guide_docs/run-a-query.md @@ -120,7 +120,7 @@ certified sketch. The tool prints plans, not query results. ### Export a query DAG -Export pre-ASAP IR for SQL or PromQL queries for use with the interactive DAG viewer: +Export the IR of SQL or PromQL queries as JSON: ```sh cargo run -p asap-devtools --bin dag_export -- --sql "" @@ -132,7 +132,13 @@ or: cargo run -p asap-devtools --bin dag_export -- --data-ingestion-interval-ms 1000 --promql "" ``` -See [`tools/dag-viewer/RUNNING.md`](../../tools/dag-viewer/RUNNING.md) for instructions on running the DAG viewer. +### See how the planner plans a workload + +```sh +cargo run -p asap-devtools --bin stage_pipeline -- --promql "" --epsilon 0.01 --out plan.json +``` + +writes the planner's four stages for the query. Open `plan.json` in the Stage Viewer, which also plans PromQL queries from its editor; see [`tools/dag-viewer/RUNNING.md`](../../tools/dag-viewer/RUNNING.md). ### Check IR variant coverage diff --git a/tools/dag-viewer/.gitignore b/tools/dag-viewer/.gitignore new file mode 100644 index 000000000..89f9ac04a --- /dev/null +++ b/tools/dag-viewer/.gitignore @@ -0,0 +1 @@ +out/ diff --git a/tools/dag-viewer/README.md b/tools/dag-viewer/README.md index 182c0f94c..626d015f4 100644 --- a/tools/dag-viewer/README.md +++ b/tools/dag-viewer/README.md @@ -1,201 +1,122 @@ -# ASAP Pre/Post-ASAP DAG viewer - -The viewer has one visualization mode: **Pre/Post-ASAP**. - -- Select one query to see that query's complete pre-ASAP and post-ASAP DAGs. -- Select multiple queries to see two workload-union DAGs: one pre-ASAP union - and one post-ASAP union. Nodes with the same exporter-assigned workload - identity are collapsed while query roots and ownership are retained. -- The **All** checkbox left of the query strip selects or deselects every - query at once, and shows an indeterminate state while only some are - selected. It changes the selection only; **Clear all** in the header is a - different operation and discards the loaded workload itself. -- Drag anywhere on the canvas to pan, including on a lane's own background. -- Pre-ASAP nodes show only their original IR content. -- Post-ASAP nodes show their translated IR content and the explicit planner - decision carried by that node. -- Click any edge to inspect its schema, source and target nodes, and how the - target operation derives its output schema in the details panel. -- The details panel shows the selected workload's bound table/metric schemas - and can be resized by dragging its left edge. -- A post-ASAP node whose winning decision carries a cost/benefit annotation - shows a concise `▼NN%`/`▲NN%` badge next to its label; the sidebar and the - workload-scope summary show the full baseline/selected/benefit breakdown, - with units and provenance, wherever the export provides one — see "Cost/ - benefit annotations" below. - -There are no separate Single, Compare, or Union modes. - -## Interactive query editor - -From the repository root: - -```sh -python3 tools/dag-viewer/server.py -``` - -Open , expand **Query editor**, add SQL or PromQL -queries, choose an epsilon, and click **Plan selected workload**. The backend -runs the real pipeline: - -1. SQL/PromQL parsing and lowering -2. pre-ASAP DAG generation -3. ASAP-aware mapping -4. post-ASAP DAG generation - -The editor includes built-in `metrics` and `hosts` table schemas. Enable the -schemas each SQL query uses with its checkboxes, or add a table with custom -columns using one readable `name TYPE [NOT NULL]` declaration per line and -an optional time-index column. The compact `name:type!` form remains accepted -for compatibility. These definitions are passed to `dag_export` and used by the SQL -schema_resolver; they are not display-only metadata. PromQL keeps its open metric -model and shows an inferred schema based on the labels and values used by the -query. The sidebar groups identical input schemas and lists every selected -query that uses each one. - -The Python terminal streams those stages while they run. The server binds to -localhost by default and invokes `dag_export` with an argv array, not a -shell command. - -## Export JSON directly - -```sh -cargo run -p asap-devtools --bin dag_export -- \ - --post-asap --epsilon 0.01 \ - --planner-cost-json "$PLANNER_PHYSICAL_EVIDENCE" \ - --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ - > /tmp/dag.json -``` - -Load the JSON with the page's file picker. `--planner-cost-json` is a complete -physical-evidence document: an immutable `evidence_version`, calibration, and -target records containing the exact target node (a serialized pre-ASAP -`OperatorNode`) and comparison scope. -Each exact replacement candidate owns its complete logical-node -`PhysicalNodeEvidence`; summary candidates additionally own their bound -`PhysicalDAG`. Candidate-local evidence prevents statistics for one physical -alternative from satisfying another. Candidate matching includes the complete -exported plan, including accuracy guarantees, and never uses a hash or strategy -name; derived floating constants allow only a one-ULP JSON round-trip tolerance. -Duplicate, conflicting, unused, or missing records fail closed. The old -`--analytical-cost-json` spelling accepts the new document as an alias; its old -compact aggregation payload is rejected with a migration error. - -`--default-cost` is the alternative cost source for a workload with no -deployment to measure yet, and is mutually exclusive with -`--planner-cost-json`: - -```sh -cargo run -p asap-devtools --bin dag_export -- \ - --post-asap --default-cost --epsilon 0.01 \ - --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ - > /tmp/dag.json -``` - -It ranks candidates with the planner's structural `DefaultCostModel`, so the -structure of the export is real — which replacements the search found, which -one won per group, and the merged post-ASAP DAG — while no cost is exported -at all. Every `CostAnnotation` stays `Unavailable` with no `value` and renders -as **Not estimated**; the structural ranking number is never serialized. Use -it to see what ASAPPlanner does with a workload before there is a deployment -to calibrate against, and `--planner-cost-json` once there is. - -Without either flag, `--post-asap` exports the raw DAG only. - -## Standalone HTML - -```sh -python3 tools/dag-viewer/render.py /tmp/dag.json -o /tmp/dag.html -``` - -The renderer embeds the workload and vendored JavaScript into one file. It -always opens in Pre/Post-ASAP mode; there is no `--mode` option. - -## JSON contract - -`NamedDAG.dag` is the original pre-ASAP DAG. `NamedDAG.post_dag` -is the complete translated DAG. Every post-ASAP node produced or carried by -a selected replacement directly contains: +# ASAPPlanner Stage Viewer + +A browser view of what the planner does with one workload, read from an +`asap-stage-pipeline/v1` document written by the `stage_pipeline` devtool +(#509): + +- **Stage 0**, the logical DAG the frontends lower the queries into, one root + per query; +- **Stage 1**, the logical ASAP candidates from Pass 1's local alternatives and + Pass 2's sharing rules; +- **Stage 2**, the physical candidates of each, which differ in what runs at + ingestion time; +- **Stage 3**, which checks accuracy, latency and the deployment's + capabilities, prices every valid plan per second, and selects the cheapest. + +To run it, see [RUNNING.md](RUNNING.md). + +## The page + +- **Examples**: one tab per entry of `examples.json`, the six #509 examples + (1, 2, 3a, 3b, 4a, 4b). Each says what the workload is and why its plan + wins. `#example4b` in the URL opens that example. +- **Workload queries** with their accuracy, latency and recurrence. +- **Deployment inputs**: the exact aggregates the executor computes, each + sketch with the estimates it can be read for, whether it maintains state + at ingestion time, its memory budget, whether it keeps raw data (when it + does not, query-time plans pay to keep the samples they read), the cost + model with its calibration constants, and the accuracy model. +- **Stage 3 · plans by cost**: every plan, cheapest first, marked selected, + valid but costlier, or invalid with the reason. Clicking one shows it in + the lanes. When a document carries only the cheapest plans, the list says + how many of how many. +- **Three lanes**: Stage 0, the chosen Stage 1 candidate, and the chosen + Stage 2 candidate with each node's timing and Stage 3 cost. Nodes are data + sources, relational operators, or summary operators; summary builds print + their configuration (for example `CmsWithHeap · depth 7 · heap 100 · width + 272`) and whether there is one per group, one per series, or one shared + instance. Query roots have a thick border. +- **Details**: click a node for its operator, output schema, coverage, + guarantee and cost; click an edge for its schema and data state. + +Other documents open with **Open stage document…**, by dropping a file on +the page, or with `?doc=` for a file served next to `index.html`. + +## Query editor + +With the local server running, **Query editor** plans PromQL queries (one +per line) with an optional ε and δ for every query and the sample interval. +The server runs `stage_pipeline --promql … --out …` and the page shows the +result. SQL needs a catalog and is not in the editor yet. + +## Document format ```json { - "decision": { - "id": 7, - "strategy": "ASAPStrategies", - "rationale": "count realizes as a Cms sketch", - "rank": 0, - "cost": 1.14001088, - "role": "replacement_root", - "baseline_cost": { "value": 104.0032, "unit": "CostUnits", "source": "Modeled", "model_version": "analytical-cost-v1+example-calibration-v1", "evidence_version": "example-evidence-v1" }, - "selected_cost": { "value": 1.14001088, "unit": "CostUnits", "source": "Modeled", "baseline": {"kind": "PreAsapRecomputation"}, "delta": 102.86318912, "benefit_ratio": 0.9890386941940248 }, - "benefit": { "value": 102.86318912, "unit": "CostUnits", "source": "Modeled", "baseline": {"kind": "PreAsapRecomputation"}, "benefit_ratio": 0.9890386941940248 } - } + "format": "asap-stage-pipeline/v1", + "workload": { "queries": [{ "id": "Q1", "language": "promql", "text": "...", + "requirements": { "accuracy": "exact", "latency_ms": 100, "repeat_interval_ms": 10000 } }] }, + "stage0_logical": { "dag": "" }, + "stage1_logical_asap": { "candidates": [{ "id": "L1", "label": "...", "dag": "" }] }, + "stage2_physical_asap": { "candidates": [{ "id": "P1", "from_logical": "L1", "label": "...", "dag": "" }] }, + "stage3_selection": { + "costs": { "P1": { "total": 12.5, "unit": "cpu_ms_per_s", "source": "analytical-cost-v1", + "per_node": { "": { "cost": 3.2, "detail": "..." } } } }, + "selected": "P1", + "rejected": [{ "id": "P2", "valid": true, "reason": "costlier" }] + }, + "deployment": { "capabilities": { "...": "..." }, "cost_model": { "...": "..." }, "accuracy_model": { "...": "..." } }, + "shown_of": { "logical": 486, "physical": 486, "priced": 486 } } ``` -The viewer reads this explicit metadata. It never guesses a strategy or -workload-sharing identity from a node label, hash, or client-side signature. -The exporter assigns `workload_node_id`; union rendering reads that mapping -directly. - -Node boxes use concrete IR fields: aggregate measures/grouping, sort keys, -filter predicates, projections, sources, summary families, and evaluation -queries. A node's `kind` is the operator variant name (`Operator::kind_name`): -a `NonASAPOp` such as `Aggregate` or `Values`, or an `ASAPOp` such as -`SummaryAgg` or `EvaluatePopulation`. `node-style.js` maps each kind to a color -category. Scalar expressions are not nodes; an operator a scalar expression -reads (`scalar(v)`, `EXISTS (subquery)`) is a child node, shown in `detail` -as `{"scalar_ref": }`. Schemas list their entries under `fields`. Category icons are deliberately omitted so they cannot be confused -with IR text. - -### Cost/benefit annotations (issue #286) - -`decision.baseline_cost` / `.selected_cost` / `.benefit` are structured -[`CostAnnotation`](../../crates/types/src/cost.rs)s: `value` + `unit` + -`source` (`Modeled` / `Measured` / `Unavailable`), optionally `baseline` + -`delta` + `benefit_ratio`, and `model_version`/`evidence_version`/ -`benchmark_id`/`inputs` for provenance. `model_version` identifies the -analytical formulas and calibration; `evidence_version` independently -identifies the immutable catalog/runtime generation. A missing `value` -(`source: "Unavailable"`) always renders as -**Not estimated** — the viewer never fabricates a number. A complete physical -planner export keeps CPU operations, peak memory, scan bytes, coefficients, -and workload statistics in `inputs`. Without complete physical evidence, the -annotation is `Unavailable`; structural node counts are never substituted, -including under `--default-cost`, where they rank the candidates and are then -discarded. -See the [analytical model design](../../docs/design_docs/asap-aware-mapping/analytical-resource-cost.md). - - -The checked-in viewer fixture makes its illustrative comparison reproducible. -It models 100 evaluations of 100 million 64-byte rows with 100,000 groups. -The raw path charges one scan plus three hash/key/accumulator operations per -row, so CPU is `100 × 100,000,000 × 4 = 40 billion` operations; it reads -`100 × 6.4 GB = 640 GB` and retains -`100,000 × (8-byte key + 8-byte accumulator + 16-byte hash metadata) = 3.2 MB`. -One incrementally built depth-5 CMS charges -`100,000,000 × 5 = 500 million` counter updates, reads the 6.4 GB source once, -and retains `272 × 5 × 8 = 10,880` bytes. With coefficients `1e-9` per CPU -operation, `1e-10` per scan byte, and `1e-9` per peak-memory byte, the displayed totals are -`104.0032` and `1.14001088` cost units. These are explicit fixture assumptions, -not statistics inferred by the viewer. - -The same three fields also appear on `TargetReplacement` -(replacement-region baseline/selected/benefit), `NamedDAG.workload_cost` / -`WorkloadDAG.workload_cost` (whole selected-workload cost/benefit, shared -decisions counted once via `decision.id` dedup). `ExportDAG.edge_annotations` -is reserved for a higher layer that has physical evidence for a particular -edge; DAG sharing alone never creates an edge cost. The sidebar shows the full breakdown -(value, unit, provenance, baseline, ratio, inputs) on node/edge click and in -the workload-scope summary; a post-ASAP node with a costed decision also -gets a concise on-DAG `▼NN%`/`▲NN%` badge next to its label. - -All of this is additive and optional: an export with none of these fields -(anything produced before issue #286) renders exactly as before. +DAGs use the serde JSON of `LogicalASAPDAG` and `PhysicalASAPDAG`: + +- `roots` lists one root per workload query, in workload order. A logical + root is `{"Operator": id}` or `{"Scalar": expr}`; a physical root is a node + id. A single `root` is still accepted. +- `requirements`, and the stages after stage 0, are optional. +- Every stage-2 candidate must be either `selected` or listed in + `rejected`. +- `deployment` (the deployment inputs Stage 3 used) and `shown_of` (present + when the document carries only the cheapest plans) are optional. + +The viewer checks the document before rendering it: + +- node, edge and root references; +- one root per query; +- `from_logical`; +- physical `output_state.timing` and `data_state`; +- Stage 3 ids, costs and `per_node` keys; +- that a later stage never appears without the stage before it. + +If any check fails, the viewer lists the problems and doesn't load the +document. + +## Files + +- `index.html`, `app.js`: the page. +- `stages.js`: document validation, ranking, lane elements and labels. +- `node-style.js`: the operator-kind table (`KIND_CATEGORY_JSON`, checked + against the IR by `crates/devtools/tests/viewer_contract.rs`) and the + three node groups. +- `editor.js`: the query editor. +- `server.py`: serves the page, writes missing example documents into + `out/`, and plans editor queries. +- `examples.json`: the example tabs; `examples/`: the hand-written sample + and the Example 1 fixture `crates/devtools/tests/stage_pipeline.rs` + compares against. +- `cytoscape.min.js`, `dagre.min.js`, `cytoscape-dagre.js`: vendored, so the + page works offline. ## Tests -```sh -python3 -m unittest discover -s tools/dag-viewer -p test_render.py -cargo test -p asap-devtools --bin dag_export +From `tools/dag-viewer`: + +```bash +python3 -m unittest test_viewer ``` + +The JavaScript runs in V8 through `py_mini_racer` (`pip install +py-mini-racer==0.6.0`), against a stub DOM, so no browser is needed; without +it those tests are skipped. diff --git a/tools/dag-viewer/RUNNING.md b/tools/dag-viewer/RUNNING.md index 2a9118d2f..27194649e 100644 --- a/tools/dag-viewer/RUNNING.md +++ b/tools/dag-viewer/RUNNING.md @@ -1,14 +1,27 @@ -# Run the DAG viewer +# Run the Stage Viewer ```bash cd ASAPPlanner python3 tools/dag-viewer/server.py ``` -Open: +The server builds `stage_pipeline`, writes any missing example documents +into `tools/dag-viewer/out/`, and serves the page at: ```text http://localhost:8000/ ``` -Press `Ctrl+C` in the server terminal to stop it and free port 8000. +- `--port 8765` serves on another port. +- `--skip-build` reuses an already-built `stage_pipeline` (under + `$CARGO_TARGET_DIR` if set, otherwise `target/`). +- `--regenerate` rewrites the example documents, for example after a + planner change. + +To open one document instead of the examples: + +```text +http://localhost:8000/?doc=examples/stage-pipeline.sample.json +``` + +Press `Ctrl+C` in the server terminal to stop it. diff --git a/tools/dag-viewer/app.js b/tools/dag-viewer/app.js new file mode 100644 index 000000000..b2a118bf1 --- /dev/null +++ b/tools/dag-viewer/app.js @@ -0,0 +1,247 @@ +// ASAPPlanner Stage Viewer: an asap-stage-pipeline/v1 document as a ranked +// Stage 3 list beside three DAG lanes (Stage 0 → 1 → 2), details below. +// Document parsing, ranking and labels live in stages.js (tested); +// node-style.js groups operator kinds. The same files run in the published +// artifact. +(function () { + if (window.cytoscapeDagre) cytoscape.use(window.cytoscapeDagre); + const $ = (id) => document.getElementById(id); + const esc = (s) => String(s).replace(/[&<>"]/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"' }[c])); + const tok = (name) => getComputedStyle(document.documentElement).getPropertyValue(name).trim(); + + let doc, ranking, queryIds, logicalById, physicalById; + let current = { logical: null, physical: null }; + const cys = [null, null, null]; + + function style() { + return [ + { selector: 'node', style: { + shape: 'round-rectangle', label: 'data(label)', 'text-wrap': 'wrap', 'text-valign': 'center', 'text-halign': 'center', + 'font-family': tok('--font-mono') || 'monospace', 'font-size': 10, color: tok('--fg'), + width: 'label', height: 'label', padding: '9px', 'border-width': 1.5, 'text-max-width': 210 } }, + { selector: 'node[category = "source"]', style: { 'background-color': tok('--cat-source'), 'border-color': tok('--cat-source-line') } }, + { selector: 'node[category = "rel"]', style: { 'background-color': tok('--cat-rel'), 'border-color': tok('--cat-rel-line') } }, + { selector: 'node[category = "summary"]', style: { 'background-color': tok('--cat-summary'), 'border-color': tok('--cat-summary-line') } }, + { selector: 'node[?root]', style: { 'border-width': 3.5, 'border-color': tok('--accent') } }, + { selector: 'node:selected', style: { 'overlay-color': tok('--accent'), 'overlay-opacity': 0.14, 'overlay-padding': 5 } }, + { selector: 'edge', style: { width: 1.6, 'line-color': tok('--edge'), 'target-arrow-color': tok('--edge'), 'target-arrow-shape': 'triangle', 'curve-style': 'bezier', 'arrow-scale': 0.9 } }, + { selector: 'edge:selected', style: { 'line-color': tok('--accent'), 'target-arrow-color': tok('--accent'), width: 2.6 } }, + ]; + } + + function laneElements(dag, physical, costPerNode) { + return stageLaneElements('lane', '', dag, { physical, costPerNode, categoryFor: nodeGroup, queryIds }) + .filter((el) => !el.data.isLane) + .map((el) => { const data = Object.assign({}, el.data); delete data.parent; return { data, classes: el.classes }; }); + } + + function render(i, dag, physical, costPerNode) { + if (cys[i]) cys[i].destroy(); + const cy = cytoscape({ + container: $('cy' + i), elements: laneElements(dag, physical, costPerNode), style: style(), + layout: { name: window.cytoscapeDagre ? 'dagre' : 'breadthfirst', rankDir: 'BT', nodeSep: 18, rankSep: 34, padding: 14, directed: true }, + wheelSensitivity: 0.25, minZoom: 0.2, maxZoom: 2.5, boxSelectionEnabled: false, + }); + cy.on('tap', 'node', (e) => showNode(e.target.data(), i)); + cy.on('tap', 'edge', (e) => showEdge(e.target.data(), i)); + cys[i] = cy; + } + + const fmtCost = (x) => (Math.abs(x) >= 100 ? x.toFixed(1) : String(Number(x.toPrecision(3)))); + + const laneName = ['Stage 0 · logical', 'Stage 1 · logical ASAP', 'Stage 2 · physical ASAP']; + + function schemaTable(schema) { + if (!schema || !Array.isArray(schema.fields)) return ''; + const rows = schema.fields.map((f) => `${esc(f.name)}${esc(compactWire(f.dtype))}${f.nullable ? 'yes' : 'no'}`).join(''); + return `
${rows}
fieldtypenullable
`; + } + + function showNode(d, lane) { + const n = d.stageNode; + const items = [['lane', laneName[lane]], ['operator', d.kind], ['node id', n.id]]; + if (n.payload && n.payload.kind === 'summary_agg') { + items.push(['summary parameters', summaryFamilyText(n.payload.family)]); + items.push(['instances', summaryInstancesText(n.payload.grouping, n.payload.reduction)]); + } + if (d.rootFor && d.rootFor.length) items.push(['query root of', d.rootFor.join(', ')]); + if (d.timing) items.push(['runs at', formatTiming(d.timing)]); + if (n.output_state && n.output_state.primitive) items.push(['output primitive', compactWire(n.output_state.primitive)]); + if (d.nodeCost) items.push(['Stage 3 cost', `${Number(d.nodeCost.cost.toFixed(4))} · ${d.nodeCost.detail || ''}`]); + items.push(['coverage', n.coverage ? compactWire(n.coverage) : 'none (not a summary state)']); + items.push(['guarantee', n.guarantee ? compactWire(n.guarantee) : 'none']); + $('details').innerHTML = `

${esc(d.kind)} node ${esc(n.id)}

+
${items.map(([k, v]) => `
${esc(k)}
${esc(v)}
`).join('')}
+
output schema
${schemaTable(n.output_schema)}
+
Full operator payload (JSON)
${esc(JSON.stringify(n.payload, null, 2))}
`; + } + + function showEdge(d, lane) { + const e = d.stageEdge; + const items = [['lane', laneName[lane]], ['from → to', `node ${e.producer} → node ${e.consumer}`], ['role', compactWire(e.role)], ['grouping', compactWire(e.grouping)]]; + if (e.data_state) items.push(['data state', compactWire(e.data_state)]); + if (e.window) items.push(['window', compactWire(e.window)]); + $('details').innerHTML = `

Edge ${esc(e.producer)} → ${esc(e.consumer)}

+
${items.map(([k, v]) => `
${esc(k)}
${esc(v)}
`).join('')}
+
schema on this edge
${schemaTable(e.intermediate_schema)}
`; + } + + function statusChip(row) { + if (row.status === 'selected') return '✓ selected'; + if (row.status === 'rejected_invalid') return '✗ invalid'; + if (row.status === 'no_selection') return ''; + return 'valid · costlier'; + } + + function renderRanking() { + $('rank').innerHTML = ranking.map((r) => `
    • `).join(''); + $('rank').querySelectorAll('button').forEach((b) => b.addEventListener('click', () => selectPhysical(b.dataset.id))); + } + + function selectLogical(id) { + current.logical = id; + $('pick1').value = id; + const c = logicalById.get(id); + render(1, c.dag, false, null); + const physicalFor = doc.stage2_physical_asap.candidates.filter((p) => p.from_logical === id).map((p) => p.id); + $('lane1-meta').innerHTML = `${esc(c.label)} · ${c.dag.nodes.length} nodes · physical plan ${physicalFor.map((p) => `${esc(p)}`).join(', ') || 'none'}`; + } + + function selectPhysical(id) { + current.physical = id; + $('pick2').value = id; + const c = physicalById.get(id); + const cost = doc.stage3_selection ? doc.stage3_selection.costs[id] : null; + render(2, c.dag, true, cost ? cost.per_node : null); + const row = ranking.find((r) => r.id === id); + $('lane2-meta').innerHTML = `${esc(c.label)} · from ${esc(c.from_logical)} · ` + + (cost ? `total ${fmtCost(cost.total)} · rank ${row.rank} of ${ranking.filter((r) => r.total !== null).length}` : doc.stage3_selection ? 'not priced (invalid)' : 'no Stage 3 result') + ` · ${statusChip(row)}`; + if (c.from_logical !== current.logical) selectLogical(c.from_logical); + renderRanking(); + } + + function restyle() { cys.forEach((cy) => cy && cy.style(style())); } + + $('pick1').addEventListener('change', (e) => selectLogical(e.target.value)); + $('pick2').addEventListener('change', (e) => selectPhysical(e.target.value)); + window.matchMedia('(prefers-color-scheme: dark)').addEventListener('change', restyle); + new MutationObserver(restyle).observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] }); + + const HINT = '

    Select a node to see its operator, output schema, coverage and Stage 3 cost; select an edge to see the schema and data state it carries.

    '; + $('details').innerHTML = HINT; + + // Show one stage document; `source` names it, `story` explains it. + function showDocument(data, source, story) { + const errors = validateStagePipeline(data); + if (errors.length) throw new Error(errors.slice(0, 3).join('; ')); + doc = data; + current = { logical: null, physical: null }; + $('details').innerHTML = HINT; + $('docName').textContent = source || ''; + $('story').textContent = story || ''; + $('story').hidden = !story; + queryIds = doc.workload.queries.map((q) => q.id.toUpperCase()); + logicalById = new Map(doc.stage1_logical_asap.candidates.map((c) => [c.id, c])); + physicalById = new Map(doc.stage2_physical_asap.candidates.map((c) => [c.id, c])); + ranking = rankPhysicalCandidates(doc); + + $('queries').innerHTML = doc.workload.queries.map((q) => { + const r = q.requirements || {}; + const acc = r.accuracy === 'exact' ? 'exact' : r.accuracy ? `ε=${r.accuracy.epsilon}${r.accuracy.delta !== undefined ? `, δ=${r.accuracy.delta}` : ''}` : ''; + const req = [acc && `accuracy ${acc}`, r.latency_ms && `latency ≤ ${r.latency_ms} ms`, r.repeat_interval_ms && `every ${r.repeat_interval_ms / 1000} s`].filter(Boolean).join(' · '); + return `
    ${esc(q.id)} · ${esc(q.language)}${esc(q.text)}${req ? `${esc(req)}` : ''}
    `; + }).join(''); + const deployment = deploymentRows(doc.deployment); + $('deployment').innerHTML = deployment.map(([label, text]) => `
    ${esc(label)}
    ${esc(text)}
    `).join(''); + $('deployment-panel').hidden = deployment.length === 0; + const invalid = ranking.filter((r) => r.status === 'rejected_invalid').length; + const hasStage3 = !!doc.stage3_selection; + $('counts').innerHTML = `Candidates per stage: 1 logical DAG → ${logicalById.size} logical ASAP → ${physicalById.size} physical` + + (hasStage3 ? ` → 1 selected (${invalid} invalid, ${physicalById.size - invalid - 1} valid but costlier)` : ' · no Stage 3 result'); + const shownOf = shownOfText(doc); + $('shown-of').textContent = shownOf; + $('shown-of').hidden = !shownOf; + const sel = hasStage3 ? doc.stage3_selection.costs[doc.stage3_selection.selected] : null; + $('cost-unit').textContent = sel ? sel.unit.replace(/_/g, ' ') : ''; + + $('pick1').innerHTML = doc.stage1_logical_asap.candidates.map((c) => ``).join(''); + $('pick2').innerHTML = ranking.map((r) => ``).join(''); + + render(0, doc.stage0_logical.dag, false, null); + $('lane0-meta').innerHTML = `${doc.stage0_logical.dag.nodes.length} nodes · roots for ${queryIds.map((q) => `${esc(q)}`).join(', ')}`; + const first = hasStage3 ? doc.stage3_selection.selected : ranking.length ? ranking[0].id : null; + if (first) selectPhysical(first); + else { + [1, 2].forEach((i) => { if (cys[i]) { cys[i].destroy(); cys[i] = null; } }); + $('rank').innerHTML = ''; + $('lane1-meta').textContent = $('lane2-meta').textContent = 'no candidates'; + } + } + + function showError(err) { + $('queries').innerHTML = `Could not load the planner output: ${esc(err.message)}`; + } + + // Loads are numbered so a slow earlier fetch cannot replace a later one. + let loading = 0; + function loadUrl(url, source, story) { + const ticket = ++loading; + $('queries').innerHTML = 'Loading planner output…'; + return fetch(url).then((r) => { if (!r.ok) throw new Error(`${url}: HTTP ${r.status}`); return r.json(); }) + .then((data) => { if (ticket === loading) showDocument(data, source, story); }) + .catch((err) => { if (ticket === loading) showError(err); }); + } + + let examples = []; + function pressTab(id) { + $('tabs').querySelectorAll('button').forEach((b) => b.setAttribute('aria-pressed', String(b.dataset.id === id))); + } + function loadExample(example) { + pressTab(example.id); + try { history.replaceState(null, '', '#' + example.id); } catch (e) { /* the hash is a convenience */ } + loadUrl(example.file, example.file, example.story); + } + + // Documents from outside the example list (a file, the editor) clear the tab. + function showExternal(data, source) { + ++loading; + pressTab(null); + try { history.replaceState(null, '', location.pathname + location.search); } catch (e) { /* ignore */ } + try { showDocument(data, source, ''); } catch (err) { showError(err); } + } + + function readFile(file) { + file.text().then((text) => showExternal(JSON.parse(text), file.name)).catch(showError); + } + $('openFile').addEventListener('click', () => $('fileInput').click()); + $('fileInput').addEventListener('change', (e) => { if (e.target.files[0]) readFile(e.target.files[0]); e.target.value = ''; }); + document.addEventListener('dragover', (e) => { e.preventDefault(); document.body.classList.add('drop-active'); }); + document.addEventListener('dragleave', (e) => { if (!e.relatedTarget) document.body.classList.remove('drop-active'); }); + document.addEventListener('drop', (e) => { + e.preventDefault(); + document.body.classList.remove('drop-active'); + if (e.dataTransfer.files[0]) readFile(e.dataTransfer.files[0]); + }); + + $('pick1').addEventListener('change', (e) => selectLogical(e.target.value)); + $('pick2').addEventListener('change', (e) => selectPhysical(e.target.value)); + window.matchMedia('(prefers-color-scheme: dark)').addEventListener('change', restyle); + new MutationObserver(restyle).observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] }); + + window.StageViewer = { showDocument: showExternal }; + + // `?doc=` opens one document; otherwise the examples listed in + // examples.json, starting from `#` or the first. + const requested = new URLSearchParams(location.search).get('doc'); + fetch('examples.json').then((r) => (r.ok ? r.json() : [])).catch(() => []).then((list) => { + examples = Array.isArray(list) ? list : []; + $('tabs').innerHTML = examples.map((x) => ``).join(''); + $('tabs').hidden = examples.length === 0; + $('tabs').querySelectorAll('button').forEach((b) => b.addEventListener('click', () => loadExample(examples.find((x) => x.id === b.dataset.id)))); + if (requested) loadUrl(requested, requested, ''); + else if (examples.length) loadExample(examples.find((x) => '#' + x.id === location.hash) || examples[0]); + }); +})(); diff --git a/tools/dag-viewer/cost-benefit-annotations.png b/tools/dag-viewer/cost-benefit-annotations.png deleted file mode 100644 index 265df5178..000000000 Binary files a/tools/dag-viewer/cost-benefit-annotations.png and /dev/null differ diff --git a/tools/dag-viewer/cost-benefit-positive-savings.png b/tools/dag-viewer/cost-benefit-positive-savings.png deleted file mode 100644 index 1b5d2f179..000000000 Binary files a/tools/dag-viewer/cost-benefit-positive-savings.png and /dev/null differ diff --git a/tools/dag-viewer/dag.example.json b/tools/dag-viewer/dag.example.json deleted file mode 100644 index 5249bb412..000000000 --- a/tools/dag-viewer/dag.example.json +++ /dev/null @@ -1,1022 +0,0 @@ -{ - "queries": [ - { - "name": "q1", - "source": "SELECT service, COUNT(*) FROM metrics GROUP BY service", - "dag": { - "nodes": [ - { - "id": 0, - "kind": "Scan", - "label": "Scan(metrics)", - "detail": { - "predicates": [], - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "source": { - "Table": { - "table_ref": "metrics" - } - } - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "children": [], - "workload_node_id": 0, - "hash": 2606922452740434172 - }, - { - "id": 1, - "kind": "Aggregate", - "label": "Aggregate(1 measures)", - "detail": { - "having": null, - "measures": [ - { - "accuracy": { - "Epsilon": 0.01 - }, - "kind": "count" - } - ], - "output_names": [ - "count(*)" - ], - "reduction": { - "Reduce": [ - 1 - ] - } - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "count(*)", - "nullable": false, - "table": null - } - ], - "time_index": null, - "unique_keys": [ - [ - 0 - ] - ] - }, - "children": [ - 0 - ], - "workload_node_id": 1, - "hash": 12396162568747900801, - "notes": [ - { - "kind": "SketchApproximation", - "reason": "count realizes as a Cms sketch \u2014 one of summary_candidates' alternatives for this intent (asap_aware_mapping::replacement::implementations_for_with); count realizes as a CountSketch sketch \u2014 one of summary_candidates' alternatives for this intent (asap_aware_mapping::replacement::implementations_for_with); count realizes as a shared HydraCms structure over Cms serving every subpopulation of this grouped aggregate, instead of one Cms instance per distinct `by` key \u2014 legal because this aggregate has a non-empty subpopulation concept and Cms has a modeled Hydra variant (asap_types::post_asap::hydra_kind_for); whether it's *worth* the shared/independent trade-off for the actual subpopulation cardinality is a CostModel's call, not this strategy's; count realizes as a shared HydraCountSketch structure over CountSketch serving every subpopulation of this grouped aggregate, instead of one CountSketch instance per distinct `by` key \u2014 legal because this aggregate has a non-empty subpopulation concept and CountSketch has a modeled Hydra variant (asap_types::post_asap::hydra_kind_for); whether it's *worth* the shared/independent trade-off for the actual subpopulation cardinality is a CostModel's call, not this strategy's" - } - ] - }, - { - "id": 2, - "kind": "Project", - "label": "Project(2 cols)", - "detail": { - "cols": [ - { - "alias": null, - "expr": { - "Column": 0 - } - }, - { - "alias": null, - "expr": { - "Column": 1 - } - } - ], - "qualifier": null - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": null - }, - { - "dtype": "int64", - "name": "count(*)", - "nullable": false, - "table": null - } - ], - "time_index": null, - "unique_keys": [ - [ - 0 - ] - ] - }, - "children": [ - 1 - ], - "workload_node_id": 2, - "hash": 17063396757587155626 - } - ], - "root": 2 - }, - "replacements": [ - { - "decision_id": 0, - "target_pre_id": 1, - "strategy": "ASAPStrategies", - "rationale": "count realizes as a Cms sketch", - "rank": 0, - "cost": 1.14001088, - "before": { - "nodes": [ - { - "id": 0, - "kind": "Scan", - "label": "Scan(metrics)", - "detail": { - "predicates": [], - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "source": { - "Table": { - "table_ref": "metrics" - } - } - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "children": [], - "hash": 2606922452740434172 - }, - { - "id": 1, - "kind": "Aggregate", - "label": "Aggregate(1 measures)", - "detail": { - "having": null, - "measures": [ - { - "accuracy": { - "Epsilon": 0.01 - }, - "kind": "count" - } - ], - "output_names": [ - "count(*)" - ], - "reduction": { - "Reduce": [ - 1 - ] - } - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "count(*)", - "nullable": false, - "table": null - } - ], - "time_index": null, - "unique_keys": [ - [ - 0 - ] - ] - }, - "children": [ - 0 - ], - "hash": 12396162568747900801 - } - ], - "root": 1 - }, - "after": { - "kind": "Summary", - "dag": { - "nodes": [ - {"id": 0, "kind": "Scan", "label": "Scan(metrics)", "detail": {"predicates": [], "schema": {"closed": true, "columns": [{"dtype": "timestamp", "name": "ts", "nullable": false, "table": "metrics"}, {"dtype": "utf8", "name": "service", "nullable": false, "table": "metrics"}, {"dtype": "utf8", "name": "region", "nullable": false, "table": "metrics"}, {"dtype": "float64", "name": "latency", "nullable": false, "table": "metrics"}, {"dtype": "int64", "name": "bytes", "nullable": false, "table": "metrics"}], "time_index": 0, "unique_keys": []}, "source": {"Table": {"table_ref": "metrics"}}}, "children": []}, - { - "id": 1, - "kind": "SummaryAgg", - "label": "SummaryAgg(Sketch(Cms))", - "detail": { - "col": "SampleValue", - "family": "Sketch(SketchKind { category: Frequency, algorithm: Cms, params: Cms { width: 272, depth: 5 } }, PerSubpopulationInstance)", - "grouping": "PerSubpopulationInstance", - "reduction": { - "Reduce": [ - 1 - ] - } - }, - "children": [ - 0 - ] - }, - { - "id": 2, - "kind": "SummaryEstimate", - "label": "SummaryEstimate(PointCount { key: SampleValue, value: None })", - "detail": { - "query": "PointCount { key: SampleValue, value: None }" - }, - "children": [ - 1 - ] - } - ], - "root": 2 - } - }, - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } - ], - "post_dag": { - "nodes": [ - { - "id": 0, - "kind": "Scan", - "label": "Scan(metrics)", - "detail": { - "predicates": [], - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "source": { - "Table": { - "table_ref": "metrics" - } - } - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "timestamp", - "name": "ts", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "utf8", - "name": "region", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "float64", - "name": "latency", - "nullable": false, - "table": "metrics" - }, - { - "dtype": "int64", - "name": "bytes", - "nullable": false, - "table": "metrics" - } - ], - "time_index": 0, - "unique_keys": [] - }, - "children": [], - "workload_node_id": 0, - "hash": 2606922452740434172, - "decision": { - "id": 0, - "strategy": "ASAPStrategies", - "rationale": "count realizes as a Cms sketch", - "rank": 0, - "cost": 1.14001088, - "role": "replacement_region", - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } - }, - { - "id": 1, - "kind": "SummaryAgg", - "label": "SummaryAgg(Sketch(Cms))", - "detail": { - "col": "SampleValue", - "family": "Sketch(SketchKind { category: Frequency, algorithm: Cms, params: Cms { width: 272, depth: 5 } }, PerSubpopulationInstance)", - "grouping": "PerSubpopulationInstance", - "reduction": { - "Reduce": [ - 1 - ] - } - }, - "schema": { - "fields": [ - { - "dtype": "Plain(Utf8)", - "name": "service", - "nullable": false - }, - { - "dtype": "Sketch(SketchKind { category: Frequency, algorithm: Cms, params: Cms { width: 272, depth: 5 } }, PerSubpopulationInstance)", - "name": "count(*)", - "nullable": false - } - ], - "time_index": null - }, - "children": [ - 0 - ], - "workload_node_id": 1, - "decision": { - "id": 0, - "strategy": "ASAPStrategies", - "rationale": "count realizes as a Cms sketch", - "rank": 0, - "cost": 1.14001088, - "role": "replacement_region", - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } - }, - { - "id": 2, - "kind": "SummaryEstimate", - "label": "SummaryEstimate(PointCount { key: SampleValue, value: None })", - "detail": { - "query": "PointCount { key: SampleValue, value: None }" - }, - "schema": { - "fields": [ - { - "dtype": "Plain(Utf8)", - "name": "service", - "nullable": false - }, - { - "dtype": "Plain(Int64)", - "name": "count(*)", - "nullable": false - } - ], - "time_index": null - }, - "children": [ - 1 - ], - "workload_node_id": 2, - "decision": { - "id": 0, - "strategy": "ASAPStrategies", - "rationale": "count realizes as a Cms sketch", - "rank": 0, - "cost": 1.14001088, - "role": "replacement_root", - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } - }, - { - "id": 3, - "kind": "Project", - "label": "Project(2 cols)", - "detail": { - "cols": [ - { - "alias": null, - "expr": { - "Column": 0 - } - }, - { - "alias": null, - "expr": { - "Column": 1 - } - } - ], - "qualifier": null - }, - "schema": { - "closed": true, - "columns": [ - { - "dtype": "utf8", - "name": "service", - "nullable": false, - "table": null - }, - { - "dtype": "int64", - "name": "count(*)", - "nullable": false, - "table": null - } - ], - "time_index": null, - "unique_keys": [ - [ - 0 - ] - ] - }, - "children": [ - 2 - ], - "workload_node_id": 3, - "hash": 17063396757587155626 - } - ], - "root": 3 - }, - "workload_cost": { - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } - } - ], - "workload_cost": { - "baseline_cost": { - "value": 104.0032, - "unit": "CostUnits", - "source": "Modeled", - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 40000000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 3200000.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 640000000000.0, - "unit": "bytes" - } - ] - }, - "selected_cost": { - "value": 1.14001088, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "delta": 102.86318912, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1", - "inputs": [ - { - "name": "estimated_cpu_ops", - "value": 500000000.0, - "unit": "operations" - }, - { - "name": "estimated_peak_memory", - "value": 10880.0, - "unit": "bytes" - }, - { - "name": "estimated_scan", - "value": 6400000000.0, - "unit": "bytes" - } - ] - }, - "benefit": { - "value": 102.86318912, - "unit": "CostUnits", - "source": "Modeled", - "baseline": { - "kind": "PreAsapRecomputation" - }, - "benefit_ratio": 0.9890386941940248, - "model_version": "analytical-cost-v1+example-calibration-v1", - "evidence_version": "example-evidence-v1" - } - } -} diff --git a/tools/dag-viewer/editor.js b/tools/dag-viewer/editor.js new file mode 100644 index 000000000..54c9cf7f2 --- /dev/null +++ b/tools/dag-viewer/editor.js @@ -0,0 +1,42 @@ +// Query editor: plans PromQL queries through the local server's /api/plan +// (server.py runs `stage_pipeline`) and shows the resulting stage document. +(function () { + const $ = (id) => document.getElementById(id); + + $('editorToggle').addEventListener('click', () => { + const open = $('editor').hidden; + $('editor').hidden = !open; + $('editorToggle').setAttribute('aria-pressed', String(open)); + }); + + let busy = false; + $('editorForm').addEventListener('submit', async (event) => { + event.preventDefault(); + if (busy) return; + const queries = $('editorQueries').value.split('\n').map((q) => q.trim()).filter(Boolean); + if (!queries.length) { + $('editorStatus').textContent = 'Enter at least one PromQL query.'; + return; + } + const body = { queries, interval_ms: Number($('editorInterval').value) || 15000 }; + if ($('editorEpsilon').value) body.epsilon = Number($('editorEpsilon').value); + if ($('editorDelta').value) body.delta = Number($('editorDelta').value); + busy = true; + $('editorStatus').textContent = 'Planning…'; + try { + const response = await fetch('/api/plan', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + const result = await response.json().catch(() => ({ error: `HTTP ${response.status}` })); + if (!response.ok) throw new Error(result.error || `HTTP ${response.status}`); + window.StageViewer.showDocument(result, `editor · ${queries.length} quer${queries.length === 1 ? 'y' : 'ies'}`); + $('editorStatus').textContent = 'Planned.'; + } catch (err) { + $('editorStatus').textContent = `Planning failed: ${err.message}`; + } finally { + busy = false; + } + }); +})(); diff --git a/tools/dag-viewer/examples.json b/tools/dag-viewer/examples.json new file mode 100644 index 000000000..9ff42877d --- /dev/null +++ b/tools/dag-viewer/examples.json @@ -0,0 +1,44 @@ +[ + { + "id": "example1", + "name": "1 · dashboard panels", + "example": "planner-layering-1", + "file": "out/example1.json", + "story": "Two panels over 1M counter series every 10 s: Q1 the exact per-job request rate, Q2 the approximate top-10 series per job. The planner keeps Q1 exact and answers Q2 with one CMS+heap over the raw samples, both reading one shared scan." + }, + { + "id": "example2", + "name": "2 · SQL flow statistics", + "example": "planner-layering-2", + "file": "out/example2.json", + "story": "Three SQL queries over the last minute of flows: distinct sources, and the entropy and L2 of the per-source counts. Pass 1 offers UnivMon to each and Pass 2 one shared UnivMon, but there is no UnivMon accuracy model yet (Q45), so every UnivMon plan is invalid and an exact plan wins." + }, + { + "id": "example3a", + "name": "3a · historical p99 batch", + "example": "planner-layering-3a", + "file": "out/example3a.json", + "story": "An ad hoc batch of five p99 reports over 1–5 years, run once. Each query gets a KLL, and sharing the input scan across all five is cheapest. Exponential histograms are not implemented yet, so no plan shares window summaries across the reports." + }, + { + "id": "example3b", + "name": "3b · live p99 panel", + "example": "planner-layering-3b", + "file": "out/example3b.json", + "story": "p99 over the last 5 min every minute, 1M series sampled every 15 s, raw data kept by the deployment. A per-series KLL built at query time wins: keeping 1-min KLL panes from ingestion time costs 768 cost/s of memory, and rebuilding the panes at query time breaks the 200 ms bound." + }, + { + "id": "example4a", + "name": "4a · monthly p99 reports", + "example": "planner-layering-4a", + "file": "out/example4a.json", + "story": "Example 3a repeated monthly and known in advance. The same shared-input KLL plan wins; its cost is now amortized over the month between runs." + }, + { + "id": "example4b", + "name": "4b · panes that pay off", + "example": "planner-layering-4b", + "file": "out/example4b.json", + "story": "p99 over the last hour every 10 min, 1k series sampled every second, and a deployment that does not keep raw data (Q49). A 10-min pane covers 600 samples per series, far more bytes than its KLL, so maintaining six panes at ingestion time beats keeping an hour of raw samples." + } +] diff --git a/tools/dag-viewer/examples/stage-pipeline.sample.json b/tools/dag-viewer/examples/stage-pipeline.sample.json new file mode 100644 index 000000000..56625227f --- /dev/null +++ b/tools/dag-viewer/examples/stage-pipeline.sample.json @@ -0,0 +1,7220 @@ +{ + "format": "asap-stage-pipeline/v1", + "workload": { + "queries": [ + { + "id": "Q1", + "language": "promql", + "text": "sum by (job) (rate(http_requests_total[1m]))", + "requirements": { + "accuracy": "exact", + "repeat_interval_ms": 10000 + } + }, + { + "id": "Q2", + "language": "promql", + "text": "topk by (job) (10, sum_over_time(http_requests_total[1m]))", + "requirements": { + "accuracy": { + "epsilon": 0.01, + "delta": 0.001 + }, + "latency_ms": 100, + "repeat_interval_ms": 10000 + } + } + ] + }, + "stage0_logical": { + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 7, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "top_k", + "k": 10, + "accuracy": "Exact" + } + ], + "output_names": [], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + } + ], + "roots": [ + { + "Operator": 3 + }, + { + "Operator": 7 + } + ] + } + }, + "stage1_logical_asap": { + "candidates": [ + { + "id": "L1", + "label": "Exact, separate inputs", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 7, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "top_k", + "k": 10, + "accuracy": "Exact" + } + ], + "output_names": [], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series sum, top 10 per job" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + } + ], + "roots": [ + { + "Operator": 3 + }, + { + "Operator": 7 + } + ] + } + }, + { + "id": "L2", + "label": "Q2: Count-Min + top-10 heap per job", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "summary_agg", + "family": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "input": { + "item": { + "EntityIdentity": { + "PromqlLabelSet": { + "excluding": [] + } + } + }, + "weight": { + "Column": "SampleValue" + }, + "weight_domain": { + "kind": "unknown_or_signed" + } + }, + "reduction": { + "Reduce": [ + 2 + ] + }, + "grouping": "PerSubpopulationInstance", + "filter": null + }, + "result_kind": "state", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": { + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "regions": [ + { + "time_ms": null, + "population": {} + } + ] + } + }, + { + "id": 7, + "payload": { + "kind": "summary_estimate", + "query": { + "TopK": { + "k": 10 + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "relative_value", + "bound": { + "op": "constant", + "value": 0.01 + }, + "failure_probability": { + "op": "constant", + "value": 0.001 + }, + "provenance": [ + { + "kind": "sketch_readout", + "algorithm": "CmsWithHeap", + "contract": "eps-delta frequency bound on heap members", + "params": { + "width": 272, + "depth": 7, + "heap_size": 10 + }, + "query": "TopK { k: 10 }" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "grouping": "Identical" + } + ], + "roots": [ + { + "Operator": 3 + }, + { + "Operator": 7 + } + ] + } + }, + { + "id": "L3", + "label": "Q2: Hydra over (job, series) + heap per job", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "summary_agg", + "family": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "input": { + "item": { + "EntityIdentity": { + "PromqlLabelSet": { + "excluding": [] + } + } + }, + "weight": { + "Column": "SampleValue" + }, + "weight_domain": { + "kind": "unknown_or_signed" + } + }, + "reduction": { + "Reduce": [ + 2 + ] + }, + "grouping": { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + }, + "filter": null + }, + "result_kind": "state", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": { + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "regions": [ + { + "time_ms": null, + "population": {} + } + ] + } + }, + { + "id": 7, + "payload": { + "kind": "summary_estimate", + "query": { + "TopK": { + "k": 10 + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "relative_value", + "bound": { + "op": "constant", + "value": 0.01 + }, + "failure_probability": { + "op": "constant", + "value": 0.001 + }, + "provenance": [ + { + "kind": "sketch_readout", + "algorithm": "HydraCms", + "contract": "eps-delta frequency bound on heap members", + "params": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + }, + "query": "TopK { k: 10 }" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "grouping": "Identical" + } + ], + "roots": [ + { + "Operator": 3 + }, + { + "Operator": 7 + } + ] + } + }, + { + "id": "L4", + "label": "Exact, Q1 and Q2 share one input", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "result_kind": "range_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "top_k", + "k": 10, + "accuracy": "Exact" + } + ], + "output_names": [], + "filters": [], + "having": null + } + }, + "result_kind": "instant_vector", + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series sum, top 10 per job" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 1, + "consumer": 4, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "grouping": "NotApplicable" + } + ], + "roots": [ + { + "Operator": 3 + }, + { + "Operator": 5 + } + ] + } + } + ] + }, + "stage2_physical_asap": { + "candidates": [ + { + "id": "P1", + "from_logical": "L1", + "label": "Exact; Q2 top 10 as sort -> limit per job", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 7, + "payload": { + "kind": "relational", + "operator": { + "kind": "sort", + "keys": [ + { + "expr": { + "Column": 1 + }, + "ascending": false, + "nulls_first": false + } + ], + "partition_by": { + "keys": [ + 2 + ], + "without": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 8, + "payload": { + "kind": "relational", + "operator": { + "kind": "limit", + "n": 10, + "offset": 0, + "partition_by": { + "keys": [ + 2 + ], + "without": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series sum, top 10 per job" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 7, + "consumer": 8, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "Identical", + "window": "NotApplicable" + } + ], + "roots": [ + 3, + 8 + ] + } + }, + { + "id": "P2", + "from_logical": "L4", + "label": "Shared input; Q2 top 10 as sort -> limit per job", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "sort", + "keys": [ + { + "expr": { + "Column": 1 + }, + "ascending": false, + "nulls_first": false + } + ], + "partition_by": { + "keys": [ + 2 + ], + "without": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "relational", + "operator": { + "kind": "limit", + "n": 10, + "offset": 0, + "partition_by": { + "keys": [ + 2 + ], + "without": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series sum, top 10 per job" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 1, + "consumer": 4, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "Identical", + "window": "NotApplicable" + } + ], + "roots": [ + 3, + 6 + ] + } + }, + { + "id": "P3", + "from_logical": "L2", + "label": "Q2: build Count-Min + heap -> estimate", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "summary_agg", + "family": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "input": { + "item": { + "EntityIdentity": { + "PromqlLabelSet": { + "excluding": [] + } + } + }, + "weight": { + "Column": "SampleValue" + }, + "weight_domain": { + "kind": "unknown_or_signed" + } + }, + "reduction": { + "Reduce": [ + 2 + ] + }, + "grouping": "PerSubpopulationInstance", + "filter": null + }, + "output_state": { + "timing": "query_time", + "primitive": "SummaryState" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": { + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "regions": [ + { + "time_ms": null, + "population": {} + } + ] + } + }, + { + "id": 7, + "payload": { + "kind": "summary_estimate", + "query": { + "TopK": { + "k": 10 + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "relative_value", + "bound": { + "op": "constant", + "value": 0.01 + }, + "failure_probability": { + "op": "constant", + "value": 0.001 + }, + "provenance": [ + { + "kind": "sketch_readout", + "algorithm": "CmsWithHeap", + "contract": "eps-delta frequency bound on heap members", + "params": { + "width": 272, + "depth": 7, + "heap_size": 10 + }, + "query": "TopK { k: 10 }" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + "PerSubpopulationInstance" + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "data_state": { + "timing": "query_time", + "primitive": "SummaryState" + }, + "grouping": "Identical", + "window": "NotApplicable" + } + ], + "roots": [ + 3, + 7 + ] + } + }, + { + "id": "P4", + "from_logical": "L3", + "label": "Q2: build Hydra -> estimate", + "dag": { + "nodes": [ + { + "id": 0, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 1, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 2, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": "PerEntity", + "measures": [ + { + "kind": "rate" + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 3, + "payload": { + "kind": "relational", + "operator": { + "kind": "aggregate", + "reduction": { + "Reduce": [ + 2 + ] + }, + "measures": [ + { + "kind": "sum", + "col": null + } + ], + "output_names": [ + "" + ], + "filters": [], + "having": null + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "sum", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "absolute_value", + "bound": { + "op": "zero" + }, + "failure_probability": { + "op": "zero" + }, + "provenance": [ + { + "kind": "exact", + "reason": "exact per-series rate and per-job sum" + } + ] + }, + "coverage": null + }, + { + "id": 4, + "payload": { + "kind": "relational", + "operator": { + "kind": "scan", + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "predicates": [], + "schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 5, + "payload": { + "kind": "relational", + "operator": { + "kind": "time_range", + "range": { + "secs": 60, + "nanos": 0 + }, + "range_kind": "range" + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "guarantee": null, + "coverage": null + }, + { + "id": 6, + "payload": { + "kind": "summary_agg", + "family": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "input": { + "item": { + "EntityIdentity": { + "PromqlLabelSet": { + "excluding": [] + } + } + }, + "weight": { + "Column": "SampleValue" + }, + "weight_domain": { + "kind": "unknown_or_signed" + } + }, + "reduction": { + "Reduce": [ + 2 + ] + }, + "grouping": { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + }, + "filter": null + }, + "output_state": { + "timing": "query_time", + "primitive": "SummaryState" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": null, + "coverage": { + "source": { + "TimeSeries": { + "metric": "http_requests_total" + } + }, + "regions": [ + { + "time_ms": null, + "population": {} + } + ] + } + }, + { + "id": 7, + "payload": { + "kind": "summary_estimate", + "query": { + "TopK": { + "k": 10 + } + } + }, + "output_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "output_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "topk_10", + "dtype": { + "Plain": "utf8" + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "guarantee": { + "metric": "relative_value", + "bound": { + "op": "constant", + "value": 0.01 + }, + "failure_probability": { + "op": "constant", + "value": 0.001 + }, + "provenance": [ + { + "kind": "sketch_readout", + "algorithm": "HydraCms", + "contract": "eps-delta frequency bound on heap members", + "params": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + }, + "query": "TopK { k: 10 }" + } + ] + }, + "coverage": null + } + ], + "edges": [ + { + "producer": 0, + "consumer": 1, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 1, + "consumer": 2, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 2, + "consumer": 3, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 4, + "consumer": 5, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 5, + "consumer": 6, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "ts", + "dtype": { + "Plain": "timestamp" + }, + "nullable": false, + "table": null + }, + { + "name": "value", + "dtype": { + "Plain": "float64" + }, + "nullable": false, + "table": null + }, + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + } + ], + "time_index": 0, + "unique_keys": [], + "closed": false + }, + "data_state": { + "timing": "query_time", + "primitive": "Raw" + }, + "grouping": "NotApplicable", + "window": "NotApplicable" + }, + { + "producer": 6, + "consumer": 7, + "role": "Input", + "intermediate_schema": { + "fields": [ + { + "name": "job", + "dtype": { + "Plain": "utf8" + }, + "nullable": true, + "table": null + }, + { + "name": "state", + "dtype": { + "Sketch": [ + { + "category": "TopK", + "algorithm": "CmsWithHeap", + "params": { + "CmsWithHeap": { + "width": 272, + "depth": 7, + "heap_size": 10 + } + } + }, + { + "SharedMultiSubpopulation": { + "kind": "HydraCms", + "params": { + "HydraCms": { + "width": 272, + "depth": 7, + "shared_rows": 5, + "shared_columns": 65536 + } + } + } + } + ] + }, + "nullable": false, + "table": null + } + ], + "time_index": null, + "unique_keys": [ + [ + 0 + ] + ], + "closed": true + }, + "data_state": { + "timing": "query_time", + "primitive": "SummaryState" + }, + "grouping": "Identical", + "window": "NotApplicable" + } + ], + "roots": [ + 3, + 7 + ] + } + } + ] + }, + "stage3_selection": { + "costs": { + "P1": { + "total": 354.0, + "unit": "cpu_ms_per_s", + "source": "analytical-cost-v1 (illustrative fixture values)", + "per_node": { + "0": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "1": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "2": { + "cost": 36.0, + "detail": "per-series rate over 1,000,000 series" + }, + "3": { + "cost": 12.0, + "detail": "hash sum into one row per job" + }, + "4": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "5": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "6": { + "cost": 90.0, + "detail": "per-series sum over 1,000,000 series" + }, + "7": { + "cost": 110.0, + "detail": "sort 1,000,000 series within each job" + }, + "8": { + "cost": 2.0, + "detail": "keep 10 rows per job" + } + } + }, + "P2": { + "total": 302.0, + "unit": "cpu_ms_per_s", + "source": "analytical-cost-v1 (illustrative fixture values)", + "per_node": { + "0": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "1": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "2": { + "cost": 36.0, + "detail": "per-series rate over 1,000,000 series" + }, + "3": { + "cost": 12.0, + "detail": "hash sum into one row per job" + }, + "4": { + "cost": 90.0, + "detail": "per-series sum over 1,000,000 series" + }, + "5": { + "cost": 110.0, + "detail": "sort 1,000,000 series within each job" + }, + "6": { + "cost": 2.0, + "detail": "keep 10 rows per job" + } + } + }, + "P3": { + "total": 196.4, + "unit": "cpu_ms_per_s", + "source": "analytical-cost-v1 (illustrative fixture values)", + "per_node": { + "0": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "1": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "2": { + "cost": 36.0, + "detail": "per-series rate over 1,000,000 series" + }, + "3": { + "cost": 12.0, + "detail": "hash sum into one row per job" + }, + "4": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "5": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "6": { + "cost": 42.8, + "detail": "insert 4.0M samples x 7 CMS rows, heap per job" + }, + "7": { + "cost": 1.6, + "detail": "read 10 heap entries per job" + } + } + }, + "P4": { + "total": 184.4, + "unit": "cpu_ms_per_s", + "source": "analytical-cost-v1 (illustrative fixture values)", + "per_node": { + "0": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "1": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "2": { + "cost": 36.0, + "detail": "per-series rate over 1,000,000 series" + }, + "3": { + "cost": 12.0, + "detail": "hash sum into one row per job" + }, + "4": { + "cost": 48.0, + "detail": "read 1 min of raw samples (4.0M) per 10 s refresh" + }, + "5": { + "cost": 4.0, + "detail": "slice the 1 min window" + }, + "6": { + "cost": 30.5, + "detail": "insert 4.0M samples x 5 shared rows, one structure for all jobs" + }, + "7": { + "cost": 1.9, + "detail": "query the shared grid and heap per job" + } + } + } + }, + "selected": "P4", + "rejected": [ + { + "id": "P1", + "valid": false, + "reason": "Q2 misses its 100 ms latency bound: an exact top 10 sorts 1,000,000 series at every refresh (about 340 ms)." + }, + { + "id": "P2", + "valid": false, + "reason": "Q2 misses its 100 ms latency bound: sharing the input does not remove the 1,000,000-series sort." + }, + { + "id": "P3", + "valid": true, + "reason": "Valid but costlier than the selected plan." + } + ] + } +} diff --git a/tools/dag-viewer/generate-sample.sh b/tools/dag-viewer/generate-sample.sh deleted file mode 100755 index 66921d8c7..000000000 --- a/tools/dag-viewer/generate-sample.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env bash -# Regenerates tools/dag-viewer/dag.example.json from real example queries. -# Run from anywhere in the repo; edit the --sql/--promql lines below to try -# your own queries instead. -set -euo pipefail -cd "$(dirname "${BASH_SOURCE[0]}")/../.." - -# --epsilon asks for an approximate accuracy target instead of the default -# Exact, so ASAPStrategies actually has a sketch alternative to -# report — without it, no query below would ever pick up a `notes` badge -# (see crates/devtools/src/bin/dag_export.rs's own `--epsilon` doc comment). -cargo run -p asap-devtools --bin dag_export -- \ - --post-asap --progress \ - --epsilon 0.01 \ - --sql "SELECT service, COUNT(*) FROM metrics GROUP BY service" --name q1 \ - --sql "SELECT service, AVG(latency) FROM metrics GROUP BY service" --name q2 \ - --promql "topk(5, rate(http_requests_total[5m]))" --name q3 \ - --promql "topk(10, rate(http_requests_total[5m]))" --name q4 \ - --sql "SELECT metrics.service, COUNT(*) FROM metrics JOIN hosts ON metrics.service = hosts.service GROUP BY metrics.service" --name q6 \ - > tools/dag-viewer/dag.example.json - -echo "wrote tools/dag-viewer/dag.example.json" diff --git a/tools/dag-viewer/index.html b/tools/dag-viewer/index.html index 17754d280..b773116a1 100644 --- a/tools/dag-viewer/index.html +++ b/tools/dag-viewer/index.html @@ -2,536 +2,211 @@ -ASAP query DAG viewer - - - - +ASAPPlanner Stage Viewer - -
    -

    ASAP query DAG viewer

    -
    Drop WorkloadDAG JSON here, or click to choose file(s)
    - -
    - -
    - - - - Click an edge to inspect its schema -
    - -
    -
    -
    -

    Input table schemas

    - - One column per line: name TYPE [NOT NULL]. PromQL schemas are inferred from metric usage. +
    +
    +

    ASAPPlanner Stage Viewer

    +

    The planner's four stages for one workload: the frontends lower the queries into one logical DAG (Stage 0), Pass 1 and Pass 2 propose logical ASAP candidates (Stage 1), each becomes physical plans that differ in what runs at ingestion time (Stage 2), and Stage 3 checks accuracy, latency and deployment capabilities, prices every valid plan and selects the cheapest.

    + +
    + + + +
    -
    -
    -
    -
    - - - - Requires the local interactive server. -
    -
    + +

    +
    Pick an example, open a stage document, or plan queries in the editor.
    +
    + + -
    - -
    -
    -
    +
    +
    +

    Stage 3 · plans by cost

    + +
      +
      -
      -
      - Load one or more JSON files produced by - cargo run -p asap-devtools --bin dag_export -- --sql "..." --name q1 > dag.json - to get started. -
      -