Store queries in schema template and warmup plan cache on startup#4157
Open
sergei-pustovykh wants to merge 22 commits into
Open
Store queries in schema template and warmup plan cache on startup#4157sergei-pustovykh wants to merge 22 commits into
sergei-pustovykh wants to merge 22 commits into
Conversation
bcb6d3b to
a146137
Compare
a146137 to
18d346f
Compare
…n catalog on startup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a new concept: RecordLayerSchemaTemplate can declare a set of SELECT queries — "stored queries" - which are populated into the RecordMetaData and serialized to protobuf alongside the rest of the schema template.
To initialise schema template, a new clause
CREATE QUERY name AS <select>is introduced, valid only insideCREATE SCHEMA TEMPLATE. The body accepts any SELECT-shaped query. DML and DDL bodies are rejected at parse time. The original SQL text is captured (whitespace and casing preserved) and stored on the template. On the wire it surfaces as astored_queriesmap<string, string> field onMetaDatainrecord_metadata.proto.On engine startup,
OfflineStoredQueriesProcessorwalks the catalog, iterates every schema template, and plans each of its stored queries offline automatically inserting the resulting plans into theRelationalPlanCache.Planner metrics from the offline path are reported through the new
OfflineMetricCollectorto a CodahaleMetricRegistry, landing alongside online-query metrics under the same event names.Note: the stored-query name is not consumed by anything yet at runtime. Names are persisted so that a future feature (e.g., named-plan addressing or plan pinning) can pick them up without another metadata migration.
Worth discussing: offline and online planner metrics currently land under the same event names. That makes the existing dashboards just work but conflates startup pre-warming costs with user-query latencies. We should decide whether to keep them merged, split into distinct events or tag them with a dimension so they can be filtered apart.