BE-614: HashQL: Generate synthetic closure bodies for intrinsics used as first-class values#8895
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
PR SummaryMedium Risk Overview Reification adds a Supporting tweaks: Reviewed by Cursor Bugbot for commit 5f1207d. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 48906c1. Configure here.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## bm/be-615-hashql-introduce-synthetic-closures-and-trivial-closure #8895 +/- ##
=====================================================================================================
+ Coverage 26.83% 31.82% +4.99%
=====================================================================================================
Files 652 683 +31
Lines 50479 54881 +4402
Branches 3324 3439 +115
=====================================================================================================
+ Hits 13547 17468 +3921
- Misses 36777 37211 +434
- Partials 155 202 +47
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 32.76%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
There was a problem hiding this comment.
Pull request overview
This PR extends the HashQL MIR reification pipeline to allow certain intrinsics (e.g. <=) to be used as first-class values (passed/bound like normal functions) by generating cached synthetic MIR wrapper bodies, and adds a thin-call specialization to avoid generating intermediary thunk bodies when forcing qualified intrinsics. It also adds diagnostics for intrinsics that are syntactic-only (graph intrinsics) when used in value position, and introduces compiletests covering the new behavior and optimization outcomes.
Changes:
- Add synthetic MIR body/thunk generation + caching for a set of “first-classable” intrinsics during reification.
- Specialize thin calls of qualified intrinsics (
Call(Thin, Qualified(...), [])) into direct closure aggregates. - Add diagnostics + extensive UI tests; reset scratch allocators after reify to avoid retaining transient allocations.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/@local/hashql/mir/src/reify/synthetic.rs | New synthetic body/thunk generation & intrinsic dispatch. |
| libs/@local/hashql/mir/src/reify/rvalue.rs | Thin-call specialization to produce closure aggregates directly for intrinsics. |
| libs/@local/hashql/mir/src/reify/mod.rs | Wire Synthetics into CrossCompileState. |
| libs/@local/hashql/mir/src/reify/error.rs | Add diagnostics for synthetic arity mismatches and non-first-class intrinsics. |
| libs/@local/hashql/mir/src/reify/atom.rs | Route qualified variables through synthetic thunk generation when applicable; avoid duplicate diagnostics. |
| libs/@local/hashql/mir/src/lib.rs | Enable stmt_expr_attributes for in-function attributes used in the synthetic matcher. |
| libs/@local/hashql/core/src/symbol/sym.rs | Add lhs/rhs symbols for synthetic local naming. |
| libs/@local/hashql/core/src/symbol/mod.rs | Add From<ConstantSymbol> for Symbol to simplify diagnostics plumbing. |
| libs/@local/hashql/compiletest/src/pipeline.rs | Reset scratch allocator after MIR reification. |
| libs/@local/graph/api/src/rest/hashql/compile.rs | Reset scratch allocator after MIR reification (and later phases). |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-value.jsonc | New UI test: intrinsic value position triggers synthetic wrapper. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-value.stdout | Expected MIR output for the above. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-reused.jsonc | New UI test: synthetic body caching (reuse). |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-reused.stdout | Expected MIR output for the above. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-multiple.jsonc | New UI test: multiple intrinsics generate distinct synthetic bodies. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-multiple.stdout | Expected MIR output for the above. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-intrinsic-bare.jsonc | New skipped UI test documenting root-level bare intrinsic limitation. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-graph-not-first-class.jsonc | New UI test: graph intrinsic rejected in value position. |
| libs/@local/hashql/mir/tests/ui/reify/synthetic-graph-not-first-class.stderr | Expected diagnostic output for the above. |
| libs/@local/hashql/mir/tests/ui/pass/post_inline/synthetic-intrinsic-hof.jsonc | New pass test: HOF + intrinsic value optimizes to constant. |
| libs/@local/hashql/mir/tests/ui/pass/post_inline/synthetic-intrinsic-hof.stdout | Expected MIR pipeline output for the above. |
| libs/@local/hashql/mir/tests/ui/pass/post_inline/synthetic-intrinsic-hof-dynamic.jsonc | New pass test: dynamic HOF retains comparison shape post-inline. |
| libs/@local/hashql/mir/tests/ui/pass/post_inline/synthetic-intrinsic-hof-dynamic.stdout | Expected MIR pipeline output for the above. |
| libs/@local/hashql/mir/tests/ui/interpret/synthetic-intrinsic-value.jsonc | New interpret test: end-to-end evaluation correctness. |
| libs/@local/hashql/mir/tests/ui/interpret/synthetic-intrinsic-value.stdout | Expected interpreter output for the above. |
| libs/@local/hashql/mir/tests/ui/interpret/synthetic-intrinsic-value.aux.mir | Expected MIR pipeline output for interpret case. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
48906c1 to
c5a86df
Compare
8efe84c to
a0c8ad1
Compare
c5a86df to
feac27a
Compare
a0c8ad1 to
0ddf199
Compare
0ddf199 to
d051ddb
Compare
feac27a to
5f1207d
Compare
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1001 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1526 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 51 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 107 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 25 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |


🌟 What is the purpose of this PR?
Allows intrinsics to be used in value position (passed as arguments, bound to variables) by generating synthetic MIR wrapper bodies during reification. Previously, intrinsics like
<=could only appear at call sites where specialization rewrites them. Now they can flow through higher-order functions:🔍 What does this change?
Synthetic body generation:
Syntheticsstruct inCrossCompileStatemanages creation and caching of wrapper bodiesSyntheticBuildergenerates MIR bodies with fat closure ABI (unit env as first parameter)T!macro for matching qualified paths againstConstantSymbolarrays:&T![::core::cmp::gt]in slice patterns,sym::path::core::cmp::gt::CONSTfor the namebinary!/unary!macros deriving path and name from the same segmentsThin-call specialization:
rvalue_call_thin_specializerecognizesCall(Thin, Qualified(intrinsic), [])(the administrative thunk-force from the thunking phase) and produces the closure aggregate directly, skipping thunk body generation() -> ClosureTypewrapper to extract the actual signatureIntrinsic classification:
gt,lt,gte,lte,eq,ne), boolean (and,or,not), arithmetic (add,sub), bitwise (and,or,not)!payload):mul,div,rem,mod,pow,xor,shl,shrsqrt,cbrt,rootgraph::head::entities,graph::body::filter,graph::tail::collectproduce a specific user-facing diagnosticDiagnostics:
intrinsic_not_first_class: user-facing error for graph intrinsics in value positionsynthetic_binary_arity_mismatch/synthetic_unary_arity_mismatch: ICE for monomorphized type invariant violationsPre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
"<="as the entire program) is not supported. The thunking phase does not wrap the root body when it is a variable, so it never reaches the synthetic machinery. Tracked with arun: skiptest.🐾 Next steps
🛡 What tests cover this?
Compiletests (8 new):
reify/synthetic-intrinsic-value: basic synthetic body generation and thin-call specializationreify/synthetic-intrinsic-reused: same intrinsic used twice shares one body (caching)reify/synthetic-intrinsic-multiple: different intrinsics get separate bodiesreify/synthetic-graph-not-first-class: graph intrinsic in value position produces errorreify/synthetic-intrinsic-bare: bare intrinsic at root (run: skip)interpret/synthetic-intrinsic-value: end-to-end correctness (evaluates totrue)post_inline/synthetic-intrinsic-hof: full optimization cascade with constants (collapses toreturn true)post_inline/synthetic-intrinsic-hof-dynamic: HOF with dynamic inputs collapses to barex <= y❓ How to test this?
cargo run --package hashql-compiletest -- run --filter "test(synthetic)"📹 Demo
The
post_inline/synthetic-intrinsic-hof-dynamictest shows the full cascade. A higher-order function receiving<=as a value with dynamic inputs:All indirection eliminated. Indistinguishable from writing
["<=", x, y]directly.