Conversation
…check (#1735) cbm_subtract_macro_invocation_regions asked "is this region a call to a file-defined macro?" before "is it inside a function?". The first question (cbm_span_is_macro_invocation) walks the source from byte 0 to find the region's lines, so every file with parse-error regions paid regions x file bytes, whatever its language and whether or not it has a single function. On a multi-MB SQL dump that is up to 256 walks over the whole file. Both predicates are pure, so asking the cheap one first changes no answer: a region no function encloses never touches the source, and the regions that do share one cbm_line_offsets table built on first use. The table gives exactly the span the walk found; the walk stays as the fallback when the table cannot be allocated. #1071 behaviour is unchanged: same regions, same verdicts. Indexing this repository's src/ before and after gives identical graphs (7,339 nodes, 28,924 edges) and the same 46 parse_partial files with byte-identical ranges. Tests count the bytes read, never time them (test-seam counter cbm_test_macro_line_scan_bytes): - 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions - 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file size), still 60 regions Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
force-pushed
the
fix/issue-1735-sql-cost
branch
from
September 25, 2026 18:08
5ef8ccf to
c0a06b3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(coverage): stop walking the source per error region in the #1071 check (#1735)
cbm_subtract_macro_invocation_regions asked "is this region a call to a
file-defined macro?" before "is it inside a function?". The first question
(cbm_span_is_macro_invocation) walks the source from byte 0 to find the
region's lines, so every file with parse-error regions paid
regions x file bytes, whatever its language and whether or not it has a
single function. On a multi-MB SQL dump that is up to 256 walks over the
whole file.
Both predicates are pure, so asking the cheap one first changes no answer:
a region no function encloses never touches the source, and the regions
that do share one cbm_line_offsets table built on first use. The table
gives exactly the span the walk found; the walk stays as the fallback when
the table cannot be allocated.
#1071 behaviour is unchanged: same regions, same verdicts. Indexing this
repository's src/ before and after gives identical graphs (7,339 nodes,
28,924 edges) and the same 46 parse_partial files with byte-identical
ranges.
Tests count the bytes read, never time them (test-seam counter
cbm_test_macro_line_scan_bytes):
size), still 60 regions
Refs #1735
fix(sql): keep literal INSERT rows out of the SQL parse (#1735)
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of
INSERT ... VALUES (..),(..),...rows. Tree-sitter built a full tree forevery row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL '
escapes, which the grammar does not know and error-recovers from) until the
5 s parse budget ran out, and the whole file, tables included, was skipped
as "parse timeout". The reporter's database/sql/world.sql was dropped this
way. The rows contribute nothing to the graph.
A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c)
finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the
first that consists only of literals: strings (standard '' and MySQL '
escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE,
FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier
or a quoted identifier stays in the parse, and so does one whose string
spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges
as the complement, so byte offsets and line/column positions of all kept
text are unchanged. The ranges are cleared right after the parse because
the parser is thread-local and reused. This is a syntax rule, not a work
cap: the same file always gives the same ranges, and the parse budget is
unchanged.
Measured on generated mysqldump-shaped dumps, release build, same machine,
index worker's own numbers (base = origin/main):
dump extraction pipeline peak RSS graph
8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical
8 MB MySQL ' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical
32 MB MySQL ' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed
59 MB MySQL ' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed
"identical" = every stored node and edge, Module end line included. For the
32/59 MB dumps the base run skipped the file as "parse timeout" (Module +
3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB
graph except for the Module's end line.
With standard escapes the extraction and the stored graph are identical
to the full parse. With MySQL ' escapes the full parse's error recovery
swallows rows next to a misread escape, a subquery row among them, and the
cut parse keeps those rows. So it loses nothing the full parse found
outside the dropped rows and recovers what the full parse lost. The only
things the full parse extracted INSIDE the dropped rows were junk usages
(DEFAULT and _binary read as identifiers, fragments of misread strings)
that name nothing.
Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the
exclusion off for one file so the same source can be compared both ways):
UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions
nothing lost outside dropped rows (MySQL escapes)
"parse timeout" (asserts the outcome, not the time)
Refs #1735
Part (c) of #1735, two commits: the #1071 macro-check cost fix, then the SQL literal-row exclusion. Part (a) is a separate PR.