Skip to content

fix(sql,coverage): make large SQL dumps cheap to parse (#1735) - #2326

Open
DeusData wants to merge 2 commits into
mainfrom
fix/issue-1735-sql-cost
Open

DeusData wants to merge 2 commits into
mainfrom
fix/issue-1735-sql-cost

Conversation

@DeusData

Copy link
Copy Markdown
Owner

fix(coverage): stop walking the source per error region in the #1071 check (#1735)

cbm_subtract_macro_invocation_regions asked "is this region a call to a
file-defined macro?" before "is it inside a function?". The first question
(cbm_span_is_macro_invocation) walks the source from byte 0 to find the
region's lines, so every file with parse-error regions paid
regions x file bytes, whatever its language and whether or not it has a
single function. On a multi-MB SQL dump that is up to 256 walks over the
whole file.

Both predicates are pure, so asking the cheap one first changes no answer:
a region no function encloses never touches the source, and the regions
that do share one cbm_line_offsets table built on first use. The table
gives exactly the span the walk found; the walk stays as the fallback when
the table cannot be allocated.

#1071 behaviour is unchanged: same regions, same verdicts. Indexing this
repository's src/ before and after gives identical graphs (7,339 nodes,
28,924 edges) and the same 46 parse_partial files with byte-identical
ranges.

Tests count the bytes read, never time them (test-seam counter
cbm_test_macro_line_scan_bytes):

  • 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions
  • 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file
    size), still 60 regions

Refs #1735

fix(sql): keep literal INSERT rows out of the SQL parse (#1735)

A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of
INSERT ... VALUES (..),(..),... rows. Tree-sitter built a full tree for
every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL '
escapes, which the grammar does not know and error-recovers from) until the
5 s parse budget ran out, and the whole file, tables included, was skipped
as "parse timeout". The reporter's database/sql/world.sql was dropped this
way. The rows contribute nothing to the graph.

A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c)
finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the
first that consists only of literals: strings (standard '' and MySQL '
escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE,
FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier
or a quoted identifier stays in the parse, and so does one whose string
spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges
as the complement, so byte offsets and line/column positions of all kept
text are unchanged. The ranges are cleared right after the parse because
the parser is thread-local and reused. This is a syntax rule, not a work
cap: the same file always gives the same ranges, and the parse budget is
unchanged.

Measured on generated mysqldump-shaped dumps, release build, same machine,
index worker's own numbers (base = origin/main):

dump extraction pipeline peak RSS graph
8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical
8 MB MySQL ' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical
32 MB MySQL ' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed
59 MB MySQL ' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed

"identical" = every stored node and edge, Module end line included. For the
32/59 MB dumps the base run skipped the file as "parse timeout" (Module +
3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB
graph except for the Module's end line.

With standard escapes the extraction and the stored graph are identical
to the full parse. With MySQL ' escapes the full parse's error recovery
swallows rows next to a misread escape, a subquery row among them, and the
cut parse keeps those rows. So it loses nothing the full parse found
outside the dropped rows and recovers what the full parse lost. The only
things the full parse extracted INSIDE the dropped rows were junk usages
(DEFAULT and _binary read as identifiers, fragments of misread strings)
that name nothing.

Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the
exclusion off for one file so the same source can be compared both ways):

  • scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY
    UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions
  • extraction equals the full parse on a small dump (standard escapes);
    nothing lost outside dropped rows (MySQL escapes)
  • the subquery and function-call rows are still parsed
  • the tree does not grow when the row count doubles (a count, not a clock)
  • a 25 MB MySQL dump under the production budget is indexed, not
    "parse timeout" (asserts the outcome, not the time)
  • stored nodes and edges of a pipeline index equal the full parse's

Refs #1735


Part (c) of #1735, two commits: the #1071 macro-check cost fix, then the SQL literal-row exclusion. Part (a) is a separate PR.

…check (#1735)

cbm_subtract_macro_invocation_regions asked "is this region a call to a
file-defined macro?" before "is it inside a function?". The first question
(cbm_span_is_macro_invocation) walks the source from byte 0 to find the
region's lines, so every file with parse-error regions paid
regions x file bytes, whatever its language and whether or not it has a
single function. On a multi-MB SQL dump that is up to 256 walks over the
whole file.

Both predicates are pure, so asking the cheap one first changes no answer:
a region no function encloses never touches the source, and the regions
that do share one cbm_line_offsets table built on first use. The table
gives exactly the span the walk found; the walk stays as the fallback when
the table cannot be allocated.

#1071 behaviour is unchanged: same regions, same verdicts. Indexing this
repository's src/ before and after gives identical graphs (7,339 nodes,
28,924 edges) and the same 46 parse_partial files with byte-identical
ranges.

Tests count the bytes read, never time them (test-seam counter
cbm_test_macro_line_scan_bytes):
- 40 top-level junk regions: 39,870 bytes read -> 0, still 40 regions
- 60 in-body real errors: 85,725 bytes read -> 2,801 (one table = file
  size), still 60 regions

Refs #1735

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of
`INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for
every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \'
escapes, which the grammar does not know and error-recovers from) until the
5 s parse budget ran out, and the whole file, tables included, was skipped
as "parse timeout". The reporter's database/sql/world.sql was dropped this
way. The rows contribute nothing to the graph.

A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c)
finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the
first that consists only of literals: strings (standard '' and MySQL \'
escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE,
FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier
or a quoted identifier stays in the parse, and so does one whose string
spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges
as the complement, so byte offsets and line/column positions of all kept
text are unchanged. The ranges are cleared right after the parse because
the parser is thread-local and reused. This is a syntax rule, not a work
cap: the same file always gives the same ranges, and the parse budget is
unchanged.

Measured on generated mysqldump-shaped dumps, release build, same machine,
index worker's own numbers (base = origin/main):

  dump            extraction   pipeline    peak RSS   graph
  8 MB std ''     1310 ->  23 ms  1431 -> 139 ms   625 -> 37 MB   identical
  8 MB MySQL \'   2712 ->  24 ms  2827 -> 146 ms   474 -> 37 MB   identical
  32 MB MySQL \'  5145 ->  82 ms  5467 -> 396 ms  1132 -> 62 MB   timeout -> indexed
  59 MB MySQL \'  5130 -> 138 ms  5667 -> 674 ms  1157 -> 89 MB   timeout -> indexed

"identical" = every stored node and edge, Module end line included. For the
32/59 MB dumps the base run skipped the file as "parse timeout" (Module +
3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB
graph except for the Module's end line.

With standard escapes the extraction and the stored graph are identical
to the full parse. With MySQL \' escapes the full parse's error recovery
swallows rows next to a misread escape, a subquery row among them, and the
cut parse keeps those rows. So it loses nothing the full parse found
outside the dropped rows and recovers what the full parse lost. The only
things the full parse extracted INSIDE the dropped rows were junk usages
(DEFAULT and _binary read as identifiers, fragments of misread strings)
that name nothing.

Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the
exclusion off for one file so the same source can be compared both ways):
- scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY
  UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions
- extraction equals the full parse on a small dump (standard escapes);
  nothing lost outside dropped rows (MySQL escapes)
- the subquery and function-call rows are still parsed
- the tree does not grow when the row count doubles (a count, not a clock)
- a 25 MB MySQL dump under the production budget is indexed, not
  "parse timeout" (asserts the outcome, not the time)
- stored nodes and edges of a pipeline index equal the full parse's

Refs #1735

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData force-pushed the fix/issue-1735-sql-cost branch from 5ef8ccf to c0a06b3 Compare September 25, 2026 18:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant