Conversation
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished indexing: `.res` maps to ReScript, so the resource was parsed as ReScript source and the definitions pass spun. Three defects, each fixed: 1. Quadratic usage walk (the hang). The ReScript `let` binding check (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript and Nickel) climbed from every named leaf with ts_node_parent, which descends from the root and scans each level's children. A binary file parses into a flat error tree (~170k root children per MB), so each climb cost O(root children) and the file O(n^2): 1 MB spent 46 s in cbm_extract_unified, 4.7 MB never returned. The check now climbs the unified walk's own cursor (O(1) per hop), like the other occurrence classifiers; the node-based climb stays only for callers outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end. 2. Quadratic lexing in error recovery. tree-sitter retries the lexer at every byte of an unparseable stretch with every external token valid; the ReScript scanner's template-string loop then ran to the next backtick/$/backslash/NUL (end of file if none) and discarded the result. That cost O(n^2) inside lexing, where the parse budget's progress callback never runs, so such files were dropped by the 5 s CPU budget instead of parsed -- plain text too (100 KB of non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the template branch when NEWLINE is also valid, which only the ERROR state allows. Valid files parse to identical trees. Recorded in vendored/grammars/MANIFEST.md. 3. Binary content parsed as source. `.res` is also Godot's binary resource format and the Windows compiled-resource format. Following the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the first 4 KB and reports a file holding a NUL byte as unsupported; ReScript text never contains one. Tests (deterministic work counters, no clocks): - extraction::extract_rescript_let_bindings_use_the_walk_cursor -- zero root-descending parent hops at 128 and 1024 statements (before: 515 / 4099). - complexity::complexity_rescript_error_recovery_lexing_is_linear -- lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio must be linear (before 3.99, after 2.00). - language::lang_res_binary_resource_unsupported and lang_res_rescript_stays_rescript. Fixes #2176 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2 tasks
…anner patch security-static (scripts/security-vendored.sh) failed on this branch: the #2176 fix patches internal/cbm/vendored/grammars/rescript/scanner.c (skip the template-string branch in the error-recovery state) and records that patch in vendored/grammars/MANIFEST.md, but the checked-in digests in scripts/vendored-checksums.txt still described the unpatched files. Both changes are intentional and reviewed in the parent commit; this records their SHA-256 via `scripts/security-vendored.sh --update`. Only the two affected lines change, and the integrity check passes again. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished
indexing:
.resmaps to ReScript, so the resource was parsed as ReScriptsource and the definitions pass spun. Three defects, each fixed:
Quadratic usage walk (the hang). The ReScript
letbinding check(is_first_named_part_of, shared with Julia, Typst, Elm, PureScript
and Nickel) climbed from every named leaf with ts_node_parent, which
descends from the root and scans each level's children. A binary
file parses into a flat error tree (~170k root children per MB), so
each climb cost O(root children) and the file O(n^2): 1 MB spent
46 s in cbm_extract_unified, 4.7 MB never returned. The check now
climbs the unified walk's own cursor (O(1) per hop), like the other
occurrence classifiers; the node-based climb stays only for callers
outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end.
Quadratic lexing in error recovery. tree-sitter retries the lexer at
every byte of an unparseable stretch with every external token
valid; the ReScript scanner's template-string loop then ran to the
next backtick/$/backslash/NUL (end of file if none) and discarded
the result. That cost O(n^2) inside lexing, where the parse budget's
progress callback never runs, so such files were dropped by the
5 s CPU budget instead of parsed -- plain text too (100 KB of
non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the
template branch when NEWLINE is also valid, which only the ERROR
state allows. Valid files parse to identical trees. Recorded in
vendored/grammars/MANIFEST.md.
Binary content parsed as source.
.resis also Godot's binaryresource format and the Windows compiled-resource format. Following
the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the
first 4 KB and reports a file holding a NUL byte as unsupported;
ReScript text never contains one.
Tests (deterministic work counters, no clocks):
zero root-descending parent hops at 128 and 1024 statements
(before: 515 / 4099).
lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio
must be linear (before 3.99, after 2.00).
lang_res_rescript_stays_rescript.
Fixes #2176
Known leftover (not in this PR): the same root-descending parent climb remains in other is_policy_binding branches (agda, llvm, wolfram, tlaplus, elixir, lisp, …) in extract_usages.c; a generic cursor-ancestor iterator would fix them all.