Conversation
A plain '&' inside JSX -- a URL query in an attribute (href="https://fonts.googleapis.com/css2?family=Inter&display=swap"), or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node in .js, .jsx and .tsx files. The file was flagged parse_partial in the coverage view, and the error recovery turned words from the URL (create-next-app -> create/next/app) into identifiers that resolved to unrelated symbols as false USAGE edges. Root cause is upstream tree-sitter-javascript, which the tsx dialect builds on: unescaped_{double,single}_jsx_string_fragment is /([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit or as a complete html_character_reference (& / &). jsx_text is scanned by the external scanner, which stops at every '&', and nothing in _jsx_child accepted a '&' that is not a reference. So '&' + letter in a string, and any lone '&' in JSX text, had no valid token. Fix: javascript and tsx become self-maintained forks of the pins we already vendored (javascript @ 58404d8cf191; typescript tsx @ 75b3874edb2d on its javascript base @ 3a837b6f3658), following the arkts model. One patch (tools/tree-sitter-javascript/patches/ jsx-lone-ampersand.patch) is applied to both javascript bases: - a one-character '&' token at lexical precedence -1, aliased to the existing string_fragment (in strings) and jsx_text (in text); the lower precedence keeps & / & lexing as html_character_reference - the string-fragment regexes exclude a '&' directly before the closing quote, so "a&" no longer runs past the end of the string Node types are unchanged (same type/named set). The external scanners stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10 (ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match arkts). Without the patch both fork dirs regenerate upstream exactly. The upstream corpus passes unchanged: javascript 116/116; tsx gives the same results as unpatched upstream. Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the two left are Babel do-expressions and Flow types), no newly flagged file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the 115 USAGE edges removed were all false cross-example edges from URL fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle from the corrected ASTs. Registered like arkts: MANIFEST.md fork rows + notes (ABI tally recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums. LICENSEs stay upstream's MIT byte-identical. Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted attribute, '&' before the quote, '&' + letter/space/digit in text, plus &/& controls) and a guard that broken JSX is still flagged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This was referenced Sep 25, 2026
The fork notes for javascript + tsx changed MANIFEST.md, which is covered by scripts/vendored-checksums.txt; its entry still held the pre-change hash, so Layer 8 (security-vendored.sh) failed. Regenerated with --update; the only line that changed is MANIFEST.md. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as
Tom & Jerry
-- produced a tree-sitter ERROR nodein .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.
Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.
Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
existing string_fragment (in strings) and jsx_text (in text); the
lower precedence keeps & / & lexing as html_character_reference
quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.
Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.
Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.
Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.
Grammar fork of
javascript+tsxfollowing the existing arkts self-maintained-fork model.scripts/audit-license-provenance.pywas not run locally (it shells out to baregh); please let CI run it.Fixes #1736
Also fixes #2163 (same construct: a raw
&in a quoted JSX prop, e.g.title="P&L Report"; verified on this branch: parse_partial 1 → 0 with an identical graph).