Skip to content

fix(grammar): accept a lone '&' in JSX strings and text (#1736) - #2327

Open
DeusData wants to merge 2 commits into
mainfrom
fix/issue-1736
Open

DeusData wants to merge 2 commits into
mainfrom
fix/issue-1736

Conversation

@DeusData

@DeusData DeusData commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as

Tom & Jerry

-- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.

Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.

Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:

  • a one-character '&' token at lexical precedence -1, aliased to the
    existing string_fragment (in strings) and jsx_text (in text); the
    lower precedence keeps & / & lexing as html_character_reference
  • the string-fragment regexes exclude a '&' directly before the closing
    quote, so "a&" no longer runs past the end of the string
    Node types are unchanged (same type/named set). The external scanners
    stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
    (ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
    arkts). Without the patch both fork dirs regenerate upstream exactly.
    The upstream corpus passes unchanged: javascript 116/116; tsx gives the
    same results as unpatched upstream.

Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.

Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.

Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.


Grammar fork of javascript + tsx following the existing arkts self-maintained-fork model. scripts/audit-license-provenance.py was not run locally (it shells out to bare gh); please let CI run it.

Fixes #1736

Also fixes #2163 (same construct: a raw & in a quoted JSX prop, e.g. title="P&L Report"; verified on this branch: parse_partial 1 → 0 with an identical graph).

A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.

Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (&amp; / &#38;). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.

Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
- a one-character '&' token at lexical precedence -1, aliased to the
  existing string_fragment (in strings) and jsx_text (in text); the
  lower precedence keeps &amp; / &#38; lexing as html_character_reference
- the string-fragment regexes exclude a '&' directly before the closing
  quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.

Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.

Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.

Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&amp;/&#38; controls) and a guard that broken JSX is still flagged.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fork notes for javascript + tsx changed MANIFEST.md, which is covered by
scripts/vendored-checksums.txt; its entry still held the pre-change hash, so
Layer 8 (security-vendored.sh) failed. Regenerated with --update; the only
line that changed is MANIFEST.md.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Indexing gap: src/app Indexing gap: app/layout.tsx

1 participant