Skip to content

fix(grammar): accept a lone '&' in JSX strings and text (#1736) #2327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DeusData wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(grammar): accept a lone '&' in JSX strings and text (#1736) #2327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
098fcc8
fix(grammar): accept a lone '&' in JSX strings and text (#1736)
DeusData Sep 25, 2026
f5c8631
fix(security): refresh the vendored checksum of grammars/MANIFEST.md
DeusData Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions THIRD_PARTY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,22 @@ License summary:
smart-coin language, written for this project because no usable public
grammar exists; its source and corpus tests live in
`tools/tree-sitter-chialisp/`.
- Seven further grammars (`arkts`, `assembly`, `cfml`,
`cfscript`, `dotenv`, `pine`, `qml`) are self-maintained forks that retain
their original upstream authors' licenses — see the manifest for per-grammar
provenance. `arkts` is a first-party derivative of
- Nine further grammars (`arkts`, `assembly`, `cfml`,
`cfscript`, `dotenv`, `javascript`, `pine`, `qml`, `tsx`) are self-maintained
forks that retain their original upstream authors' licenses — see the manifest
for per-grammar provenance. `arkts` is a first-party derivative of
[tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)
(MIT, (c) 2017 Max Brunsfeld; on the
[tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript)
base, MIT, (c) 2014 Max Brunsfeld) with (c) 2026 DeusData ArkTS additions;
its grammar source lives in `tools/tree-sitter-arkts/`.
its grammar source lives in `tools/tree-sitter-arkts/`. `javascript`
([tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript),
MIT, (c) 2014 Max Brunsfeld) and `tsx`
([tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s
`tsx` dialect, MIT, (c) 2017 Max Brunsfeld, on the same javascript base) are
the upstream grammars with one (c) 2026 DeusData patch that lets a lone `&`
appear in JSX strings and text; their grammar sources live in
`tools/tree-sitter-javascript/` and `tools/tree-sitter-tsx/`.

### tree-sitter-plsql

Expand Down
10 changes: 6 additions & 4 deletions internal/cbm/vendored/grammars/MANIFEST.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r

## Summary

- Grammars: **162** — vendored-from-upstream: **143**, first-party/self-maintained: **14**, registry-disagreement: **5** (nim removed 2026-06-12; objectscript_udl + objectscript_routine added 2026-06-24; mojo added 2026-07-01; arkts added 2026-08-26; plsql added 2026-08-27; chialisp added 2026-08-28 — see notes below)
- ABI distribution: **9×** ABI-13 **79×** ABI-14 **74×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
- Grammars: **162** — vendored-from-upstream: **141**, first-party/self-maintained: **16**, registry-disagreement: **5** (nim removed 2026-06-12; objectscript_udl + objectscript_routine added 2026-06-24; mojo added 2026-07-01; arkts added 2026-08-26; plsql added 2026-08-27; chialisp added 2026-08-28; javascript + tsx became self-maintained forks 2026-09-25 — see notes below)
- ABI distribution: **9×** ABI-13 **78×** ABI-14 **75×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
— recounted from the tree 2026-09-25 after the tsx fork was regenerated with tree-sitter-cli 0.25.10, moving `tsx` from ABI 14 to ABI 15 (was `9×/79×/74×`).
— recounted from the tree 2026-08-30 after the perl v1.2.1 refresh moved `perl` from ABI 14 to ABI 15 (was `9×/80×/73×`). Neither side of the rebase had this right: main's line was correct for main, and this branch still carried the pre-2026-08-28 `7×/84×/65×`. Regenerate, never increment.
— recounted from the tree 2026-08-28. This line had drifted: it read `7×/86×/65×`, which sums to 158 against 161 vendored grammars, so it was wrong before Chialisp was added and incrementing it would have carried the error forward. Regenerate with:
`grep -h '#define LANGUAGE_VERSION' internal/cbm/vendored/grammars/*/parser.c | sort | uniq -c`
Expand All @@ -32,6 +33,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c`/`scanner.c` are otherwise byte-for-byte upstream — on re-vendor, re-apply only that single include rename. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
- **mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored.
- **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. **LICENSE:** tree-sitter-typescript's MIT text **verbatim and byte-identical** (© 2017 Max Brunsfeld) — which is what MIT requires of a derivative work, and what lets the provenance audit byte-verify it against upstream rather than take a note on trust. The fork is registered in the audit's `FORKS` map. Our own copyright for the ArkTS additions, and the tree-sitter-javascript attribution (© 2014 Max Brunsfeld), live with the SOURCE in `tools/tree-sitter-arkts/grammar.js` and in `THIRD_PARTY.md`, which is what ships in the release archives.
- **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&amp;`/`&#38;` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
- **plsql** (added 2026-08-27): vendored from [AndreasMaierDe/tree-sitter-plsql](https://github.com/AndreasMaierDe/tree-sitter-plsql) @ `28aebef209be` (full: `28aebef209be57169600e1aa41ca1431cc6c916f`, upstream tip; repo dormant since 2023-02) — MIT, ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`). Not listed in nvim-treesitter/Helix (`community-niche`, an individual-maintained grammar); provenance verified directly against upstream (pin = `git ls-remote` HEAD; vendored `parser.c` byte-identical to the pinned clone apart from the include-quote local patch below; 0 non-ASCII bytes; 0 dangerous calls). Security review covered only the vendored C surface (`parser.c`, `tree_sitter/parser.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored. Known upstream limitation: `CREATE TYPE ... AS OBJECT` currently yields ERROR nodes (pinned by `plsql_create_type_as_object_limitation` in `tests/test_extraction.c` + `tests/fixtures/plsql/create_type_as_object_limitation.tps`). Originally contributed as PR #1033 by Oğuz (@ouzsrcm); re-vendored from upstream per vendoring policy with the PR's language wiring distilled on top.

- **chialisp** (added 2026-08-28): **first-party** — authored in this repository, not vendored from anywhere. Grammar source + corpus tests live in `tools/tree-sitter-chialisp/`; regenerate with `npx tree-sitter-cli@0.25.10 generate --abi 14` and copy `src/parser.c` + `src/tree_sitter/*.h` here. ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`), 0 non-ASCII bytes. It is a deliberately GENERIC s-expression grammar (`source_file`/`list`/`symbol`/`string`/`number`/`hex`/`dot`/`comment`) modelling the clvm_tools reader rather than the Chialisp form vocabulary: `mod`/`defun`/`defconstant`/`include` are ordinary head symbols, and which lists are definitions is decided in `internal/cbm/extract_defs.c`, so a dialect that adds a form does not need a regenerated parser. Written because the only public grammar (`Quexington/tree-sitter-chialisp`) cannot parse the language: it required CRLF to terminate a comment (`/;.*\r\n/`) while real files are LF, rejected the `.` in `(include foo.clib)`, and accepted only a primitive after `(defconstant NAME …)` — each of which desynchronised the rest of the file. Acceptance gate: all five `chia-blockchain@main` reference files parse with **zero ERROR and zero MISSING nodes**. **LICENSE:** the project's own LICENSE, byte-identical to the repository root — no third-party copyright is carried, because there is no third party.
Expand Down Expand Up @@ -151,7 +153,6 @@ row instead.
| ini | 15 | justinmk/tree-sitter-ini | `e4018b517613` | VERIFIED-BOTH | ✅ |
| ispc | 14 | tree-sitter-grammars/tree-sitter-ispc | `9b2f9aec2106` | VERIFIED-NVIM | ✅ |
| java | 14 | tree-sitter/tree-sitter-java | `e10607b45ff7` | VERIFIED-BOTH | ✅ |
| javascript | 15 | tree-sitter/tree-sitter-javascript | `58404d8cf191` | VERIFIED-BOTH | ✅ |
| jsdoc | 15 | tree-sitter/tree-sitter-jsdoc | `658d18dcdddb` | VERIFIED-BOTH | ✅ |
| json | 14 | tree-sitter/tree-sitter-json | `001c28d7a298` | VERIFIED-BOTH | ✅ |
| json5 | 15 | Joakker/tree-sitter-json5 | `aa630ef48903` | VERIFIED-BOTH | ✅ |
Expand Down Expand Up @@ -225,7 +226,6 @@ row instead.
| thrift | 14 | tree-sitter-grammars/tree-sitter-thrift | `68fd0d80943a` | VERIFIED-BOTH | ✅ |
| tlaplus | 14 | tlaplus-community/tree-sitter-tlaplus | `add40814fda3` | VERIFIED-BOTH | ✅ |
| toml | 14 | tree-sitter-grammars/tree-sitter-toml | `64b56832c2cf` | MISMATCH | ✅ |
| tsx | 14 | tree-sitter/tree-sitter-typescript | `75b3874edb2d` | VERIFIED-BOTH | ✅ |
| typescript | 14 | tree-sitter/tree-sitter-typescript | `75b3874edb2d` | VERIFIED-BOTH | ✅ |
| typst | 14 | uben0/tree-sitter-typst | `46cf4ded12ee` | VERIFIED-BOTH | ✅ |
| verilog | 14 | tree-sitter/tree-sitter-verilog | `4457145e795b` | VERIFIED-HELIX | ✅ |
Expand Down Expand Up @@ -264,6 +264,8 @@ upstream author (correctly retained). The table now records the true origin.
| grammar | cur ABI | original upstream | license |
|---|:---:|---|---|
| arkts | 15 | **self-maintained fork** of tree-sitter/tree-sitter-typescript @ `75b3874edb2d` (v0.23.2; javascript base @ `3a837b6f3658` v0.23.1); grammar source in `tools/tree-sitter-arkts/` | MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; ArkTS additions (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
| javascript | 15 | **self-maintained fork** of tree-sitter/tree-sitter-javascript @ `58404d8cf191` (the previously vendored pin, VERIFIED-BOTH) + the JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-javascript/` | MIT, (c) 2014 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
| tsx | 15 | **self-maintained fork** of tree-sitter/tree-sitter-typescript's `tsx` dialect @ `75b3874edb2d` (v0.23.2, the previously vendored pin, VERIFIED-BOTH; javascript base @ `3a837b6f3658` v0.23.1) + the same JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-tsx/` | MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
| assembly | 14 | RubixDev/tree-sitter-assembly (**repo deleted from GitHub** — our retained MIT copy, (c) 2023 RubixDev, is the surviving grant) | MIT |
| cfml | 15 | cfmleditor/tree-sitter-cfml | MIT, (c) 2014 Gareth Edwards & Gavin Baumanis — byte-identical |
| cfscript | 15 | cfmleditor/tree-sitter-cfml | MIT, same — byte-identical |
Expand Down
Loading
Loading