Skip to content

feat: syntax highlighting for Markdown code blocks and source files - #41

Open
pierreneter wants to merge 2 commits into
mainfrom
feat/syntax-highlighting
Open

pierreneter wants to merge 2 commits into
mainfrom
feat/syntax-highlighting

Conversation

@pierreneter

Copy link
Copy Markdown
Member

Fixes #32.

This adds language-aware syntax highlighting to fenced Markdown code blocks and to whole source files, using one shared tokenizer. It has no dependencies and no DOM.

Tokenizer: src/core/syntax.ts

It replaces src/react/source/highlight.ts, so the source pane now uses it too.

  • Languages:
    • JS / TS / JSX / TSX, including nested template literals, regex literals and JSX;
    • JSON, including comments;
    • Python;
    • CSS / SCSS / Less;
    • HTML / XML, with <script> coloured as JS and <style> as CSS;
    • YAML;
    • shell, including here-docs;
    • SQL, Go, Rust, Java, C, C++, C#;
    • Markdown, with each fenced block in its own language;
    • diff, TOML / INI;
    • Mermaid, PlantUML and Graphviz, moved over from the old file with their regexes now bounded.
  • Aliases: the usual ones, such as js, ts, py, sh, bash, yml, c++, c#, rs and golang.
  • Unknown language: shown as plain code.
  • Token classes: the existing one-letter set, plus f (function) and y (type).
  • Bounded work:
    • it reads line by line, carrying a short state string across lines;
    • lines already read are cached, so an edit re-reads only the lines it changed;
    • text past 300k characters, code blocks past 50k, and lines past 10k (minified code) are shown plain;
    • nesting depth is capped at 32.
  • Speed: 238 kB of TypeScript takes 53 ms cold and 5–7 ms warm.

Markdown code blocks

  • One paragraph per block. Each fenced block becomes a single paragraph whose style is Source Code (<lang>). That style is based on Source Code, pandoc's style name for code blocks, which sets Courier New and w:noProof.
    • The language travels as the paragraph style. The writer already keeps pStyle for edited and split paragraphs, so the language survives an edited block.
    • The parser sets a new Paragraph.code field from the style.
  • Exact text. The block keeps its exact code and whitespace. Fences follow CommonMark:
    • a longer fence holds shorter ones;
    • a fence line that has a language never closes a block;
    • the fence's indentation is removed from the code lines.
  • Saving.
    • An unedited Markdown file is saved byte-identical.
    • An edited block is written as a fence with its language. When the code contains backticks, a longer fence is used.
    • In HTML export a block becomes <pre><code class="language-…">.
  • Colour is presentation only.
    • Colours are nested de-tk-* spans, drawn inside the run's span at render time and cached per paragraph.
    • They are never stored in runs: opening or repainting does not mark the document dirty and adds no undo step.
    • A saved DOCX has no w:color and no run properties on code.
  • Visual editing keeps working inside a block:
    • Enter inserts a new line, indented like the previous one;
    • Enter on an empty last line leaves the block;
    • pasted lines stay inside the block.
  • Diagrams: Mermaid, PlantUML and Graphviz blocks are still drawn as diagrams.

Whole-file code view

  • Detection: source files are detected by extension or MIME type, as a new PlainFormat value, 'code'.
  • The language hint wins over both. It is available as:
    • OpenParams.language;
    • the DocViewer prop language;
    • the <office-viewer language> attribute.
  • The view (CodeView) is highlighted and readable in light and dark themes.
    • It is read-only in view mode and editable in edit mode, but read-only during collaboration.
  • Saving:
    • with no edit, the original bytes are saved (BOM and CRLF kept);
    • after an edit, the plain source is saved, with CRLF kept when the file used it.
  • Exports: Download as Word, print / PDF / HTML and text all work.
  • HTML, SVG and Markdown keep their rendered view. Their existing source pane now uses the shared highlighter.

Theme and packages

  • Theme: new tokens src-function and src-type, in light and dark. New paper-colour tokens --de-code-* for code on the page; they stay the same in dark mode, because the page stays white.
  • packages/markdown: a new highlight option, off by default, that emits <span class="hl-…">.

Tests

  • test/syntax.test.ts (22 tests):
    • every language and the aliases;
    • unterminated strings and comments;
    • nested template literals, and JSX versus TS generics;
    • tokens always rebuild the exact text on random inputs in all 25 languages;
    • the size and depth limits;
    • a time bound on 20 pathological inputs × 25 languages;
    • the line cache.
  • test/code-blocks.test.ts (10 tests):
    • language and whitespace preserved, and the fence rules;
    • the DOCX holds the style and no colours;
    • not dirty after open; unedited Markdown saved byte-identical;
    • an edited block keeps its fence and language through save and reopen;
    • original bytes with BOM and CRLF;
    • the language hint and an unknown language;
    • the package option.
  • e2e/code.spec.ts (6 tests in Chromium):
    • a Markdown block shows coloured tokens; an unknown language is plain; the Mermaid block is still drawn;
    • in dark theme, contrast is above 4.5;
    • opening adds no dirty event, and Ctrl+Z changes nothing;
    • editing inside a block keeps the colours and saves the right Markdown;
    • .ts and .py files open in the code view, with original bytes saved, edit and save, and the Word and Markdown downloads;
    • the SQL hint, an unknown language, CRLF / BOM, and the embed attribute.

Results

In the Linux test VM with Bun 1.4.4:

Check Result
typecheck clean
bun test 1224 pass, 0 fail
e2e code, viewer, docx, file-menu, embed, sdk 89 pass, 0 fail
bun run build ok
bun run build:packages ok (@p10node/markdown 54.1 KB gzip)

The docx, docx-qa2, docx-track, docx-table, security and color e2e specs passed (52 tests) on the commit before the last one. The last commit only changes two bounded regexes and a dependency list.

After rebasing onto af19290 (which only changes package.json), I re-ran typecheck and the new unit tests.

Limitations

  • Edited blocks:
    • only the first word of the info string survives (title="x" is kept only while the block is unedited);
    • an edited block nested in a list is written back at the top level.
  • Any DOCX style named "Source Code" (for example in pandoc output) is now treated as a code block.
  • The code view is not virtualised. Files past 300k characters are shown plain, though still editable.
  • The language hint forces the code view for any text file, .md and .html included. This is documented.
  • Long code blocks are not split across pages.
  • Code blocks have no background shading, only colour.
  • HTML import is unchanged: <pre><code class="language-x"> in HTML files does not become a code paragraph.
  • The tokenizer is heuristic for regex and JSX detection, CSS nesting, and unquoted YAML values.

Merging with #40 (Markdown math)

Both PRs touch the Markdown import and export. Merging them conflicts in these files:

  • README.md
  • docs/api.md
  • docs/features.md
  • packages/markdown/README.md
  • packages/markdown/src/index.ts
  • packages/markdown/src/markdown.ts
  • src/core/convert.ts
  • src/core/plainmerge.ts
  • src/react/docx/render.tsx
  • src/react/file/text.ts

Whichever PR lands second needs a rebase onto the other.

🤖 Generated with Claude Code

pierreneter and others added 2 commits November 20, 2026 12:16
One DOM-free tokenizer in the core (src/core/syntax.ts) colors JavaScript /
TypeScript / JSX / TSX, JSON, Python, CSS / SCSS, HTML / XML (scripts and
styles in their language), YAML, shell, SQL, Go, Rust, Java, C / C++ / C#,
Markdown, diff, TOML / INI, Mermaid, PlantUML and Graphviz, with the usual
aliases. It reads line by line, carrying only what spans lines, every scanner
moves forward and stops at its line's end, and lines already read are cached:
the work is linear, an edit reads again only the lines it changed, and a text
over 300,000 characters (a code block over 50,000, a line over 10,000) is plain.

- Markdown code blocks become one code paragraph each, with the exact code and
  the fence's language, which travels as the paragraph style "Source Code
  (<language>)" (based on "Source Code", Pandoc's name for code blocks). The
  page draws the tokens over the runs (paper colors, readable in both themes);
  nothing is stored, so opening or showing a document is no edit, and a saved
  DOCX has no token colors. Editing keeps the colors; Enter is a new line in the
  block (an empty last line leaves it), pasted lines stay in it. Saved back as
  Markdown, an edited block is written with its fence and language; fences
  follow CommonMark (longer fences hold shorter ones). Diagrams are still drawn.
- Source files (main.ts, script.py, config.json…, by extension, MIME type or
  the new `language` prop / attribute / open parameter) open in a code view:
  line numbers, theme colors, editable as text, saved as the plain text (the
  file's own bytes while not edited, its CRLF line breaks after an edit).
- The source pane uses the same tokenizer: Markdown code blocks and web page
  scripts / styles are colored in their language.
- @p10node/markdown: `highlight` option (off by default) wraps tokens in
  `<span class="hl-…">`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or tokens

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Viewer: syntax highlighting for Markdown code blocks and source files

1 participant