Repository navigation
feat: syntax highlighting for Markdown code blocks and source files - #41
Open
pierreneter wants to merge 2 commits into
Open
pierreneter wants to merge 2 commits into
pierreneter wants to merge 2 commits into
Conversation
One DOM-free tokenizer in the core (src/core/syntax.ts) colors JavaScript / TypeScript / JSX / TSX, JSON, Python, CSS / SCSS, HTML / XML (scripts and styles in their language), YAML, shell, SQL, Go, Rust, Java, C / C++ / C#, Markdown, diff, TOML / INI, Mermaid, PlantUML and Graphviz, with the usual aliases. It reads line by line, carrying only what spans lines, every scanner moves forward and stops at its line's end, and lines already read are cached: the work is linear, an edit reads again only the lines it changed, and a text over 300,000 characters (a code block over 50,000, a line over 10,000) is plain. - Markdown code blocks become one code paragraph each, with the exact code and the fence's language, which travels as the paragraph style "Source Code (<language>)" (based on "Source Code", Pandoc's name for code blocks). The page draws the tokens over the runs (paper colors, readable in both themes); nothing is stored, so opening or showing a document is no edit, and a saved DOCX has no token colors. Editing keeps the colors; Enter is a new line in the block (an empty last line leaves it), pasted lines stay in it. Saved back as Markdown, an edited block is written with its fence and language; fences follow CommonMark (longer fences hold shorter ones). Diagrams are still drawn. - Source files (main.ts, script.py, config.json…, by extension, MIME type or the new `language` prop / attribute / open parameter) open in a code view: line numbers, theme colors, editable as text, saved as the plain text (the file's own bytes while not edited, its CRLF line breaks after an edit). - The source pane uses the same tokenizer: Markdown code blocks and web page scripts / styles are colored in their language. - @p10node/markdown: `highlight` option (off by default) wraps tokens in `<span class="hl-…">`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or tokens Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #32.
This adds language-aware syntax highlighting to fenced Markdown code blocks and to whole source files, using one shared tokenizer. It has no dependencies and no DOM.
Tokenizer:
src/core/syntax.tsIt replaces
src/react/source/highlight.ts, so the source pane now uses it too.<script>coloured as JS and<style>as CSS;js,ts,py,sh,bash,yml,c++,c#,rsandgolang.f(function) andy(type).Markdown code blocks
Source Code (<lang>). That style is based onSource Code, pandoc's style name for code blocks, which sets Courier New andw:noProof.pStylefor edited and split paragraphs, so the language survives an edited block.Paragraph.codefield from the style.<pre><code class="language-…">.de-tk-*spans, drawn inside the run's span at render time and cached per paragraph.w:colorand no run properties on code.Whole-file code view
PlainFormatvalue,'code'.languagehint wins over both. It is available as:OpenParams.language;DocViewerproplanguage;<office-viewer language>attribute.CodeView) is highlighted and readable in light and dark themes.Theme and packages
src-functionandsrc-type, in light and dark. New paper-colour tokens--de-code-*for code on the page; they stay the same in dark mode, because the page stays white.packages/markdown: a newhighlightoption, off by default, that emits<span class="hl-…">.Tests
test/syntax.test.ts(22 tests):test/code-blocks.test.ts(10 tests):languagehint and an unknown language;e2e/code.spec.ts(6 tests in Chromium):.tsand.pyfiles open in the code view, with original bytes saved, edit and save, and the Word and Markdown downloads;Results
In the Linux test VM with Bun 1.4.4:
bun testcode,viewer,docx,file-menu,embed,sdkbun run buildbun run build:packages@p10node/markdown54.1 KB gzip)The
docx,docx-qa2,docx-track,docx-table,securityandcolore2e specs passed (52 tests) on the commit before the last one. The last commit only changes two bounded regexes and a dependency list.After rebasing onto
af19290(which only changespackage.json), I re-ran typecheck and the new unit tests.Limitations
title="x"is kept only while the block is unedited);languagehint forces the code view for any text file,.mdand.htmlincluded. This is documented.<pre><code class="language-x">in HTML files does not become a code paragraph.Merging with #40 (Markdown math)
Both PRs touch the Markdown import and export. Merging them conflicts in these files:
README.mddocs/api.mddocs/features.mdpackages/markdown/README.mdpackages/markdown/src/index.tspackages/markdown/src/markdown.tssrc/core/convert.tssrc/core/plainmerge.tssrc/react/docx/render.tsxsrc/react/file/text.tsWhichever PR lands second needs a rebase onto the other.
🤖 Generated with Claude Code