Skip to content

fix: respect CommonMark tilde code fences and prevent silent AST truncation - #2

Merged
doug merged 1 commit into
mainfrom
fix-tilde-prose-and-parser-eof
Sep 8, 2026
Merged

doug merged 1 commit into
mainfrom
fix-tilde-prose-and-parser-eof

Conversation

@doug

@doug doug commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Summary

This PR addresses two parser/expander bugs that cause math suppression and text loss:

  1. Inline Tildes Suppressing Math Expansion (markdown.go):

    • Issue: termtex.Expand previously treated any unescaped tilde (~) identically to a backtick code span opener (inCode = true). Any single tilde in prose (e.g. ~1,110 tasks) or GFM strikethrough (~~text~~) suppressed math expansion ($...$) for the remainder of the text.
    • Fix: In CommonMark (§4.5), tildes only open code blocks when a run of 3 or more tildes appears at the start of a line (indented $\le 3$ spaces). We disentangle backtick and tilde handling, add isFenceLineStart and isFenceLineEnd, and validate that opening info strings cannot contain tilde characters.
  2. Silent AST Truncation on Incomplete Parses (parser.go):

    • Issue: If an expression had unparseable tokens, an unmatched \end{...}, or dangling delimiters, parseRows() stopped early but parse() returned err == nil. Render then rendered only the leading prefix, silently dropping the rest of the expression.
    • Fix: In parse(), assert that all tokens have been consumed (p.peek().typ == tokEOF). If unconsumed tokens remain, return an error. This causes Expand to fall back to preserving the raw markdown verbatim rather than emitting a truncated fragment.

Tests

  • TestExpandTildeInProse: covers single tildes in prose, ranges, strikethrough, comments inside fences, and indentation.
  • TestExpandPreservesUnparseableMathVerbatim: verifies Expand preserves unparseable math verbatim.
  • TestUnconsumedTokensError: verifies parse() errors on dangling tokens.

…cation

- In CommonMark, tildes (~) only open fenced code blocks when a run of 3 or more tildes appears at the start of a line (indented <= 3 spaces). Previously, any unescaped single tilde in prose (e.g. "~1,110 tasks") or strikethrough was treated as an inline code span opener, suppressing inline math expansion across subsequent prose. Disentangle backticks and tildes, add isFenceLineStart and isFenceLineEnd to prevent premature closure on comments, and validate info strings.

- In parser.go, assert all tokens are consumed in parse() (p.peek().typ == tokEOF). Previously, dangling unconsumed tokens or unmatched \end constructs caused parse() to succeed with a partial AST, silently dropping the rest of the expression. Failing with an error allows Expand to preserve raw markdown verbatim rather than emitting a truncated fragment.

- Add regression tests covering inline tildes, tilde ranges, strikethrough, comments inside fences, and unconsumed token error handling.
@doug
doug merged commit 6ea3f30 into main Sep 8, 2026
2 checks passed
@doug
doug deleted the fix-tilde-prose-and-parser-eof branch September 8, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant