mirror of
https://github.com/EKKOLearnAI/hermes-web-ui.git
synced 2026-07-17 22:20:19 +00:00
paulocavallari
016df1e8ad
* fix(context-compressor): guard token counting against quadratic BPE hang on long CJK runs
js-tiktoken's BPE merge loop is O(n^2) over the bytes of a single
pre-tokenizer piece. The GPT pat_str groups contiguous \p{L} characters
into one piece, and CJK text has no spaces, so a long CJK run becomes a
single huge piece. A ~14k-char CJK message (~42k bytes) produced ~1.8B
operations, pinning the Node event loop at 100% CPU. encode() never
throws in this case — it just hangs — so the existing catch-based
heuristic fallback never fired, and the whole server became unresponsive
(local HTTP requests timed out, nginx returned 504).
Detect a pathological contiguous letter/CJK run (>2000 chars) up front in
countTokens and countTokensForModel and use the cheap CJK-aware heuristic
instead. Normal text, even very long but space-separated, still takes the
exact tiktoken path.
Adds regression tests covering the CJK hang, normal text, and long
space-separated text.
* docs: add chat-chain-changes fragment for CJK token-counting guard
---------
Co-authored-by: paulocavallari <paulocavallari@users.noreply.github.com>