v4.5.0: dark mode, text darkness, X3 polish, stability + bookshelf cleanup#39
Merged
Conversation
…-flight Two atlas-pipeline bugs caught when prebake'd glyph atlases shipped to the device produced unreadable rendering: 1. GlyphAtlas.h rowBytes/glyphBytes assumed byte-aligned-per-row but the on-device renderer (GfxRenderer::renderCharImpl) reads the bitmap as one continuous bitstream -- pixel index runs 0..(width*height-1) and the byte address is pixelPosition >> 3. Per-row alignment misaligned every row past the first for any glyph whose width is not a multiple of 8 (most glyphs in a proportional font), giving a "ghost stroke per character" rendering artifact. Switch the builder to continuous packing matching the renderer's blit loop. atlas size shrinks ~3% as a side effect. 2. tools/crumble-prebake/src/main.cpp silently produced "section N wrote 1 pages" for SD-font books when the user forgot --sd-font-path. Root cause: section settings declared an SD-font fontId, the renderer's fontMap did not contain it, every glyph metric (advanceX, advanceY, ascender) returned 0, addLineToPage's `currentPageNextY + 0 > viewportHeight` never fired, all chapter content stacked onto page 1 with zero-height lines. Add a pre-flight check at the top of prebakeSections: if s.fontId is not in renderer.getFontMap(), refuse the build with an error message naming the missing flag. Bump the call-site failure log so the CLI exits non-zero instead of swallowing the hard failure as a "skipped".
…lback Three reader-rendering improvements to fix '?' glyphs that the SD-font glyph atlas could not cover: 1. Parser-level typography substitution (ChapterHtmlSlimParser). Smart quotes / dashes / ellipsis / bullets in the 0xE2 0x80 UTF-8 block get replaced with ASCII equivalents at parse time, so the stored page text -- and the prewarmed glyph atlas baked from it -- only contain ASCII codepoints fonts reliably ship. Without this step the renderer-level aliasCodepoint() had no atlas target to resolve to because the chapter never used the ASCII form directly. Same parser runs on host (prebake CLI) and device (live rebuild), keeping the substitution consistent. 2. Renderer aliasCodepoint() table expanded (EpdFontData.h). Adds smart-quote / dash / ellipsis / decoration (diamond / star / bullet) substitutions as a runtime safety net, useful when the parser-level substitution did not run (e.g. live UI strings, dictionary entries). 3. getFallbackCodepoint() cascade fix (EpdFontFamily.cpp). Two missing fallback steps previously short-circuited to REPLACEMENT_GLYPH: (a) when both cp and alias miss findGlyphData, give the SD font's miss handler a chance to lazy-load the alias target from the .cpfont full glyph table; (b) for non-regular styles, probe the regular chain so bold/italic text rendered against an atlas that only carries the REGULAR style (the prebake's prewarm default) falls back to the regular glyph instead of '?'. Chapter titles in bold were the dominant visible symptom -- this fixes them at the cost of losing the bold weight (text reads as regular).
Two BT stability fixes addressing reliability of first quick-connect and post-connect MaxAlloc collapse: 1. BluetoothHIDManager always retries the initial connect with a fresh client after a ~300 ms cool-down. Previously the retry path only fired when hadExistingClient was true and ran instantly. User pattern was reliable: first connect attempt times out (controller still discovering / advertising and our scan window misses it), second one succeeds. Make that retry unconditional and add the short cool-down so RF state settles. 2. NimBLE config round 2 (platformio.ini): - ATT_PREFERRED_MTU: 64 -> 23 (BT spec minimum; HID does not benefit) - MSYS1_BLOCK_COUNT: 4 -> 3 - TRANSPORT_ACL_FROM_LL_COUNT: 4 -> 2 - TRANSPORT_EVT_COUNT: 6 -> 4 - TRANSPORT_EVT_DISCARD_COUNT: 2 -> 1 - HOST_TASK_STACK_SIZE: 4096 -> 3072 (new) Reclaims roughly 6-8 KB of static NimBLE budget. Trade-off surfaces as "missed page-turn notify" not "crash" under rapid button presses, which is the right direction to fail.
Four reader-activity changes addressing memory pressure and UX bugs surfaced during v4.4 atlas testing: 1. EpubReaderActivity lazy-reload skips embedded-subset install when the v40 atlas is already covering the section. atlasOk previously only flipped true when the lazy reload itself installed the atlas; when the atlas was installed at section-open and the lazy reload only needed to install the subset, the subset install fired unconditionally and burned ~6 KB of MaxAlloc on redundant data. Track atlasUsable = atlasOk || section->glyphAtlasInstalled() and only fall through to subset install when no atlas data exists. 2. EpubReaderActivity prebake-mismatch dialog: treat the Cancel / Back button as "return to library" rather than silently falling through to "Keep current settings" and entering the book with mismatched settings. finish() pops the reader activity so the library renders again immediately. The two visible buttons keep their semantics; only the implicit back-out semantics change. 3. EpubReaderMenuActivity hides Lookup / Looked-Up Words / Add Highlight / Finish Highlight / Cancel Highlight entries when a BLE remote is currently active. NimBLE pins ~58 KB of fragmented heap while connected; the lookup word-select pass and highlight word-walk both allocate WordInfo vectors that bad_alloc'd under tight MaxAlloc, then dragged the auto-disable-BT recovery into a fragile reconnect race that aborted with maxAlloc=5 KB. Hiding the entries is the honest tradeoff: user disconnects BT first, runs lookup, then reconnects. 4. FontSelectionActivity two fixes for the in-book Font Family submenu: (a) onEnter requests an immediate update so the first frame paints before the user has to scroll, (b) Back is handled on wasReleased instead of wasPressed so the trailing release event does not leak to the parent ReaderOptionsActivity (which also listens on Back release) and cause a double-pop to the in-book main menu.
Rendering-layer additions for v4.5: - Dark Reader Mode (1:1 port from CrossInk v1.3.2): black page background, white body text, status bar inverted, selection highlight + lookup popup flip via reverse-video, drawer panel goes dark. foregroundBlack threaded through Page/PageLine/TextBlock/PageHorizontalRule render chain so SD font glyphs render the right colour in either mode. - Text Darkness (ported from CPR-vCodex with 1px BW outline addition for Extra Dark): 4-mode setting (Normal / Legacy BW / Dark / Extra Dark) tunes the 2-bit grayscale glyph blit's MSB/LSB hit pattern. Extra Dark adds a 1px horizontal outline in the BW pass so the bolder weight reads even when Text AA is off or the AA path silently skips on low heap. - Paragraph Spacing 3-way enum (Tight / Normal / Wide) replacing the prior binary toggle; on-disk byte format is unchanged so old configs migrate. - Font sizes shown as Npt labels instead of friendly names (Tiny / Small / ...) so SD-card font sizes interleave naturally with built-in sizes. - URL-decode for OPF hrefs + nav doc + NCX + img src; books with %20 or %26 in chapter filenames no longer hang at chapter entry (the encoded names never matched the literal ZIP central directory entries). - Theme primitives (BaseTheme/Lyra/Minimal/RoundedRaff drawStatusBar + drawButtonHints) gained darkMode param so the in-reader status bar inverts correctly, and button hints auto-flip when called from a dark-mode wordselect context. - Prebake viewer: font value rendered as Npt (instead of step/range) and split into two labeled rows (Font + Font Size) so the second line isn't a blank-label gap.
…split Reader-activity stability + reading-stats infrastructure: - Defensive save-on-exit: if a render-time progress save failed under heap fragmentation (FsFile alloc failed mid-page), a fallback save fires at reader exit on a usually-cleaner heap. Eliminates the user-fatal "stuck on last saved page" case where only deleting the book unblocked it. - BT enable pre-flight floor 40K -> 55K (X3 + X4). Field-tested under SD-card-font + dark-mode where NimBLE+IINE-subscribe consumed 66K of free heap; the prior 40K floor let pre-flight pass at MaxAlloc=43K and the next page deserialize crashed with TextBlock alloc < needed. - BT scan results capped at MAX_SCAN_RESULTS=12 with weakest-RSSI eviction + pre-scan heap gate; X3 first-pair was OOM-crashing on crowded RF environments where the 22+ device std::string allocs ran the post-scan picker out of heap. - Reading stats split (port from CrossInk v1.3.3): Clear Reading Cache preserves per-book stats.bin; new Delete Book's Reading Stats menu entry in the reader long-press menu and file-browser long-press menu. BookCacheUtils gains preserveStateFiles/restorePreservedFiles helpers + clearBookCacheDirectoryPreservingStats public entry point. - [STATS]/[ERS] serial log demote: BookReadingStats::exists/load and the ERS progress-load path now Storage.exists() short-circuit before the noisy openFileForRead call, so never-opened books no longer spam the serial output during home navigation and carousel pre-render. - "Going home" popup: heap-conditional FAST/HALF refresh. On a tight heap the FAST_REFRESH LUT produces a dim/partially-inverted popup (BW backup compression failures leave the framebuffer state divergent from the controller's view). Below 32K MaxAlloc, falls back to HALF_REFRESH for a clean render at the cost of ~770ms.
… Retry Failed Covers Home + bookshelf + file browser polish, plus a one-shot post-update sweep so cover-gen fixes actually take effect on existing libraries: - EOCD scan window progressive (1KB -> 4KB -> 16KB -> 64KB) so re-packaged EPUBs (Anna's Archive, calibre-rewritten, signed bundles) with trailing metadata past the prior 1KB tail have their cover-jpg readable. Falls back through smaller windows under heap pressure so books that worked before still work. - Cover-thumb marker auto-sweep on firmware-version change: APP_STATE gains lastCrumbleVersion; on boot mismatch CrumBLE walks /.crosspoint/ and deletes every thumb_failed_v3_*.marker so books whose cover gen failed under an earlier bug get re-attempted automatically. - Settings > System > Retry Failed Covers: manual lever for the same sweep so users can retry without waiting for the next update (e.g. after freeing heap, replacing a book file, etc). - Bookshelf delete cascade fix: BookActions::clearFileMetadata now also removes the book from CollectionsStore / LibraryIndex / SeriesIndex (previously only HomeActivity's inline delete did the full sweep, so bookshelf + file browser deletes left a coverless placeholder that re-appeared in Collections and couldn't be opened). - Author trailing-`;` normalizer promoted to RecentBooksStore.h as a shared free function; applied at the storage layer (addOrUpdateBook, updateBook, getDataFromBook) + JSON load. Existing recent.json content cleans on next load. - FileBrowserActionActivity badge layout: the prebake "IMG..." badge gets its own row below the title block when present, so long filenames on never-opened books (no author subtitle) get full line-1 width instead of being squeezed to half by the right-justified badge. - HomeActivity carousel frame count 1 -> 2 (back-and-forth nav hits resident cache instead of SD-paging), HALF_REFRESH_DEEP on the home entry transition so the polarity-drift scrub fires. - README: noted the FT badge hover tooltip + on-device long-press for prepared-layout view; added an Additional Features bullet covering Dark Reader Mode, Text Darkness, Paragraph Spacing, Retry Failed Covers.
…ion scope X3-specific stability + UX: - Sync Clock Now auto-connects to a saved WiFi network if not already up. Iterates lastConnectedSsid first, then every saved credential in storage order, 6s per attempt. Disconnects on exit only if the activity initiated the session. Pre-mechanism saves (no lastConnectedSsid) now work via the iteration fallback. - UTC offset polish: starts at the sign field (so Americas users see +/- caret first), per-segment drawText with running-X tracking so the focus frame and the rendered glyph use identical coords (was drifting by kerning previously, leaving sign/hours right-justified inside the frame), thicker dashed box (2px on 3-on/2-off pattern) around the active field. - HalDisplay HALF_REFRESH_DEEP mode (X3-only): the extra resync(2) cycle scoped to home-entry transitions instead of every HALF_REFRESH; sleep refreshes drop back to single resync, saving ~770ms per sleep cycle on X3. HomeActivity opts into DEEP for the entry refresh. - Sleep image cycling: FAST_REFRESH for 2 of 3 cycles, HALF on every 3rd to scrub ghost buildup. RTC_NOINIT_ATTR counter survives deep- sleep wakes (resets on power loss). Applies to X3 + X4.
…th + multi-delete chunk Web (File Transfer) + WASM prebake CLI: - Atlas dual-emit always-on in crumble-prebake CLI: dropped the 16pt threshold so even 10/12/14pt cpfonts ship both 1-bit (BT-friendly primary) and 2-bit (BT-cold alt) slots. Device install picks based on heap headroom at section load. Adds ~1.5-3KB section-file disk per style; resident RAM unchanged since only one slot loads. - Registered all shipped Bitter sizes (10/12/14/16pt) in the WASM optimizer so users on those sizes get a built-in glyph atlas baked into their section files instead of falling through to the runtime miss-handler path. - FT badge tooltip enrichment: server formatPrebakeTooltip() parses the per-book prebake-manifest.json and returns a 5-line summary (Font / Font Size / Orientation / Line Spacing / Margin) appended to the badge's data-tooltip. Heap-guarded -- skipped below 8KB MaxAlloc. Client renders via CSS ::after bubble (instant) instead of native title (~500ms browser delay), with pointer cursor. - Upload modal: hides preflight layout-settings prompt and skips the cache-upload step when the outer "Optimize EPUB" toggle is off (previously the inner Pre-Cache toggle's checked state ignored the outer gate). Pre-Cache toggle renamed Pre-Bake (Image, Chapter, Custom Font) with a 3-bullet description. - Multi-delete chunked client-side: bulk-delete of 100+ files in a cache dir splits into 20-path batches so the form-arg body fits the ESP32 WebServer parser. Stops on first chunk failure and surfaces it in the alert. - WS DONE protocol upgrade (DONE -> DONE:<path>): backward-compatible. Eliminates the heap-fragile /api/files listing call from the post-upload prebake step on tight heap (Onyx Storm reproducer: listing bailed at 6 rows, prebake then couldn't find the file).
Minor build-system updates supporting the 4.5 release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v4.5.0 release bundle. Six thematic commits + four prior commits already on this branch.
Rendering & typography
%20/%26in filenamesStability
Home, bookshelf, cover gen
Settings → System → Retry Failed Covers;cleanup — normalized at storage layer; old recent.json content cleans on next loadX3
Web (File Transfer) + WASM prebake CLI
Test plan
crumble-firmware-4.5.0.binon X3 and X4