Skip to content

fix: HA renders full lidar map (drain chunked history stream)#5

Merged
Leicas merged 3 commits into
mainfrom
fix/ha-lidar-map
May 30, 2026
Merged

fix: HA renders full lidar map (drain chunked history stream)#5
Leicas merged 3 commits into
mainfrom
fix/ha-lidar-map

Conversation

@Leicas

@Leicas Leicas commented May 30, 2026

Copy link
Copy Markdown
Owner

Problem

Reported in renjfk/OpenNeato#132: the Home Assistant integration renders only a partial lidar/history map, while the OpenNeato web dashboard shows the full map for the same robot and session.

Root cause

A multi-agent investigation (3 architects + dev) converged with high confidence on a chunked-stream truncation in the API client — not a rendering bug.

The firmware serves GET /api/history/<file> as an HTTP chunked transfer with no Content-Length (firmware/src/web_server.cpp:454, beginChunkedResponse). The HA client did a single bounded read:

raw = await response.content.read(MAX_HISTORY_RESPONSE_BYTES + 1)

On a chunked aiohttp response, read(n) returns as soon as any data is buffered — it does not block until n bytes or EOF. HA therefore decoded only the first chunk(s): a truncated JSONL prefix, far below the 2 MiB cap (so the cap check passed silently). parse_session_jsonl then silently dropped every pose past the cut (its json.loads is wrapped in try/except → continue), rendering a subset of the path → a partial map.

The frontend hits the same endpoint but uses await res.text() (frontend/src/api.ts:81), which drains the stream to EOF — hence the full dashboard map.

Changes

  • api.py get_history_session() — single read → async for chunk in response.content.iter_chunked(65536) accumulate-to-EOF, with the size cap enforced incrementally (matches the frontend's res.text() semantics). Primary fix.
  • history_renderer.py parse_session_jsonl() — log a warning when the final line fails to parse (the classic truncation signature) so any future silent truncation is observable.
  • lidar_renderer.py to_canvas()radians(90 - angle)radians(90 + angle) to match the frontend (lidar-map.tsx); fixes a separate horizontal mirror on the live lidar view (orientation only, orthogonal to the partial-map bug).

Firmware needs no change — it already serves complete data. This is an HA-integration-only fix shipped via HACS.

Verification

Against a live bridge:

  1. Log len(buf) in get_history_session — before, much smaller than the firmware session file; after, equals the full decompressed JSONL.
  2. Log len(poses) in parse_session_jsonl — jumps to the full set; no truncation warning on a healthy download.
  3. Compare the HA history camera image vs the frontend replay for the same session — extents now match.

All three files py_compile clean.

Fixes renjfk#132

🤖 Generated with Claude Code

Antoine Weill--Duflos and others added 3 commits May 30, 2026 12:27
The HA integration rendered only a partial lidar/history map while the web
dashboard showed the full one. get_history_session() did a single
response.content.read(N) against the firmware GET /api/history/<file>, which
is served as a chunked transfer with no Content-Length. aiohttp read(n)
returns the first buffered chunk rather than blocking to EOF, so the JSONL was
silently truncated and parse_session_jsonl dropped every pose past the cut.

Drain the stream to EOF via iter_chunked (matching the frontend res.text()),
enforcing the size cap incrementally, and warn when the final history line
fails to parse so future truncation is observable.

Fixes renjfk#132

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
to_canvas() used radians(90 - angle) which flips the cos sign and mirrors the
live lidar scan horizontally versus the web dashboard. Use radians(90 + angle)
to match the frontend reference (lidar-map.tsx).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discussions are now enabled on this fork; repoint the contact link from the
upstream repo to Leicas/OpenNeato.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 30, 2026 16:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a Home Assistant integration bug where the lidar history map was rendered partially because get_history_session() did a single bounded content.read(n) against the firmware's chunked-transfer endpoint, which returns as soon as any buffered data is available rather than blocking to EOF. The fix drains the chunked response with iter_chunked while incrementally enforcing the size cap, mirroring the frontend's res.text() semantics. Two adjacent improvements are bundled: a warning when the final JSONL line fails to parse (truncation signature), and a sign-flip in lidar_renderer.to_canvas to match the frontend orientation.

Changes:

  • Drain the chunked /api/history/<file> response to EOF in api.py to avoid truncated downloads.
  • Log a warning in parse_session_jsonl when the final line fails to parse, surfacing future silent truncation.
  • Align Python lidar to_canvas formula (90 + angle) with lidar-map.tsx, fixing a horizontal mirror.
  • Update .github/ISSUE_TEMPLATE/config.yml discussions URL to the Leicas fork (consistent with README fork positioning).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
custom_components/openneato/api.py Replace single bounded read with iter_chunked accumulation + incremental size cap.
custom_components/openneato/history_renderer.py Warn when final JSONL line is unparseable (truncation indicator).
custom_components/openneato/lidar_renderer.py Switch to_canvas to 90 + angle to match frontend reference.
.github/ISSUE_TEMPLATE/config.yml Point discussions link at the Leicas fork.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Leicas Leicas merged commit f723536 into main May 30, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants