fix: HA renders full lidar map (drain chunked history stream)#5
Conversation
The HA integration rendered only a partial lidar/history map while the web dashboard showed the full one. get_history_session() did a single response.content.read(N) against the firmware GET /api/history/<file>, which is served as a chunked transfer with no Content-Length. aiohttp read(n) returns the first buffered chunk rather than blocking to EOF, so the JSONL was silently truncated and parse_session_jsonl dropped every pose past the cut. Drain the stream to EOF via iter_chunked (matching the frontend res.text()), enforcing the size cap incrementally, and warn when the final history line fails to parse so future truncation is observable. Fixes renjfk#132 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
to_canvas() used radians(90 - angle) which flips the cos sign and mirrors the live lidar scan horizontally versus the web dashboard. Use radians(90 + angle) to match the frontend reference (lidar-map.tsx). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Discussions are now enabled on this fork; repoint the contact link from the upstream repo to Leicas/OpenNeato. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Fixes a Home Assistant integration bug where the lidar history map was rendered partially because get_history_session() did a single bounded content.read(n) against the firmware's chunked-transfer endpoint, which returns as soon as any buffered data is available rather than blocking to EOF. The fix drains the chunked response with iter_chunked while incrementally enforcing the size cap, mirroring the frontend's res.text() semantics. Two adjacent improvements are bundled: a warning when the final JSONL line fails to parse (truncation signature), and a sign-flip in lidar_renderer.to_canvas to match the frontend orientation.
Changes:
- Drain the chunked
/api/history/<file>response to EOF inapi.pyto avoid truncated downloads. - Log a warning in
parse_session_jsonlwhen the final line fails to parse, surfacing future silent truncation. - Align Python lidar
to_canvasformula (90 + angle) withlidar-map.tsx, fixing a horizontal mirror. - Update
.github/ISSUE_TEMPLATE/config.ymldiscussions URL to the Leicas fork (consistent with README fork positioning).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| custom_components/openneato/api.py | Replace single bounded read with iter_chunked accumulation + incremental size cap. |
| custom_components/openneato/history_renderer.py | Warn when final JSONL line is unparseable (truncation indicator). |
| custom_components/openneato/lidar_renderer.py | Switch to_canvas to 90 + angle to match frontend reference. |
| .github/ISSUE_TEMPLATE/config.yml | Point discussions link at the Leicas fork. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Problem
Reported in renjfk/OpenNeato#132: the Home Assistant integration renders only a partial lidar/history map, while the OpenNeato web dashboard shows the full map for the same robot and session.
Root cause
A multi-agent investigation (3 architects + dev) converged with high confidence on a chunked-stream truncation in the API client — not a rendering bug.
The firmware serves
GET /api/history/<file>as an HTTP chunked transfer with noContent-Length(firmware/src/web_server.cpp:454,beginChunkedResponse). The HA client did a single bounded read:On a chunked aiohttp response,
read(n)returns as soon as any data is buffered — it does not block untilnbytes or EOF. HA therefore decoded only the first chunk(s): a truncated JSONL prefix, far below the 2 MiB cap (so the cap check passed silently).parse_session_jsonlthen silently dropped every pose past the cut (itsjson.loadsis wrapped intry/except → continue), rendering a subset of the path → a partial map.The frontend hits the same endpoint but uses
await res.text()(frontend/src/api.ts:81), which drains the stream to EOF — hence the full dashboard map.Changes
api.pyget_history_session()— single read →async for chunk in response.content.iter_chunked(65536)accumulate-to-EOF, with the size cap enforced incrementally (matches the frontend'sres.text()semantics). Primary fix.history_renderer.pyparse_session_jsonl()— log a warning when the final line fails to parse (the classic truncation signature) so any future silent truncation is observable.lidar_renderer.pyto_canvas()—radians(90 - angle)→radians(90 + angle)to match the frontend (lidar-map.tsx); fixes a separate horizontal mirror on the live lidar view (orientation only, orthogonal to the partial-map bug).Firmware needs no change — it already serves complete data. This is an HA-integration-only fix shipped via HACS.
Verification
Against a live bridge:
len(buf)inget_history_session— before, much smaller than the firmware session file; after, equals the full decompressed JSONL.len(poses)inparse_session_jsonl— jumps to the full set; no truncation warning on a healthy download.All three files
py_compileclean.Fixes renjfk#132
🤖 Generated with Claude Code