MQTT telemetry and remote settings

CLAUDE.md

@@ -30,9 +30,11 @@ the dashboard first (a fresh cube has it off). To flash over USB, comment out
 ## Configuration
 
 `sdkconfig.defaults` is the source of truth. `sdkconfig.esp32-s3-devkitc-1` is
-generated from it on every build and is gitignored. To change a setting, run
-`pio run -t menuconfig`, then `pio run -t save-defconfig`, and diff before
-committing — save-defconfig strips comments.
+generated from it and gitignored -- but only when it is missing, so after
+editing `sdkconfig.defaults` delete it to make the change take effect (a full
+rebuild follows). To change a setting, run `pio run -t menuconfig`, then
+`pio run -t save-defconfig`, and diff before committing — save-defconfig
+strips comments.
 
 `lib/cube/config.h` holds pins, panel geometry and build identity. Runtime
 settings live in `lib/cube/settings.*`: one NVS key each in namespace `cube`,
@@ -55,6 +57,9 @@ lib/cube/cube.*              Cube: startup, display, WiFi/time, dashboard cards,
 lib/cube/renderer.*          the render task, canvas and pattern switching
 lib/cube/updates.*           upload card, ArduinoOTA, GitHub updater, update progress screen
 lib/cube/settings.*          persistent settings (NVS)
+lib/cube/boot_log.*          the last few boots: version, reset reason, how far startup got
+lib/cube/telemetry.*         MQTT reports to the broker and settings from it (see README, Telemetry)
+lib/cube/lost_mode.*         lost mode's unlock PIN and its search for open networks (Cube::setLost does the rest)
 lib/cube_utils/              Pattern base class; SinglePanel/BottomPanels views (one ChainView base)
 lib/cube_geometry/           hardware-free: face mappings, seam stepping, 3D surface mapping, projection
 lib/life/                    hardware-free Game of Life step
@@ -71,6 +76,7 @@ lib/langton/                 hardware-free Langton's ant and turmites on any cel
 lib/arcade/                  hardware-free game rules: Pong and Breakout
 lib/wordclock/               hardware-free word clock: which letters spell the time
 lib/weather/                 hardware-free weather codes, units and URL encoding for the Weather pattern
+lib/remote/                  hardware-free names, types and checks for settings set over MQTT
 test/                        host unit tests for the hardware-free libraries ([env:native])
 lib/patterns/<name>/         one folder per pattern; registered in lib/patterns/all_patterns.cpp
 lib/fonts/                   GFX fonts, each defined once in fonts.cpp: include fonts.h, never a font's own header
@@ -85,7 +91,7 @@ for a pixel's 3D position on the cube surface (`cube::fromCube` back), and `cube
 isometric projection continuous across the seams.
 
 `lib/cube_geometry`, `lib/life`, `lib/timezones`, `lib/firmware_image`,
-`lib/noise`, `lib/color`, `lib/rubiks`, `lib/sand`, `lib/bounce`, `lib/particles`, `lib/maze`, `lib/langton`, `lib/arcade`, `lib/wordclock` and `lib/weather` include nothing from Arduino or ESP-IDF, so
+`lib/noise`, `lib/color`, `lib/rubiks`, `lib/sand`, `lib/bounce`, `lib/particles`, `lib/maze`, `lib/langton`, `lib/arcade`, `lib/wordclock`, `lib/weather` and `lib/remote` include nothing from Arduino or ESP-IDF, so
 `[env:native]` can test them on the host. Keep it that way, and put new pure
 logic in libraries like these so it can be tested too. The board env takes its
 settings from `[esp32_base]` rather than `[env]`, which would leak the
@@ -164,6 +170,15 @@ scripts/capture_patterns.py --host cube.local <pattern id>` (frames come from
   A reset before that point rolls the update back. `BootLog` records every
   boot's version, reset reason and how far startup got (`/api/v1/boots`,
   and the Startup card): a failed startup shows the stage it stopped at.
+- **Lost mode locks every control.** A new dashboard handler or API POST
+  needs the `locked()` check its neighbours have, or it becomes a way round
+  lost mode. Reset WiFi and the setup portal stay open deliberately. The Lost
+  pattern is `hidden`: never listed, saved or selectable.
+- **Telemetry credentials are public.** `MQTT_URL`, `MQTT_USER` and
+  `MQTT_PASSWORD` come from the gitignored `secrets.h` (CI writes it from
+  repository secrets), but anything compiled in can be read out of a release.
+  The broker's ACL is what keeps the shared login to a cube's own topics; a
+  new setting reachable over MQTT goes in `lib/remote` and `Cube::applyRemote`.
 - **Updates only go forwards.** The GitHub updater installs a release only
   if `firmware_image::isNewer()` says its tag is newer than `FW_VERSION`.
   GitHub's "latest" is the newest *stable* release, older than any beta.

README.md

@@ -32,6 +32,7 @@ Recorded from a running cube with `scripts/capture_patterns.py` and drawn as the
     - [Uploading](#uploading)
   - [Spotify](#spotify)
   - [Weather](#weather)
+  - [Telemetry](#telemetry)
 
 ## Development
 
@@ -89,6 +90,69 @@ By default the cube works out where it is from its IP address, which can be off
 
 The same settings are at `/api/v1/weather`: GET for what the pattern knows, POST `location=` and/or `metric=true|false`. POST `preview=<WMO code>` (and `day=false` for night) shows a kind of weather for two minutes, to try the animations: 0 clear, 2 partly cloudy, 3 overcast, 45 fog, 53 drizzle, 63 rain, 73 snow, 95 thunderstorm.
 
+## Telemetry
+
+Release builds report to an MQTT broker, and take settings from it, so cubes in the field can be followed and managed. **While the project is in testing this is always on, with no opt-out.** A build reports only if it was given a broker (`MQTT_URL`, `MQTT_USER`, `MQTT_PASSWORD` in `lib/cube/secrets.h`, written by CI from repository secrets); other builds never connect.
+
+Each cube picks a random ID once and publishes under `cube/<id>/`, as JSON:
+
+| Topic | When | Contents |
+|---|---|---|
+| `status` | retained | `online`, or `offline` (the broker publishes that if the cube drops) |
+| `boot` | each boot (retained) | firmware version and build, chip, MAC address, WiFi network (SSID, BSSID, signal, channel), local and **public IP**, the boot log (reset reasons, rollbacks, how far startup got) |
+| `network` | each (re)connection (retained) | SSID, BSSID, signal, local and public IP, and whether it is on its own network or an open one |
+| `health` | every interval | uptime, memory and its low-water marks, signal, current network (SSID, BSSID), pattern |
+| `usage` | every interval | seconds per pattern, brightness, whether Spotify, a weather location and a ticker message are set |
+| `perf` | every interval | the renderer's frame rate and tick/push times |
+| `wifi` | boot, then hourly (retained) | a scan of the networks in range: SSID, BSSID, signal, channel, security |
+| `location` | while lost: every 2 minutes (retained) | a fresh public IP, its network and access point, and a scan of what is in range |
+| `lost` | when lost mode changes (retained) | whether it is on, silent, its message, whether a PIN is set |
+| `crash` | once per crash | a summary (task, PC, cause, backtrace), then the core dump in base64 parts on `crash/<n>` |
+| `settings` | retained | current settings |
+| `ack` | per command | whether a command was applied, and why not |
+
+The interval defaults to 5 minutes. No Spotify or WiFi credentials are ever sent.
+
+Commands are messages to `cube/<id>/set/<name>`:
+
+| Name | Value |
+|---|---|
+| `pattern` | a pattern id (see `/api/v1/patterns`) |
+| `brightness` | 0-255 |
+| `ticker`, `timezone`, `weather_location` | text |
+| `weather_metric`, `github_updates`, `development`, `ota` | `true` / `false` |
+| `telemetry_interval` | seconds, 60-86400 |
+| `report_health`, `report_usage`, `report_perf` | `true` / `false` |
+| `lost_message` | text shown in lost mode (up to 120 characters) |
+| `lost_pin` | 4-12 digits that unlock it on its dashboard; empty removes it |
+| `lost_silent` | `true`: in lost mode, keep looking normal |
+| `lost_mode` | `true` / `false` |
+| `restart`, `check_updates`, `resend_crash` | anything (actions) |
+
+### Lost mode
+
+If a cube goes missing, set `lost_message` (and `lost_pin`, if whoever finds it should be able to unlock it), then `lost_mode` `true`:
+
+- It shows the message, scrolling round the sides, with LOST and its ID on top -- unless `lost_silent` is set, when it keeps showing patterns as usual.
+- Its dashboard and API refuse changes (status still reads), and the upload card and ArduinoOTA refuse firmware. GitHub updates still install.
+- It reports a `location` every two minutes, and `network` whenever it comes online anywhere.
+- WiFi setup stays open on purpose: someone setting it up on their own network is its likeliest way back online. If it is offline for a few minutes, it also tries nearby open networks, keeping one only if the broker can be reached through it; its saved network is never overwritten.
+- It persists through restarts, power loss and WiFi resets. `lost_mode` `false`, or the PIN on the dashboard's System tab, turns it off. PIN guesses are limited: after five wrong ones, one every 15 minutes.
+
+### Broker setup
+
+The firmware connects with `wss://` (MQTT over WebSockets, TLS on a standard HTTPS port), checking the certificate against ESP-IDF's bundle, so the broker needs a publicly trusted certificate. Every cube shares one login, which is readable from any published image, so the broker must confine it. With EMQX:
+
+```
+{allow, {username, "admin"}, all, ["#"]}.
+{deny,  {username, "cube"}, publish, ["cube/${clientid}/set/#"]}.
+{allow, {username, "cube"}, publish, ["cube/${clientid}/#"]}.
+{allow, {username, "cube"}, subscribe, ["cube/${clientid}/set/#"]}.
+{deny, all}.
+```
+
+A cube can then only publish its own reports and read its own commands; commands come from a separate admin login that never goes in firmware.
+
 ## Recording the pattern animations
 
 The animations above come from the cube itself: `GET /api/v1/frame` returns the frame it is showing, and the script switches through every pattern and records each one.