MQTT telemetry and remote settings

[elliotmatson]

Cubes in the field report to an MQTT broker (EMQX, behind Tailscale Funnel) and take settings from it. Always on with no opt-out, as agreed while the project is in testing. The README's new Telemetry section lists everything that's sent.

What's published (cube/<id>/…, JSON)

• Identity: a random 16-hex ID made once and kept in NVS. It's also the MQTT client ID, which lets the broker's ACL use ${clientid}.
• status: retained online, with offline as the last will.
• boot: version, build, chip, MAC, SSID, BSSID, RSSI, channel, local IP, public IP (from ipify), reset reason, rollbacks and the boot log.
• health / usage / perf: every telemetry_interval (default 300 s), each switchable. Uptime and memory low-water marks, the current BSSID, seconds per pattern, brightness and features in use, and renderer timings.
• wifi: a scan at boot and hourly, up to 30 networks.
• crash: the core dump summary (task, PC, cause, backtrace, ELF SHA), then the dump in base64 parts. It's sent once per dump, and only marked sent (by CRC) once the broker has acknowledged every part. resend_crash sends it again.
• settings: retained, republished each interval and after every command. ack: the result of each command.

Remote settings (cube/<id>/set/<name>)

• What: pattern, brightness, ticker, time zone, weather location and units, GitHub updates, development channel, ArduinoOTA, telemetry interval and report toggles, plus the actions restart, check_updates and resend_crash.
• Checks: lib/remote (hardware free, 6 host tests) validates each one: known names, integer ranges, flag spellings, text limits.
• Applying: Cube::applyRemote applies them through the same code paths as the dashboard, so the dashboard stays in sync.

Lost mode

• Turning it on: set lost_message and lost_pin (4–12 digits, stored as a salted SHA-256), optionally lost_silent, then lost_mode true.
• Display: a hidden Lost pattern scrolls the message round the sides, with LOST and the cube's ID on top. It's never listed, saved or selectable. With lost_silent, patterns run as usual.
• Lockdown:
  • every dashboard control and API POST is refused (423), though status still reads;
  • the upload card and ArduinoOTA refuse firmware, while GitHub updates still install;
  • Reset WiFi and the setup portal stay open, deliberately, since someone setting the cube up on their own network is its likeliest way back online.
• Tracking:
  • a retained location every 2 minutes: a fresh public IP, network, BSSID and a scan;
  • a retained lost status;
  • a network report whenever it comes online anywhere.
• Open networks: after 3 minutes offline (and not in the setup portal), it tries the strongest open networks. It keeps one only if the broker answers through it, and the saved credentials are never overwritten.
• Persistence: restarts, power loss and WiFi resets. It turns off with lost_mode false, or the PIN on the dashboard's System tab: five guesses, then one per 15 minutes, counted across restarts.
• Tested on the cube:
  • commands were applied and acknowledged, and out-of-range values rejected;
  • lost mode showed the message, refused API changes and firmware uploads with 423, and kept Lost out of the pattern list;
  • it survived a reset;
  • clearing it over MQTT unlocked the cube and restored the previous pattern.
• Still to test: the PIN unlock on the dashboard and the open-network search.

Retained reports

boot and wifi are now retained, so a client that subscribes later still sees the SSID and scan. Health reports include the SSID, and each (re)connection publishes a retained network message with a re-fetched public IP.

Plumbing

• Client: ESP-IDF's MQTT client over wss://, with certificates checked against the existing bundle, its own task and auto-reconnect. Publishing happens from a low-priority Telemetry task.
• Credentials: MQTT_URL / MQTT_USER / MQTT_PASSWORD from the gitignored secrets.h. Builds without them don't connect (this PR's CI build included).
• Visibility: a Telemetry card on the System tab and /api/v1/telemetry.
• Updates::checkNow(): wakes the GitHub updater.
• CONFIG_MBEDTLS_DEBUG off: ESP-IDF's TLS layer, unlike Arduino's, turned it into verbose logging of every TLS record. On the USB-Serial-JTAG console those writes block when no host is reading.

Tested on the cube against EMQX

• Connection: connected through Funnel (wss://emqx.tail1ca91.ts.net:8443/mqtt) with the cube login. Anonymous connections are refused.
• Reports: boot, settings and the WiFi scan arrived. The stored core dump was uploaded once and then skipped as already sent.
• Rollback: the boot log caught my own test resets during first boots, and rollback protection put the previous image back each time.
• Tests: 123 host tests pass.
• Commands: tested against a local broker (see Lost mode). They're not yet tested through EMQX's ACL with the admin login.

Follow-up

Release builds need build-release.yml to write secrets.h from repository secrets, which depends on elliotmatson/pio-actions#27 (a build-env secret) being released. That change follows as a separate small PR.

🤖 Generated with Claude Code

[github-actions]

Tests

124/124 passed.

Suite	Tests	Passed	Failed	Time
native:test_firmware_image	14	14	0	3.24s
native:test_pong	6	6	0	0.75s
native:test_remote	7	7	0	0.79s
native:test_rubiks	7	7	0	0.60s
native:test_timezones	5	5	0	0.61s
native:test_sand	6	6	0	0.61s
native:test_bounce	6	6	0	0.60s
native:test_maze	4	4	0	0.60s
native:test_cube_geometry	22	22	0	0.73s
native:test_life	7	7	0	0.63s
native:test_noise	5	5	0	0.74s
native:test_breakout	6	6	0	0.60s
native:test_wordclock	5	5	0	0.61s
native:test_color	6	6	0	0.61s
native:test_langton	5	5	0	0.64s
native:test_particles	7	7	0	0.64s
native:test_weather	6	6	0	0.69s

[github-actions]

Static analysis

No defects found.

_{pio check, failing on high and above. 1 defect in downloaded dependencies withheld; pass include-dependencies to see them.}

[github-actions]

Firmware size

-24.9 KB across 1 environment.

Env	Image	Δ	App partition	Flash Δ	RAM Δ
esp32-s3-devkitc-1 ⚠️	1.80 MB	-24.9 KB	23.92% → 23.6%	-24.9 KB	-16.7 KB

esp32-s3-devkitc-1 — 4 section(s) changed

Section	Base	Head	Δ
.flash.rodata	534.8 KB	517.6 KB	-17.2 KB
.flash.text	1.18 MB	1.17 MB	-7.7 KB
.dram0.bss	37.3 KB	37.8 KB	+504 B
.dram0.data	23.8 KB	23.8 KB	+16 B

esp32-s3-devkitc-1 — 401 symbol(s) changed

Symbol	Base	Head	Δ
ssl_parse_client_hello	2.7 KB	0 B	-2.7 KB
mbedtls_ssl_handshake_client_step	5.7 KB	3.5 KB	-2.2 KB
LostMode::tryOpenNetworks()	0 B	1.8 KB	+1.8 KB
Cube::initAPI()::{lambda(AsyncWebServerRequest*)#13}::operator()(AsyncWebServerRequest*) const	0 B	1.8 KB	+1.8 KB
Cube::initAPI()::{lambda(AsyncWebServerRequest*)#12}::operator()(AsyncWebServerRequest*) const	1.7 KB	0 B	-1.7 KB
mbedtls_x509_crt_info	1.2 KB	0 B	-1.2 KB
void std::__introsort_loop<__gnu_cxx::__normal_iterator<LostMode::tryOpenNetworks()::Open*, std::vector<LostMode::tryOpenNetworks()::Open, std::allocator<LostMode::tryOpenNetworks()::Open> > >, int, __gnu_cxx::__ops::_Iter_comp_iter<LostMode::tryOpenNetworks()::{lambda(LostMode::tryOpenNetworks()::Open const&, LostMode::tryOpenNetworks()::Open const&)#1}> >(__gnu_cxx::__normal_iterator<LostMode::tryOpenNetworks()::Open*, std::vector<LostMode::tryOpenNetworks()::Open, std::allocator<LostMode::tryOpenNetworks()::Open> > >, __gnu_cxx::__normal_iterator<LostMode::tryOpenNetworks()::Open*, std::vector<LostMode::tryOpenNetworks()::Open, std::allocator<LostMode::tryOpenNetworks()::Open> > >, int, __gnu_cxx::__ops::_Iter_comp_iter<LostMode::tryOpenNetworks()::{lambda(LostMode::tryOpenNetworks()::Open const&, LostMode::tryOpenNetworks()::Open const&)#1}>)	0 B	1.1 KB	+1.1 KB
Cube::initAPI()::{lambda(AsyncWebServerRequest*)#6}::operator()(AsyncWebServerRequest*) const	0 B	1.0 KB	+1.0 KB
_ZZN4Cube7initAPIEvENKUlP21AsyncWebServerRequestE5_clES1_$isra$0	0 B	998 B	+998 B
_ZZN4Cube7initAPIEvENKUlP21AsyncWebServerRequestE4_clES1_$isra$0	974 B	0 B	-974 B
LostMode::setPin(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)	0 B	934 B	+934 B
Cube::applyRemote[abi:cxx11](remote::Command const&)	0 B	930 B	+930 B
mbedtls_x509_info_subject_alt_name	871 B	0 B	-871 B
mbedtls_x509_dn_gets	798 B	0 B	-798 B
ssl_parse_certificate_request	748 B	0 B	-748 B
+386 more

_{Baseline: 9ddd045. Image size is firmware.bin against the smaller app partition slot. RAM is static allocation only — the linker cannot see heap or stack.}