Repository navigation
Production-readiness fixes: clock, self-update safety, kiosk liveness, TLS - #4
Conversation
CapClockFloor lowers clockFloor and signedClockFloor to at most a given time, in one locked update. The clock keeper uses it when the clock is NTP-synchronised and the floor is ahead of it: such a floor was raised by something wrong and must not keep pushing the clock forward.
The backend accepted any WebSocket on /api/refresh-musallahboard that named a known device id, and a newcomer replaced the board's own connection. Device ids are not secret, so anyone could keep evicting a board from its push channel and watch when pushes arrive. The upgrade is now a device-signed GET (section 9.2: the path without the query, an empty body), signed afresh for every dial with the same helper as content-bundle and weather. The deviceId query stays, so a backend that predates this still finds the board by it. The backend verifies the signature when present; unsigned upgrades stay accepted, without taking over from a signed one, until the owner sets musallahboard.board-stream.require-signature.
checkKiosk fell through to `systemctl is-active` for any probe failure
other than a missing getStatus() hook. A crashed tab ("Aw, Snap!") or hung
JavaScript leaves the kiosk unit active, so a blank board showed as
healthy in the admin console.
- cdp: the page probe opens one connection, sends Inspector.enable (which
Chromium answers with Inspector.targetCrashed on a tab whose renderer is
gone) and then Runtime.evaluate. Failures are classified:
ErrDebuggerUnreachable / ErrNoPageTarget (no browser to ask yet),
ErrPageCrashed, ErrPageUnresponsive (connection accepted, no answer in
time), ErrPageNavigating (context destroyed by a navigation),
ErrPageThrew. CDP error replies are a typed RPCError. Error texts are
unchanged.
- telemetry: crashed or unresponsive is kioskAlive=false; splash, update
screen, old board apps, a page mid-navigation or a throwing hook are
alive; only an unreachable debugger falls back to the unit state.
- First tests for internal/cdp, against a fake debugger endpoint.
The kiosk watcher only navigated away from chrome-error: pages. A renderer crash or hung JavaScript left a blank board until someone noticed. Every 30 s the watcher now probes the page (Inspector.enable plus a trivial Runtime.evaluate, 5 s): - a crashed renderer after one probe, or a page that has not answered for three probes in a row (90 s; a Pi decoding a large poster can miss one), is reloaded; - if it is still dead at the next threshold, the kiosk unit is restarted through the same sudo path as the kiosk.restart command; - restarts are at least 5 min apart, doubling to 1 hour, and that backoff only starts over after an hour of a healthy page. A page that never comes back costs at most one reload and one restart an hour; one that recovers on reload is reloaded whenever it dies. The splash, the update screen (skipped entirely while it is up), old board apps without getStatus(), a page mid-navigation and a browser that is not up yet are never counted as failures.
A laptop with a wrong clock uploading on the service port could pin a board in the future for good: the uploader's time became the clock floor, the keeper persisted the (now wrong) wall clock into it every ten minutes, and when timesyncd corrected the clock the keeper pushed it forward again, every ten minutes. Prayer times were wrong and, past five minutes of skew, the backend refused the board's signed requests. A manual date or a wrong RTC did the same. Only editing state.json helped. - With NTP synchronised, the keeper never moves the clock and ignores uploaders' clocks. A floor (or signed floor) ahead of NTP time is wrong: it is lowered to NTP time, with a warning in the journal. Boards already in the bad state recover by themselves once NTP has synchronised. - NTP time is written to the floor once two checks an interval apart agree with the monotonic clock, so a clock stepped by hand between timesyncd's polls is not taken for NTP time. - Without NTP the wall clock is never written to the floor: the floor advances from the last trusted time (the floor itself, a signed createdAt, NTP time) by the monotonic clock. A board that loses power still comes back no earlier than the time it last knew. - An uploader's time sets the clock (and the RTC) but no longer the floor, so the next uploader with a right clock, or NTP, undoes a wrong one. It still cannot move the clock below the floor or more than 90 days past it. The timesyncd check moves to internal/clock/timesync so the importer can ask it too.
enroll took any --backend scheme and saved whatever websocketUrl the backend returned. Over http the enrollment reply, which pins the content signing keys and the WebSocket URL, can be rewritten by anyone on the LAN. - config: CheckBackendURL (https://, a host, no credentials, query or fragment) and CheckWebSocketURL (wss://, same host as the backend; the port may differ, since TLS binds the host name and not the port). No override for split-host deployments. A new explicit key, insecure_dev = true, allows http:// and ws:// for development boards; the host rule still applies. Config.CheckTransport applies both. - enroll: the backend URL is checked before the token is sent, and the returned WebSocket URL before anything is saved, so a refused reply leaves the board unenrolled and pins no keys. --insecure-dev (with a loud warning) writes insecure_dev = true. Tests run against TLS servers now.
A backend (or build machine) whose clock was briefly far ahead signs a
createdAt in, say, 2030. Every verified createdAt raises the board's clock
floor, so that one package pinned every board that took it in 2030.
With NTP synchronised, a package whose createdAt is more than a day past
the board's clock is now refused ("An update is dated in the future: the
computer that made it has a wrong clock"; the times are in the detail),
and so is its Preflight. Nothing about it is recorded: not its time, and
not its sequence, so the backend's next right package installs even if
its sequence is lower. A createdAt less than a day ahead installs, but
raises the floor only up to NTP time.
Without NTP the board has nothing better than the signed time, and it
counts as before. Content sequence is unchanged: still strictly
increasing, with no upper bound.
The page told the uploader a drifting board clock would be "corrected when you send an update". A board set from the internet no longer takes an uploader's time, so the page now says the uploader's clock is probably the wrong one.
Section 10 says when the keeper moves the clock and what it writes to the floor, how a floor ahead of NTP time heals, that an uploader's time no longer reaches the floor, and what this trades: what rollback resistance remains without NTP, and what whoever controls NTP can do. Sections 4 and 8 and the threat table say that packages signed more than a day ahead of an NTP clock are refused.
Nothing checked the backend and WebSocket URLs after enrollment, so a board enrolled over http kept syncing content, dialling ws:// and taking remote commands in the clear. - startBoard runs config.CheckTransport before it builds anything that talks to the backend. When it fails, no syncer (content sync, refresh channel, release checks) and no command WebSocket are started; the board keeps showing what it has. The reason and the fix go to the log at every start, to `systemctl status` (STATUS=), and to the local status, so `musallahboard-agent status` shows "LensBridge: NOT connected" and "Sync: failing" with it, as does the board app. - With insecure_dev = true a loud warning is logged at every start. - wsclient never dials a WebSocket URL that fails the check (backstop). - trust fetch refuses a plain-http backend unless insecure_dev is set in the config, or --insecure-dev is passed with --backend.
- README and vm/README: enroll a local plain-http backend with --insecure-dev; what happens to a board already enrolled over http and how to fix it. - Makefile: install-remote examples use an https backend. - architecture.md: the transport rule and insecure_dev (section 11), the kiosk probe and its reload/restart backoff (section 14), and the security table.
The updater judged a new agent only by "/api/local/status reports the new version within 120 s". A crash after that, a release that cannot reach LensBridge, or a power cut between the swap and the verdict all left the new agent installed and never judged again. - The swap is a transaction with a root-owned journal, /usr/lib/musallahboard/update-in-progress.json, written (temp, fsync, rename, fsync dir) before the binary is replaced and at every phase: swapping, probation, committing, rolling-back. The next run finishes whatever it finds, deciding by phase and by the SHA-256 of what is installed, so a power cut at any step resolves the same way. The `ready` marker is put back while the transaction is open, so boards whose path unit predates the journal also start the updater at boot. - Probation: the new agent must report its version within 2 minutes, then run 4 minutes with no restart (systemd's MainPID, NRestarts and ActiveState) and, if the board was connected to LensBridge before the update, connect again (authenticated WebSocket or a content sync) within 3 minutes. A board that was offline is not asked to be online. - A crash loop, a restart or not coming up rolls back and rejects the version. Not reconnecting rolls back but only counts a strike: the backend or network may be at fault, so the version is rejected the second time. - A probation cut short (shutdown, the unit's timeout) leaves the journal and is judged again from scratch, at most 3 times. A rollback that has begun always finishes; one that cannot restore agent.prev says so (rollback-failed) instead of claiming it went back.
- The path unit also watches the update journal, so an update a power cut or reboot interrupted is finished at boot. - TimeoutStartSec 15 minutes: room for the probation and a rollback. - The update unit no longer gets write access to /etc/musallahboard: it only reads trust.json (the comment about `trust fetch` was stale). It gets /usr/share/musallahboard instead, which `splash install` after an update writes and which ProtectSystem=strict made read-only, and only loopback network access. - The agent unit's comments say what the watchdog and the probation do. setup.sh's inline copies match.
The WATCHDOG=1 pings came from a goroutine of their own, so a daemon with a wedged local server or a deadlock still pinged and was never restarted, contrary to the unit's comment. The pings now depend on a heartbeat: before enrollment the poll loop bumps it, and once enrolled a probe of the agent's own /api/local/status (the HTTP server plus the sync, backend, update and clock state behind it) every 10 seconds. A heartbeat older than WatchdogSec withholds the ping, so a stuck agent is restarted 1 to 2.25 minutes after its last sign of life. Nothing in the probe waits on the network, so an idle or offline board is never restarted for it; the heartbeat uses the monotonic clock, so the agent stepping the wall clock does not either.
Its comment claimed a board in safe mode refuses commands. It does not: safe mode is only the safeMode flag in heartbeats, so the portal shows a board that keeps restarting. The comment now says so, and where crash loops after a self-update are actually handled. The counter is written with fsync like every other state file.
Section 12 describes the probation, rollback rules and strikes, and the journal with what the next run does from each state. The layout lists the two new root-owned files, section 14 says what the watchdog does and does not catch, and safe mode is described as the report it is.
…secure-dev Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6ae00becbb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case err == nil && h == rec.NewSHA256 && (rec.Phase == PhaseSwapping || rec.Phase == PhaseProbation): | ||
| // The new agent is in place and was not judged: judge it now. | ||
| return u.startProbation(ctx, rec, true) |
There was a problem hiding this comment.
Start the new agent when resuming an interrupted swap
If the updater is stopped after replacing the binary but before restarting the agent, then resumed without a reboot, the installed hash matches rec.NewSHA256 but the old agent process is still running. Passing resumed=true skips the restart in startProbation, so probation waits five minutes for a version that was never started, then rolls back and permanently rejects the valid release. This is reproducible with the existing harness by interrupting at swapped and applying again without calling reboot. Check the running version and start the installed agent when necessary before resuming probation.
Useful? React with 👍 / 👎.
The refresh stream is being removed in favour of a content-changed message on the agent WebSocket, so signing its upgrade is wasted work. This reverts merge 6ae00be. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
A run stopped after the swap but before the restart, then resumed without a reboot, left the previous agent running; probation waited for a version that never came and rejected a good release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
Fixes from the production-readiness survey of the MusallahBoard agent.
1. Clock keeper no longer fights NTP
Previously the keeper stepped the clock up to a persisted floor and then wrote the wall clock back into that floor. One upload from a laptop with a wrong clock pinned a board in the future permanently.
createdAtis more than a day ahead of NTP time are refused before anything is recorded.2. Self-update is a crash-safe transaction with a real probation
update-in-progress.json) is written before the swap; a power cut at any step is resolved deterministically on the next run.3. Kiosk liveness and TLS
https/wss, with the WebSocket on the backend's host.enroll --insecure-devopts out for development.Dropped from this PR
The signed refresh-channel upgrade was reverted: the refresh stream is being removed in favour of a content-changed message on the agent WebSocket.
Behaviour changes
httprefuses backend contact after this update until re-enrolled overhttpsor giveninsecure_dev = true.Not verified on hardware
Renderer crash detection (
Inspector.targetCrashed), path unit re-trigger after an interrupted probation.Testing
go build ./... && go vet ./... && go test -race ./...pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE