Skip to content

Production-readiness fixes: clock, self-update safety, kiosk liveness, TLS - #4

Merged
IbraTech04 merged 23 commits into
developfrom
claude/wizardly-curie-s90b5i
Nov 19, 2026
Merged

IbraTech04 merged 23 commits into
developfrom
claude/wizardly-curie-s90b5i

Conversation

@IbraTech04

@IbraTech04 IbraTech04 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes from the production-readiness survey of the MusallahBoard agent.

1. Clock keeper no longer fights NTP

Previously the keeper stepped the clock up to a persisted floor and then wrote the wall clock back into that floor. One upload from a laptop with a wrong clock pinned a board in the future permanently.

  • With NTP synchronised, NTP is the authority. The keeper never moves the clock and lowers a floor that is ahead of NTP time. Boards already in the bad state heal themselves once NTP syncs.
  • Without NTP, the floor advances by monotonic elapsed time from the last trusted time, never from the unconfirmed wall clock.
  • Uploader time is ignored when NTP is synced and never raises the floor.
  • Packages whose signed createdAt is more than a day ahead of NTP time are refused before anything is recorded.

2. Self-update is a crash-safe transaction with a real probation

  • A durable journal (update-in-progress.json) is written before the swap; a power cut at any step is resolved deterministically on the next run.
  • Probation: the new agent must stay up without restarts for 4 minutes and, if the board was connected before the update, reconnect within 3 minutes. Offline boards are not penalised.
  • Crash loop or failure to start: roll back and reject. Failure to reconnect only: roll back with a strike, reject on the second.
  • Resuming an interrupted swap without a reboot now restarts the agent unless it already reports the new version (review finding).
  • The systemd watchdog ping is gated on a liveness heartbeat.

3. Kiosk liveness and TLS

  • A crashed or hung Chromium page is reported as not alive and healed: reload, then restart the kiosk unit with backoff.
  • Enrollment and the daemon require https / wss, with the WebSocket on the backend's host. enroll --insecure-dev opts out for development.

Dropped from this PR

The signed refresh-channel upgrade was reverted: the refresh stream is being removed in favour of a content-changed message on the agent WebSocket.

Behaviour changes

  • A board enrolled over plain http refuses backend contact after this update until re-enrolled over https or given insecure_dev = true.
  • Boards installed before this change keep their old systemd units.

Not verified on hardware

Renderer crash detection (Inspector.targetCrashed), path unit re-trigger after an interrupted probation.

Testing

go build ./... && go vet ./... && go test -race ./... pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE

CapClockFloor lowers clockFloor and signedClockFloor to at most a given
time, in one locked update. The clock keeper uses it when the clock is
NTP-synchronised and the floor is ahead of it: such a floor was raised by
something wrong and must not keep pushing the clock forward.
The backend accepted any WebSocket on /api/refresh-musallahboard that
named a known device id, and a newcomer replaced the board's own
connection. Device ids are not secret, so anyone could keep evicting a
board from its push channel and watch when pushes arrive.

The upgrade is now a device-signed GET (section 9.2: the path without
the query, an empty body), signed afresh for every dial with the same
helper as content-bundle and weather. The deviceId query stays, so a
backend that predates this still finds the board by it.

The backend verifies the signature when present; unsigned upgrades stay
accepted, without taking over from a signed one, until the owner sets
musallahboard.board-stream.require-signature.
checkKiosk fell through to `systemctl is-active` for any probe failure
other than a missing getStatus() hook. A crashed tab ("Aw, Snap!") or hung
JavaScript leaves the kiosk unit active, so a blank board showed as
healthy in the admin console.

- cdp: the page probe opens one connection, sends Inspector.enable (which
  Chromium answers with Inspector.targetCrashed on a tab whose renderer is
  gone) and then Runtime.evaluate. Failures are classified:
  ErrDebuggerUnreachable / ErrNoPageTarget (no browser to ask yet),
  ErrPageCrashed, ErrPageUnresponsive (connection accepted, no answer in
  time), ErrPageNavigating (context destroyed by a navigation),
  ErrPageThrew. CDP error replies are a typed RPCError. Error texts are
  unchanged.
- telemetry: crashed or unresponsive is kioskAlive=false; splash, update
  screen, old board apps, a page mid-navigation or a throwing hook are
  alive; only an unreachable debugger falls back to the unit state.
- First tests for internal/cdp, against a fake debugger endpoint.
The kiosk watcher only navigated away from chrome-error: pages. A renderer
crash or hung JavaScript left a blank board until someone noticed.

Every 30 s the watcher now probes the page (Inspector.enable plus a
trivial Runtime.evaluate, 5 s):

- a crashed renderer after one probe, or a page that has not answered
  for three probes in a row (90 s; a Pi decoding a large poster can miss
  one), is reloaded;
- if it is still dead at the next threshold, the kiosk unit is restarted
  through the same sudo path as the kiosk.restart command;
- restarts are at least 5 min apart, doubling to 1 hour, and that backoff
  only starts over after an hour of a healthy page. A page that never
  comes back costs at most one reload and one restart an hour; one that
  recovers on reload is reloaded whenever it dies.

The splash, the update screen (skipped entirely while it is up), old board
apps without getStatus(), a page mid-navigation and a browser that is
not up yet are never counted as failures.
A laptop with a wrong clock uploading on the service port could pin a
board in the future for good: the uploader's time became the clock
floor, the keeper persisted the (now wrong) wall clock into it every ten
minutes, and when timesyncd corrected the clock the keeper pushed it
forward again, every ten minutes. Prayer times were wrong and, past five
minutes of skew, the backend refused the board's signed requests. A
manual date or a wrong RTC did the same. Only editing state.json helped.

- With NTP synchronised, the keeper never moves the clock and ignores
  uploaders' clocks. A floor (or signed floor) ahead of NTP time is wrong:
  it is lowered to NTP time, with a warning in the journal. Boards already
  in the bad state recover by themselves once NTP has synchronised.
- NTP time is written to the floor once two checks an interval apart
  agree with the monotonic clock, so a clock stepped by hand between
  timesyncd's polls is not taken for NTP time.
- Without NTP the wall clock is never written to the floor: the floor
  advances from the last trusted time (the floor itself, a signed
  createdAt, NTP time) by the monotonic clock. A board that loses power
  still comes back no earlier than the time it last knew.
- An uploader's time sets the clock (and the RTC) but no longer the
  floor, so the next uploader with a right clock, or NTP, undoes a wrong
  one. It still cannot move the clock below the floor or more than 90
  days past it.

The timesyncd check moves to internal/clock/timesync so the importer can
ask it too.
enroll took any --backend scheme and saved whatever websocketUrl the
backend returned. Over http the enrollment reply, which pins the content
signing keys and the WebSocket URL, can be rewritten by anyone on the LAN.

- config: CheckBackendURL (https://, a host, no credentials, query or
  fragment) and CheckWebSocketURL (wss://, same host as the backend; the
  port may differ, since TLS binds the host name and not the port). No
  override for split-host deployments. A new explicit key,
  insecure_dev = true, allows http:// and ws:// for development boards;
  the host rule still applies. Config.CheckTransport applies both.
- enroll: the backend URL is checked before the token is sent, and the
  returned WebSocket URL before anything is saved, so a refused reply
  leaves the board unenrolled and pins no keys. --insecure-dev (with a
  loud warning) writes insecure_dev = true. Tests run against TLS
  servers now.
A backend (or build machine) whose clock was briefly far ahead signs a
createdAt in, say, 2030. Every verified createdAt raises the board's clock
floor, so that one package pinned every board that took it in 2030.

With NTP synchronised, a package whose createdAt is more than a day past
the board's clock is now refused ("An update is dated in the future: the
computer that made it has a wrong clock"; the times are in the detail),
and so is its Preflight. Nothing about it is recorded: not its time, and
not its sequence, so the backend's next right package installs even if
its sequence is lower. A createdAt less than a day ahead installs, but
raises the floor only up to NTP time.

Without NTP the board has nothing better than the signed time, and it
counts as before. Content sequence is unchanged: still strictly
increasing, with no upper bound.
The page told the uploader a drifting board clock would be "corrected
when you send an update". A board set from the internet no longer takes
an uploader's time, so the page now says the uploader's clock is probably
the wrong one.
Section 10 says when the keeper moves the clock and what it writes to the
floor, how a floor ahead of NTP time heals, that an uploader's time no
longer reaches the floor, and what this trades: what rollback resistance
remains without NTP, and what whoever controls NTP can do. Sections 4 and
8 and the threat table say that packages signed more than a day ahead of
an NTP clock are refused.
Nothing checked the backend and WebSocket URLs after enrollment, so a
board enrolled over http kept syncing content, dialling ws:// and taking
remote commands in the clear.

- startBoard runs config.CheckTransport before it builds anything that
  talks to the backend. When it fails, no syncer (content sync, refresh
  channel, release checks) and no command WebSocket are started; the
  board keeps showing what it has. The reason and the fix go to the log
  at every start, to `systemctl status` (STATUS=), and to the local
  status, so `musallahboard-agent status` shows "LensBridge: NOT
  connected" and "Sync: failing" with it, as does the board app.
- With insecure_dev = true a loud warning is logged at every start.
- wsclient never dials a WebSocket URL that fails the check (backstop).
- trust fetch refuses a plain-http backend unless insecure_dev is set in
  the config, or --insecure-dev is passed with --backend.
- README and vm/README: enroll a local plain-http backend with
  --insecure-dev; what happens to a board already enrolled over http and
  how to fix it.
- Makefile: install-remote examples use an https backend.
- architecture.md: the transport rule and insecure_dev (section 11), the
  kiosk probe and its reload/restart backoff (section 14), and the
  security table.
The updater judged a new agent only by "/api/local/status reports the new
version within 120 s". A crash after that, a release that cannot reach
LensBridge, or a power cut between the swap and the verdict all left the
new agent installed and never judged again.

- The swap is a transaction with a root-owned journal,
  /usr/lib/musallahboard/update-in-progress.json, written (temp, fsync,
  rename, fsync dir) before the binary is replaced and at every phase:
  swapping, probation, committing, rolling-back. The next run finishes
  whatever it finds, deciding by phase and by the SHA-256 of what is
  installed, so a power cut at any step resolves the same way. The `ready`
  marker is put back while the transaction is open, so boards whose path
  unit predates the journal also start the updater at boot.
- Probation: the new agent must report its version within 2 minutes, then
  run 4 minutes with no restart (systemd's MainPID, NRestarts and
  ActiveState) and, if the board was connected to LensBridge before the
  update, connect again (authenticated WebSocket or a content sync) within
  3 minutes. A board that was offline is not asked to be online.
- A crash loop, a restart or not coming up rolls back and rejects the
  version. Not reconnecting rolls back but only counts a strike: the
  backend or network may be at fault, so the version is rejected the
  second time.
- A probation cut short (shutdown, the unit's timeout) leaves the journal
  and is judged again from scratch, at most 3 times. A rollback that has
  begun always finishes; one that cannot restore agent.prev says so
  (rollback-failed) instead of claiming it went back.
- The path unit also watches the update journal, so an update a power cut
  or reboot interrupted is finished at boot.
- TimeoutStartSec 15 minutes: room for the probation and a rollback.
- The update unit no longer gets write access to /etc/musallahboard: it
  only reads trust.json (the comment about `trust fetch` was stale). It
  gets /usr/share/musallahboard instead, which `splash install` after an
  update writes and which ProtectSystem=strict made read-only, and only
  loopback network access.
- The agent unit's comments say what the watchdog and the probation do.

setup.sh's inline copies match.
The WATCHDOG=1 pings came from a goroutine of their own, so a daemon with
a wedged local server or a deadlock still pinged and was never restarted,
contrary to the unit's comment.

The pings now depend on a heartbeat: before enrollment the poll loop bumps
it, and once enrolled a probe of the agent's own /api/local/status (the
HTTP server plus the sync, backend, update and clock state behind it)
every 10 seconds. A heartbeat older than WatchdogSec withholds the ping,
so a stuck agent is restarted 1 to 2.25 minutes after its last sign of
life. Nothing in the probe waits on the network, so an idle or offline
board is never restarted for it; the heartbeat uses the monotonic clock,
so the agent stepping the wall clock does not either.
Its comment claimed a board in safe mode refuses commands. It does not:
safe mode is only the safeMode flag in heartbeats, so the portal shows a
board that keeps restarting. The comment now says so, and where crash
loops after a self-update are actually handled. The counter is written
with fsync like every other state file.
Section 12 describes the probation, rollback rules and strikes, and the
journal with what the next run does from each state. The layout lists the
two new root-owned files, section 14 says what the watchdog does and does
not catch, and safe mode is described as the report it is.
…secure-dev

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T06:25:28.701930Z 6ae00be PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ae00becbb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +539 to +541
case err == nil && h == rec.NewSHA256 && (rec.Phase == PhaseSwapping || rec.Phase == PhaseProbation):
// The new agent is in place and was not judged: judge it now.
return u.startProbation(ctx, rec, true)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Start the new agent when resuming an interrupted swap

If the updater is stopped after replacing the binary but before restarting the agent, then resumed without a reboot, the installed hash matches rec.NewSHA256 but the old agent process is still running. Passing resumed=true skips the restart in startProbation, so probation waits five minutes for a version that was never started, then rolls back and permanently rejects the valid release. This is reproducible with the existing harness by interrupting at swapped and applying again without calling reboot. Check the running version and start the installed agent when necessary before resuming probation.

Useful? React with 👍 / 👎.

The refresh stream is being removed in favour of a content-changed
message on the agent WebSocket, so signing its upgrade is wasted work.

This reverts merge 6ae00be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
A run stopped after the swap but before the restart, then resumed
without a reboot, left the previous agent running; probation waited for
a version that never came and rejected a good release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YY5BHXmNUhs48eubkKCNuE
@IbraTech04 IbraTech04 changed the title Production-readiness fixes: clock, self-update safety, kiosk liveness, TLS, signed refresh stream Production-readiness fixes: clock, self-update safety, kiosk liveness, TLS Nov 19, 2026
@IbraTech04
IbraTech04 merged commit 1f6ddac into develop Nov 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants