Skip to content

fix(collaboration): give inspection the lane SCH-2 already promised it - #38

Merged
benjipeng merged 1 commit into
mainfrom
fix/collaboration-busy-read
Sep 19, 2026
Merged

benjipeng merged 1 commit into
mainfrom
fix/collaboration-busy-read

Conversation

@benjipeng

Copy link
Copy Markdown
Contributor

Fixes the delegate smoke failure that has made main red since #35 — and it was never a flaky
terminal.

What actually happened

The dying screen from CI names it:

Error: read the collaboration log for root placement: collaboration writer command lane is busy

CollaborationWriterError::Busy documents itself as transient — "the single command slot is
occupied; a query was not accepted". Two plexmaton-cli read paths propagate it with ?, so the
process exits 1 and the user's session dies.

The rule already existed

SCH-2: "disposable inspection and control use separate bounded lanes… inspection consumes no
control capacity."

The writer had one lane, COMMAND_CAPACITY = 1, carrying all seven read commands and every
mutation. So any command in flight refused every read. Nine production call sites in
plexmaton-cli (collaboration.rs, attention.rs, placement.rs) each turn that into a fatal
error. Nine copies of a retry loop was the wrong shape; the lane was the bug.

It reproduces under load and not when idle, which is how it reached main twice.

What changed

  • The seven read commands admit on their own bounded lane and wait for room. A read carries no
    mutation to hand back, so waiting costs the caller nothing it owns — while refusing cost the
    session.
  • The control slot keeps its capacity and its busy refusal, so a mutation still retains its exact
    attempt and shutdown still declines rather than blocking.
  • One worker still owns the file (SCH-1) and takes work from both lanes with control biased
    first
    , so queued reads cannot delay a mutation. recv is cancel-safe, so the losing branch
    loses no command; a closed lane disables only its own branch so the other still drains.
  • The lane policy now lives in one helper instead of seven copies of the same send — which is
    what let the shared slot hide in plain sight. projection.rs loses 147 lines of duplication.
  • CollaborationWriterError::Busy stopped describing reads.

specs/owned-scheduling.md gains the sentence that a read waits while a mutation reports busy, and a
Rejected: note recording that the spec already forbade the single shared lane and the writer did it
anyway.

Proof

sch_2_a_read_is_admitted_while_the_control_slot_is_occupied — with the worker held and a mutation
queued on the control lane, a read is admitted and waits rather than completing as busy. Restoring
the shared slot fails that assertion
(verified by reverting the helper and watching it fail with
"the read is admitted beside the occupied control slot rather than refused").

Verification

Run outside the sandbox — 17 runtime tests bind loopback fixture servers, which a sandbox refuses.

Check Result
cargo test -p plexmaton-runtime --lib 248 pass, ~1.09 s, 10 consecutive clean runs
cargo test --workspace --locked exit 0
cargo clippy --workspace --all-targets --locked -- -D warnings exit 0
All seven CI smoke scripts every one exit 0, smoke-delegate included
smoke-delegate.py under four parallel workspace test runs 3/3 pass
citations / frames / fmt / file length / crate graph / typos pass

The load run matters: that is the condition CI fails under.

No new document exceeds its budget — the phase file was trimmed back under after the stage entry
pushed it over.

SCH-2 required disposable inspection and control to use separate bounded lanes, and said inspection
consumes no control capacity. The collaboration writer had one lane of capacity one carrying both.
So any command in flight refused every read as busy, and `plexmaton-cli` reported that refusal as
fatal from nine call sites: a delegation under load ended the session with "read the collaboration
log for root placement: collaboration writer command lane is busy".

That was the delegate smoke script's failure on main, not a flaky terminal. It reproduces on a
loaded machine and not on an idle one, which is why it reached main twice.

The seven read commands now admit on their own lane and wait for room there. A read carries no
mutation to hand back, so waiting costs the caller nothing it owns, while refusing cost the whole
session. The control slot keeps its capacity and its busy refusal, so a mutation still retains its
exact attempt and shutdown still declines rather than blocking.

One worker still owns the file (SCH-1) and now takes work from both lanes with control biased first,
so queued reads cannot delay a mutation. `recv` is cancel-safe, so the losing branch loses no
command, and a closed lane disables only its own branch so the other still drains.

The lane policy lives in one helper instead of seven copies of the same send, which is what let the
shared slot hide in plain sight. `CollaborationWriterError::Busy` stopped describing reads.

`sch_2_a_read_is_admitted_while_the_control_slot_is_occupied` proves it: with the worker held and a
mutation queued on the control lane, a read is admitted and waits instead of completing as busy.
Restoring the shared slot fails that assertion.
@benjipeng
benjipeng merged commit 1a3f652 into main Sep 19, 2026
3 checks passed
@benjipeng
benjipeng deleted the fix/collaboration-busy-read branch September 19, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant