fix(collaboration): give inspection the lane SCH-2 already promised it - #38
Merged
Merged
Conversation
SCH-2 required disposable inspection and control to use separate bounded lanes, and said inspection consumes no control capacity. The collaboration writer had one lane of capacity one carrying both. So any command in flight refused every read as busy, and `plexmaton-cli` reported that refusal as fatal from nine call sites: a delegation under load ended the session with "read the collaboration log for root placement: collaboration writer command lane is busy". That was the delegate smoke script's failure on main, not a flaky terminal. It reproduces on a loaded machine and not on an idle one, which is why it reached main twice. The seven read commands now admit on their own lane and wait for room there. A read carries no mutation to hand back, so waiting costs the caller nothing it owns, while refusing cost the whole session. The control slot keeps its capacity and its busy refusal, so a mutation still retains its exact attempt and shutdown still declines rather than blocking. One worker still owns the file (SCH-1) and now takes work from both lanes with control biased first, so queued reads cannot delay a mutation. `recv` is cancel-safe, so the losing branch loses no command, and a closed lane disables only its own branch so the other still drains. The lane policy lives in one helper instead of seven copies of the same send, which is what let the shared slot hide in plain sight. `CollaborationWriterError::Busy` stopped describing reads. `sch_2_a_read_is_admitted_while_the_control_slot_is_occupied` proves it: with the worker held and a mutation queued on the control lane, a read is admitted and waits instead of completing as busy. Restoring the shared slot fails that assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the delegate smoke failure that has made
mainred since #35 — and it was never a flakyterminal.
What actually happened
The dying screen from CI names it:
CollaborationWriterError::Busydocuments itself as transient — "the single command slot isoccupied; a query was not accepted". Two
plexmaton-cliread paths propagate it with?, so theprocess exits 1 and the user's session dies.
The rule already existed
SCH-2: "disposable inspection and control use separate bounded lanes… inspection consumes no
control capacity."
The writer had one lane,
COMMAND_CAPACITY = 1, carrying all seven read commands and everymutation. So any command in flight refused every read. Nine production call sites in
plexmaton-cli(collaboration.rs,attention.rs,placement.rs) each turn that into a fatalerror. Nine copies of a retry loop was the wrong shape; the lane was the bug.
It reproduces under load and not when idle, which is how it reached
maintwice.What changed
mutation to hand back, so waiting costs the caller nothing it owns — while refusing cost the
session.
attempt and
shutdownstill declines rather than blocking.first, so queued reads cannot delay a mutation.
recvis cancel-safe, so the losing branchloses no command; a closed lane disables only its own branch so the other still drains.
what let the shared slot hide in plain sight.
projection.rsloses 147 lines of duplication.CollaborationWriterError::Busystopped describing reads.specs/owned-scheduling.mdgains the sentence that a read waits while a mutation reports busy, and aRejected:note recording that the spec already forbade the single shared lane and the writer did itanyway.
Proof
sch_2_a_read_is_admitted_while_the_control_slot_is_occupied— with the worker held and a mutationqueued on the control lane, a read is admitted and waits rather than completing as busy. Restoring
the shared slot fails that assertion (verified by reverting the helper and watching it fail with
"the read is admitted beside the occupied control slot rather than refused").
Verification
Run outside the sandbox — 17 runtime tests bind loopback fixture servers, which a sandbox refuses.
cargo test -p plexmaton-runtime --libcargo test --workspace --lockedcargo clippy --workspace --all-targets --locked -- -D warningssmoke-delegateincludedsmoke-delegate.pyunder four parallel workspace test runsThe load run matters: that is the condition CI fails under.
No new document exceeds its budget — the phase file was trimmed back under after the stage entry
pushed it over.