Skip to content

refactor(work): run adjudications and placements as writer jobs - #4142

Merged
ScriptedAlchemy merged 3 commits into
masterfrom
refactor/storage-work-leaf-jobs
Oct 16, 2026
Merged

ScriptedAlchemy merged 3 commits into
masterfrom
refactor/storage-work-leaf-jobs

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Step 5 of the storage redesign (docs/plans/storage/README.md), fourth slice. Four more Work operations run as one writer job each, in the shape #4139 set: the unchanged contracts decision runs inside WorkSqliteStorage::transact over a storage scoped to that transaction. A native worker awaiting any of them parks on neither the writer nor a reader. One commit per operation:

  • AdjudicateDuplicate. The record step moves to WorkDuplicateAdjudicationRecordPortV1, a trait without Send + Sync that only the job storage implements. Its evidence (the current Work and topology generation) was already read on a reader before the transaction, so the job never replays the product journal while holding the writer.
  • AdjudicateLeak. The attempt, placement, effect holder, the global-db delivery census and the process-registry answer are collected before the job and passed in owned. The replay check, the decision and the write run in the job.
  • AdmitPlacement and ReleasePlacement. The Git observation (discovery, status and git rev-list) is hoisted out of the job. Admit observes command.target first. Release reads the placement on a reader (load_placement_async), refuses a stale expected version before touching Git, and observes. Inside the job it publishes only if the target and authority version are unchanged, otherwise it returns the existing authority conflict. publish_placement moves to WorkPlacementPublicationPort. The handle keeps the reads that preflight, status and cleanup use, and shares their SQL with the job.

Each handle-based write (duplicate record, leak record, placement publish) is deleted in the commit that moved its last caller. work_authority is now exported from contracts, so the release pre-read uses the same authority the service derives.

Behavior changes:

  • Admit now observes the target even when the request replays an existing placement. Before, a replay skipped Git.
  • The service's deadline and cancellation check now runs after the observation. It still runs before anything is published.

Test plan

crates/tracedecay-application/tests/application_suite/work_leaf_jobs.rs adds four tests. Each was mutation-checked:

  • Duplicate. No writer or reader waits on a native worker, revision 1 commits, and a second command on the same pair gets the revision conflict. Restoring the handle-based record fails it (writer waits left 7). Removing the in-job revision compare fails it.
  • Leak. No waits; append, replay, then the revision conflict. Restoring the handle path fails it (writer waits left 16). Removing the revision compare fails it.
  • Placement, observation. Admit then release with no waits. A recording observer stores each thread it runs on, and none is the writer thread. Moving either observation into the job fails it (it saw tracedecay-rusqlite-writer). A synchronous release pre-read fails the reader-wait assertion.
  • Placement, race. A competing release commits while the first release observes, and the first gets authority-conflict. Removing the in-job version check fails it (it returned already-released).
  • bazel test --config=clippy:
    • rusqlite-runtime: unit_test and rusqlite_suite
    • contracts: unit_test and contracts_suite
    • application: unit_test and application_suite
    • daemon-service: unit_test
  • bazel build --config=clippy for //crates/tracedecay:unit_test, :daemon_suite and :mcp_suite
  • //crates/tracedecay:mcp_suite passes release_placement_publishes and work_attempt_consumers_read_the_public_start_attempt_effect, which cover admit, duplicate adjudication and release end to end through the daemon.
  • rustfmt_check_2024 for each touched crate
  • Full //crates/tracedecay:daemon_suite, :mcp_suite, :session_suite, :unit_test in CI

Refs #4052


Devin Review

AdjudicateDuplicate now runs the contracts duplicate service inside one ExactSqlHandle::transact job over WorkSqliteJobStorage. The command replay check, the attempt checks, the revision compare and the append commit together, and the awaiting task parks no native thread.

The append moves to a WorkDuplicateAdjudicationRecordPortV1 without a Send + Sync bound, implemented only by the job storage; the handle-based append had no other production caller and is deleted. The current Work and topology generations the command must cite are still read from the product graph before the job, on a reader, and passed in owned, so the graph read never holds the writer.
AdjudicateLeak now runs the contracts leak service inside one ExactSqlHandle::transact job over WorkSqliteJobStorage. The command replay read, the verdict, the attempt check, the revision compare and the append commit together, and the awaiting task parks no native thread.

The leak evidence the verdict reads (the attempt, its placement and effect holder, the global-db delivery census and the process registry's live-owner answer) is still collected before the job and handed to it owned, so no global-db read or registry lookup runs on the writer. WorkLeakAdjudicationStoragePortV1 drops its Send + Sync bound; the job storage is its only implementation and the handle-based one is deleted.
AdmitPlacement and ReleasePlacement now run the contracts placement service inside one ExactSqlHandle::transact job over WorkSqliteJobStorage, and the awaiting task parks no native thread. The Git observation (discovery, status and the rev-list count) no longer runs inside a storage transaction.

Admit observes command.target before the job and hands the job the owned observation; the replay check, the exclusivity read and the publication commit together. Release reads the placement on a reader, refuses a stale expected authority version before touching Git, observes the target, and the job then publishes only while the placement still has that target and version, refusing with the existing authority conflict when another release committed in between.

publish_placement moves to a WorkPlacementPublicationPort, and WorkPlacementStoragePort drops its Send + Sync bound so the job storage can implement both. The handle-based publication had no other production caller and is deleted; the handle keeps the reads that preflight, status and cleanup use, sharing their SQL with the job. work_authority is exported so the release pre-read uses the same authority the service derives.
@changeset-bot

changeset-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 76cc22d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +333 to +336
self.storage
.transact(move |storage| {
tracedecay_contracts::WorkLeakAdjudicationServiceV1::new(storage, evidence.clone())
.adjudicate(&context, command.clone(), scan_started_at, scan_deadline)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Queued leak scan records stale verdict

When Work state changes while the writer is queued, adjudicate_leak classifies an older snapshot as current. DaemonWorkLeakEvidenceV1::inspect timestamps that snapshot only after the job starts, so it can publish a false leak verdict.

Learn more

The daemon gathers the attempt, process-owner status, placement, effect and delivery census before enqueuing this job in read_leak_evidence. The writer can process other work before this job begins. inspect then assigns a new completion timestamp to the previously gathered snapshot and derives a verdict from it. State transitions during the queue wait are invisible, but the receipt attributes the verdict to the later scan completion.

Example: A running attempt has no live owner when the daemon reads it. The owner resumes before the queued job starts. The job still records AttemptWithoutLiveOwner as though the scan had just found an ownerless attempt.

Recommended fix: Determine the scan result and its completion timestamp before enqueueing, or revalidate the versioned evidence at the writer boundary before committing. Keep the replay and revision comparison in the writer job; do not timestamp frozen evidence at job execution time.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +396 to +397
if current.authority_version() != command.expected_authority_version {
return Err(tracedecay_contracts::WorkPlacementStorageError::AuthorityConflict.into());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Stale placement masks admission refusal

For a cancelled or expired release with a stale version, release_placement returns authority conflict before checking admission. The service's admission check would instead return the cancellation or expiry refusal.

Suggested change
if current.authority_version() != command.expected_authority_version {
return Err(tracedecay_contracts::WorkPlacementStorageError::AuthorityConflict.into());
tracedecay_contracts::ApplicationProblem::ensure_admitted(&context, command.occurred_at)?;
if current.authority_version() != command.expected_authority_version {
return Err(tracedecay_contracts::WorkPlacementStorageError::AuthorityConflict.into());
}

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@ScriptedAlchemy
ScriptedAlchemy merged commit c6d7dcc into master Oct 16, 2026
11 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant