Skip to content

fix(daemon-service): free a hook cycle's permit before its terminal - #4147

Merged
ScriptedAlchemy merged 3 commits into
masterfrom
fix/scout-lifecycle-less-boundary-flake
Oct 16, 2026
Merged

ScriptedAlchemy merged 3 commits into
masterfrom
fix/scout-lifecycle-less-boundary-flake

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Problem

context_scout_tests::lifecycle_less_boundary_settles_and_resumes_on_its_sessions_lifecycle failed 7 of 40 runs on master. Five failures were a saved edit refused as Backpressured, one was ([214], true) where ([214], false) was expected, and one run hung until the 300 s test timeout.

Root cause

There were two causes, one in production and one in the test.

  1. Production. BoundedHookOrchestratorV1 fired a settled cycle's completion terminals while the cycle still held its concurrency permit. The permit was dropped only after settle_operation returned. A host that sends the session's next boundary as soon as it sees the terminal was refused as Backpressured. A lifecycle resume admitted in that window was dropped silently, which caused the hang.
  2. Test. The test treated the resumed cycle as settled once its work reported running. A resumed cycle has no terminal. While it is still registered, the next saved edit supersedes it and inherits its lifecycle, as designed. That produced the ([214], true) failure.

Fix

  • settle_operation now takes the cycle's permit. It drops the permit after it removes the in-flight entry and before it fires the terminals.
  • The test waits for the resumed cycle to settle (active_tasks() == 0) before it sends the next boundary.

Verification

  • New test settled_cycle_admits_the_sessions_next_boundary_from_its_terminal admits the next boundary from inside the terminal callback. Before the fix it fails every run with left: Backpressured, right: Enqueued. That commit comes first in the branch.
  • //crates/tracedecay-daemon-service:unit_test --runs_per_test=40: both Context Scout tests passed 40/40. Two runs failed in an unrelated test, profile_host_admission_replay::tests::not_applicable_pending_record_does_not_back_off (left: 3, right: 2).
  • Mutation. A resume that leaves the boundary parked fails the lifecycle-less test 20/20.
  • bazel build --config=clippy //crates/tracedecay-daemon-service/... and rustfmt_check_2024 pass. The types_tests and advisory tests in //crates/tracedecay:unit_test pass (32 tests).

Reconciled with #4144

#4144 fixed the same race from the other side. It dropped the permit before settle_operation removed the in-flight entry. The merge keeps one implementation. settle_operation now drops the permit while it holds the in-flight lock, right after it removes the entry. Admission checks the in-flight maps and takes its permit under that same lock, so the two steps are atomic for every admission. With that, neither ordering's window remains:

The terminals fire after the lock is released.

Verification after the merge:

  • Daemon-service unit_test ran 40x. The Context Scout tests passed 40/40. One run failed in the unrelated not_applicable_pending_record_does_not_back_off.
  • //crates/tracedecay:unit_test, filtered to types_tests and advisory, passed 20x. That includes fix(daemon): root-cause seven load-only daemon test failures #4144's replayed_admission_after_retryable_failure_completes_a_fresh_cycle.
  • With a temporary 20 ms std::thread::sleep between the unlock and the terminals, the Context Scout tests and the replayed-admission test passed 10/10.
  • Clippy and rustfmt pass.

A settled cycle fired its completion terminals while it still held its
concurrency permit, so a host that sent the session's next boundary on
that terminal was refused as Backpressured, and a lifecycle resume
admitted in that window was dropped. Release the permit after the
in-flight entry is removed and before the terminals fire.

The lifecycle-less boundary test also assumed the resumed cycle had
settled once its work reported running. That cycle reports no terminal,
so the test now waits for it to settle before the next boundary, which
otherwise supersedes it and inherits its lifecycle by design.
@changeset-bot

changeset-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e8d3d0e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

…-less-boundary-flake

# Conflicts:
#	crates/tracedecay-daemon-service/src/invocation/types.rs
@ScriptedAlchemy
ScriptedAlchemy merged commit 21429c5 into master Oct 16, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant