Repository navigation
fix(daemon-service): free a hook cycle's permit before its terminal - #4147
Merged
Merged
Conversation
A settled cycle fired its completion terminals while it still held its concurrency permit, so a host that sent the session's next boundary on that terminal was refused as Backpressured, and a lifecycle resume admitted in that window was dropped. Release the permit after the in-flight entry is removed and before the terminals fire. The lifecycle-less boundary test also assumed the resumed cycle had settled once its work reported running. That cycle reports no terminal, so the test now waits for it to settle before the next boundary, which otherwise supersedes it and inherits its lifecycle by design.
|
ScriptedAlchemy
enabled auto-merge
October 16, 2026 18:02
…-less-boundary-flake # Conflicts: # crates/tracedecay-daemon-service/src/invocation/types.rs
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
context_scout_tests::lifecycle_less_boundary_settles_and_resumes_on_its_sessions_lifecyclefailed 7 of 40 runs on master. Five failures were a saved edit refused asBackpressured, one was([214], true)where([214], false)was expected, and one run hung until the 300 s test timeout.Root cause
There were two causes, one in production and one in the test.
BoundedHookOrchestratorV1fired a settled cycle's completion terminals while the cycle still held its concurrency permit. The permit was dropped only aftersettle_operationreturned. A host that sends the session's next boundary as soon as it sees the terminal was refused asBackpressured. A lifecycle resume admitted in that window was dropped silently, which caused the hang.([214], true)failure.Fix
settle_operationnow takes the cycle's permit. It drops the permit after it removes the in-flight entry and before it fires the terminals.active_tasks() == 0) before it sends the next boundary.Verification
settled_cycle_admits_the_sessions_next_boundary_from_its_terminaladmits the next boundary from inside the terminal callback. Before the fix it fails every run withleft: Backpressured, right: Enqueued. That commit comes first in the branch.//crates/tracedecay-daemon-service:unit_test --runs_per_test=40: both Context Scout tests passed 40/40. Two runs failed in an unrelated test,profile_host_admission_replay::tests::not_applicable_pending_record_does_not_back_off(left: 3, right: 2).bazel build --config=clippy //crates/tracedecay-daemon-service/...andrustfmt_check_2024pass. Thetypes_testsandadvisorytests in//crates/tracedecay:unit_testpass (32 tests).Reconciled with #4144
#4144 fixed the same race from the other side. It dropped the permit before
settle_operationremoved the in-flight entry. The merge keeps one implementation.settle_operationnow drops the permit while it holds the in-flight lock, right after it removes the entry. Admission checks the in-flight maps and takes its permit under that same lock, so the two steps are atomic for every admission. With that, neither ordering's window remains:The terminals fire after the lock is released.
Verification after the merge:
unit_testran 40x. The Context Scout tests passed 40/40. One run failed in the unrelatednot_applicable_pending_record_does_not_back_off.//crates/tracedecay:unit_test, filtered totypes_testsandadvisory, passed 20x. That includes fix(daemon): root-cause seven load-only daemon test failures #4144'sreplayed_admission_after_retryable_failure_completes_a_fresh_cycle.std::thread::sleepbetween the unlock and the terminals, the Context Scout tests and the replayed-admission test passed 10/10.