Skip to content

fix: do not panic in current::clock() when there is no current task - #357

Open
sarsko wants to merge 2 commits into
awslabs:mainfrom
sarsko:fix-clock-no-current-task
Open

sarsko wants to merge 2 commits into
awslabs:mainfrom
sarsko:fix-clock-no-current-task

Conversation

@sarsko

@sarsko sarsko commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

current::clock() called ExecutionState::with and then ExecutionState::current(), which unwraps current_task.id():

pub fn clock() -> VectorClock {
    ExecutionState::with(|state| {
        let me = state.current();          // -> self.current_task.id().unwrap()
        state.get_clock(me.id()).clone()
    })
}

There are three ways that aborts the process rather than reporting a failure:

  1. No current task. During ExecutionState::cleanup the current task is Stopped or Finished, so id() is None and the unwrap panics.
  2. AlreadyBorrowed. with panics if ExecutionState is already borrowed, which happens when a wake re-enters it — including from the panic hook while it tries to name the failing task.
  3. NotSet. with panics outside a Shuttle execution.

All three are reachable from ordinary user code, because every BatchSemaphore operation calls clock() — so any Drop handler that touches a modelled Mutex, RwLock or semaphore reaches it. Cleanup force-unwinds tasks that were still parked when the execution ended, which runs their destructors outside any task context.

Because the panic originates inside a destructor, Rust escalates it to a non-unwinding abort. The process dies and the failure report, the shrunk counterexample and the persisted schedule are all lost — so the user sees an abort instead of the bug they were actually hunting.

We hit this once in a 1000-job soak run, as exit 250:

ExecutionState::cleanup
  Continuation::drop -> corosensei force_unwind of a parked task
    <parked task drops its last Arc to a shared structure>
      <that structure's Drop> -> Mutex::try_lock
        BatchSemaphore::try_acquire -> acquire_permits -> current::clock()
          ExecutionState::current() -> current_task.id().unwrap()   <-- None
            panic in a destructor during cleanup -> abort

Fix

Use ExecutionState::try_with together with try_current(), so all three cases degrade to an empty clock instead of panicking:

pub fn clock() -> VectorClock {
    ExecutionState::try_with(|state| {
        let id = state.try_current().map(|me| me.id());
        id.map(|id| state.get_clock(id).clone())
    })
    .ok()
    .flatten()
    .unwrap_or_else(VectorClock::new)
}

An operation that belongs to no task has no causality to record, so an empty clock is the correct value rather than a fallback. try_with and try_current both already exist and are used defensively elsewhere.

Relationship to #346

This complements #346 rather than duplicating it. #346 stops the portfolio runner from leaving a stopped execution visible to drop handlers, by extending force_reset to leak state. This change makes the clock lookup itself safe, which also covers single-runner executions that reach cleanup after an ordinary failure — the case we hit, which #346 does not address. clock() still unwraps on main as of fa6f0be.

Note on the vector-clocks feature

Disabling vector clocks does not avoid this. The feature stubs the VectorClock type, but clock(), ExecutionState::get_clock and the BatchSemaphore call sites are not gated. A clocks-off build therefore still performs the panicking task lookup and then throws away a zero-sized result.

That also suggests a separate, optional improvement: gating the BatchSemaphore call sites on the feature would remove a per-acquire task lookup from every clocks-off build. Happy to do that in a follow-up if it is wanted — it is a performance change rather than a correctness one, so it is deliberately not in this PR.

Testing

cargo check -p shuttle-engine passes. I have not added a regression test: reproducing this needs an execution that ends with a task still parked whose unwind drops a modelled primitive, and I would rather have a maintainer confirm the intended shape than guess at one. Happy to add a test in the style of the portfolio_stop_force_unwinds_atomic_drop test from #346 if you point me at the preferred harness.

`current::clock()` unconditionally calls `ExecutionState::current()`, which
unwraps `current_task.id()`. During `ExecutionState::cleanup` the current task
is `Stopped` or `Finished`, so that unwrap panics.

This is reachable from ordinary user code. Every `BatchSemaphore` operation
calls `clock()`, so any `Drop` handler that touches a modelled `Mutex`,
`RwLock` or semaphore reaches it. Cleanup force-unwinds parked tasks, so a task
that is still parked when an execution ends runs its destructors from cleanup.
If such a destructor drops the last handle to a structure whose own `Drop`
takes a modelled lock, the unwrap fires inside a destructor and Rust escalates
it to a non-unwinding abort: the process dies and the failure report, the
shrunk counterexample and the schedule file are all lost.

Observed as an `exit 250` abort in a large soak run, from:

    ExecutionState::cleanup
      Continuation::drop -> force_unwind of a parked task
        <user task drops its last Arc>
          <user Drop> -> Mutex::try_lock
            BatchSemaphore::try_acquire -> acquire_permits -> current::clock()
              ExecutionState::current() -> current_task.id().unwrap()  <- None

Use `try_current()` and return an empty clock when there is no current task.
An operation that belongs to no task has no causality to record, so an empty
clock is the correct value rather than a fallback.

This complements awslabs#346. That change stops the portfolio runner from leaving a
stopped execution visible to drop handlers; this one makes the clock lookup
itself safe, which also covers single-runner executions reaching cleanup after
an ordinary failure.

Note that the `vector-clocks` feature does not avoid this. It stubs the
`VectorClock` type, but `clock()`, `get_clock` and the `BatchSemaphore` call
sites are not gated, so a clocks-off build still performs the panicking lookup
and then discards a zero-sized result.
`ExecutionState::with` panics on both `NotSet` and `AlreadyBorrowed`, so the
previous version still aborted when a destructor re-entered `ExecutionState`
while it was already borrowed -- the panic hook does exactly that while trying
to name the failing task.

Switch to `try_with` so all three cases degrade to an empty clock: no
`ExecutionState`, state already borrowed, and no current task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant