Stop leaking entered spans when another thread holds a scoped subscriber

[sarsko]

Fixes the intermittent full-suite failure that hits a different test each run:

assertion `left != right` failed: tried to clone a span (Id(..)) that already closed

It is independent of #352 and #353 and branches from main.

Root cause

Shuttle's span bookkeeping called Span::current() from inside tracing::dispatcher::get_default:

tracing::dispatcher::get_default(|subscriber| {
    while let Some(span_id) = tracing::Span::current().id().as_ref() {   // nested
        state.current_mut().span_stack.push(tracing::Span::current().clone());
        subscriber.exit(span_id);
    }
    if let Some(span_id) = state.top_level_span.id().as_ref() {
        subscriber.enter(span_id)                                          // unconditional
    }
});

get_default marks the thread's dispatcher state as in use while its callback runs. Whenever any thread in the process holds a scoped default subscriber, a Span::current() nested inside that callback returns Span::none(). That happens even on a thread that never set a default of its own. Standalone, with nothing from Shuttle involved:

nested Span::current(), no scoped dispatcher anywhere:   Some(Id(1))
nested Span::current(), another thread holds a scoped:   None
unnested Span::current(), another thread holds a scoped: Some(Id(1))

So in exit_task_span the drain loop exits nothing, but the unconditional enter(top_level_span) still runs. Each scheduling step leaks one entry on the thread's entered-span stack. After those spans' handles drop, the entries point at closed spans, and the next Span::current() that resolves to one of them panics in clone_span.

The scoped defaults come from the tests themselves. basic::{tag, task, metrics, labels} install one with tracing::subscriber::set_default, and the victim is whichever test is mid-execution on another thread at that moment.

Evidence

I captured one failure with a locally patched tracing-subscriber. It journals every entered-span-stack mutation and refcount change inside the subscriber, so Shuttle's own code wasn't touched. Every earlier attempt that instrumented Shuttle's hot path hid the bug completely. At the moment of the panic, the failing thread's entered-span stack was ~96 deep, made mostly of repeated duplicate pushes of the same execution span id (one id appeared ~55 times), and every entry pointed at an already-freed span. The id being cloned belonged to a closed new_task span.

Why every standard way of isolating it failed:

	span-closed failures
unfixed, default parallel suite	~1 run in 4-6
unfixed, --test-threads=1	0 / 18 runs
unfixed, skipping the 4 set_default modules	0 / 16 runs
fixed, default parallel suite	0 / 20 runs

Fix

Every exit and enter now goes through Span::with_subscriber, which hands over the span's own dispatcher without calling get_default. There were three sites:

• exit_task_span: the flake. Leaked one entry per scheduling step.
• ResetSpanOnDrop::drop: leaked one entry per Runner::run. This site is not the flake, but the leak is real and accumulates on test threads.
• enter_task_span: same pattern, but I could not make it leak. It only runs right after exit_task_span has drained the stack, so it has exactly one span to exit. Changed for consistency, not as a tested fix.

Regression test

basic::tracing::span_bookkeeping_with_scoped_default_on_another_thread holds a scoped default on another thread. It enters one span around a Shuttle run and another inside it, across a context switch, and asserts three things: no entries leak, the inner span is still current after the switch, and the caller's span is current again after the run.

I checked that the test catches each leaking site independently:

code under test	result
unfixed main	fails, 290 entries leaked
fix with only the exit_task_span hunk reverted	fails, 102 leaked
fix with only the ResetSpanOnDrop hunk reverted	fails, 1 leaked
fix applied	passes

On main this is a correctness problem, not only leftover state. In a probe run, the span a user entered before a context switch was no longer current after it, and the caller's own span was not current after check_random returned.

Testing

• cargo fmt --check, cargo clippy --workspace --all-targets, cargo doc clean
• cargo test --release -p shuttle: 427 passed, 0 failed (the 426 existing tests plus the new one)
• All other workspace crates pass
• The previously #[ignore]d tracing tests (test_tracing_nested_spans, instrumented_futures, test_stacks_cleaned_on_panic) pass with --include-ignored. I haven't un-ignored them here, since their comment gives separate reasons.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.