Fix macOS freeze when PortAudio playback ends as the stream is stopped

[Muskworker]

(tl;dr I was encountering a freeze, Claude found it was apparently due to a deadlock when using PortAudio and wrote up a suggested fix. appears to work for me; hopefully it can be useful to y'all.)

Problem

On macOS, Aegisub occasionally freezes permanently (0% CPU, must be force quit) at the end of an audio "play line". Seen on macOS 26.6.1, Apple Silicon, PortAudio 19.7.0.

sample of the frozen process shows a lock-order inversion between two threads that are both inside AudioOutputUnitStop:

Main thread (holds the audio unit's mutex, wants the HAL IO mutex):

AudioController::Stop()  audio_controller.cpp
FinishStoppingStream  (libportaudio)
AudioOutputUnitStop
AudioDeviceStop_mac_imp
HALC_ProxyIOContext::StopIOProc
HALB_Mutex::Lock
__psynch_mutexwait

CoreAudio IO thread (holds the HAL IO mutex, wants the audio unit's mutex):

HALC_ProxyIOContext::IOWorkLoop
...
AudioIOProc  (libportaudio)
AudioOutputUnitStop
std::recursive_mutex::lock
__psynch_mutexwait

PortAudioPlayer::paCallback returns paAbort once the playback range is exhausted, which makes PortAudio's CoreAudio backend call AudioOutputUnitStop from the IO thread (stop_stream: in pa_mac_core.c). AudioController::OnPlaybackTimer notices the end of the range at about the same moment and calls Pa_StopStream from the main thread. When the two coincide, they deadlock.

This is related to, but not fixed by, PortAudio/portaudio#1174 / PortAudio/portaudio#1175: that PR removes the same inversion for startStopCallback, but leaves the stop_stream: path alone. The reproducer below still deadlocks against both PortAudio master (873e3c8) and the PR branch (1240f06).

Fix

Never stop the stream from the callback. Past the end of the range the callback pads with silence and returns paContinue; IsPlaying() reports false once the end has been reached, so AudioController's existing timer stops the stream from the main thread via Stop(). current and end, which are shared with the callback, become atomics.

A side effect is that Play() on a stream that has run out but has not been stopped yet now simply resumes, instead of calling Pa_StartStream on a stream that is not stopped.

Testing

• A standalone reproducer that mimics the play/stop pattern against PortAudio 19.7.0 (silent output; source below) deadlocks with the same two stacks within a few hundred iterations when the callback returns paAbort (iterations 64 and 459 in two runs), and completes 3000 iterations when the callback pads with silence instead.
• Built on macOS 26.6.1 (arm64); meson test passes; play line / play to end behave as before in manual use.
• Not tested on Windows or Linux.

pa_stop_race.c

/*
 * Reproducer for the PortAudio/CoreAudio stop deadlock seen in Aegisub.
 *
 * Mimics Aegisub's PortAudioPlayer: a callback stream plays a short range,
 * and the main thread calls Pa_StopStream() as soon as it sees the range
 * has ended (as AudioController::OnPlaybackTimer does).
 *
 *   abort   - old behaviour: callback returns paAbort at end of range, so
 *             PortAudio stops the audio unit from the IO thread while the
 *             main thread stops it too.
 *   silence - fixed behaviour: callback keeps running and outputs silence;
 *             only the main thread ever stops the stream.
 *
 * Only silence is ever written to the device.
 *
 * Build: cc -O2 pa_stop_race.c -o pa_stop_race $(pkg-config --cflags --libs portaudio-2.0)
 * Usage: pa_stop_race abort|silence [iterations]
 * Exit:  0 = all iterations completed, 2 = deadlock detected
 */
#include <portaudio.h>

#include <mach/mach_time.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define SAMPLE_RATE 48000
#define CHANNELS 2
#define WATCHDOG_SECONDS 5

static int use_abort;
static atomic_long remaining;
static atomic_long iteration;
static _Atomic uint64_t last_cb;   /* mach time of the callback that finished the range */
static _Atomic uint64_t cb_period; /* mach time between consecutive callbacks */

static int callback(const void *in, void *out, unsigned long frames,
	const PaStreamCallbackTimeInfo *time, PaStreamCallbackFlags flags, void *user)
{
	(void)in; (void)time; (void)flags; (void)user;
	memset(out, 0, frames * CHANNELS * sizeof(short));

	static uint64_t prev;
	uint64_t now = mach_absolute_time();
	if (prev) atomic_store(&cb_period, now - prev);
	prev = now;

	long left = atomic_load(&remaining);
	if (left > 0) {
		atomic_store(&last_cb, now);
		atomic_store(&remaining, left - (long)frames);
		return paContinue;
	}
	return use_abort ? paAbort : paContinue;
}

static void *watchdog(void *arg)
{
	(void)arg;
	long last = -1;
	for (;;) {
		sleep(WATCHDOG_SECONDS);
		long now = atomic_load(&iteration);
		if (now == last) {
			printf("DEADLOCK: no progress for %d seconds at iteration %ld (pid %d)\n",
				WATCHDOG_SECONDS, now, getpid());
			fflush(stdout);
			if (getenv("PA_DEADLOCK_PARK")) pause();
			_exit(2);
		}
		last = now;
	}
	return NULL;
}

#define CHECK(call) do { PaError e = (call); if (e != paNoError) { \
	printf("%s failed at iteration %ld: %s\n", #call, atomic_load(&iteration), Pa_GetErrorText(e)); \
	return 1; } } while (0)

int main(int argc, char **argv)
{
	if (argc < 2 || (strcmp(argv[1], "abort") && strcmp(argv[1], "silence"))) {
		fprintf(stderr, "usage: %s abort|silence [iterations]\n", argv[0]);
		return 1;
	}
	use_abort = !strcmp(argv[1], "abort");
	long iterations = argc > 2 ? atol(argv[2]) : 2000;

	CHECK(Pa_Initialize());

	PaStreamParameters p = {0};
	p.device = Pa_GetDefaultOutputDevice();
	p.channelCount = CHANNELS;
	p.sampleFormat = paInt16;
	p.suggestedLatency = Pa_GetDeviceInfo(p.device)->defaultLowOutputLatency;

	PaStream *stream;
	CHECK(Pa_OpenStream(&stream, NULL, &p, SAMPLE_RATE, 0,
		paPrimeOutputBuffersUsingStreamCallback, callback, NULL));

	pthread_t tid;
	pthread_create(&tid, NULL, watchdog, NULL);

	printf("mode=%s device=\"%s\" iterations=%ld\n", argv[1], Pa_GetDeviceInfo(p.device)->name, iterations);
	srandom(1);
	for (long i = 0; i < iterations; ++i) {
		atomic_store(&iteration, i);
		atomic_store(&remaining, 1024 + random() % 2048);
		CHECK(Pa_StartStream(stream));

		/* Wait for the callback to finish the range, then aim Pa_StopStream at
		   the following callback (the one that aborts in the old behaviour),
		   jittered by +-10% of a callback period. Aegisub's playback timer
		   lands in this window by chance. */
		while (atomic_load(&remaining) > 0) usleep(20);
		uint64_t period = atomic_load(&cb_period);
		uint64_t target = atomic_load(&last_cb) + period - period / 10 + (uint64_t)(random() % (long)(period / 5 + 1));
		while (mach_absolute_time() < target) ;

		CHECK(Pa_StopStream(stream));
		if (i % 200 == 0) { printf("iteration %ld\n", i); fflush(stdout); }
	}
	atomic_store(&iteration, iterations);

	CHECK(Pa_CloseStream(stream));
	Pa_Terminate();
	printf("OK: %ld iterations without deadlock\n", iterations);
	return 0;
}