diff --git a/examples/livekit/README.md b/examples/livekit/README.md index 21d2a6ecf..7ed94cbd5 100644 --- a/examples/livekit/README.md +++ b/examples/livekit/README.md @@ -1,101 +1,62 @@ # LiveKit voice tracing and recording -The integration supports LiveKit 1.8.9+; this example pins 1.8.10. Set the usual LiveKit and OpenAI -credentials, plus `BRAINTRUST_API_KEY`. Use Python 3.12–3.14 and install with -`uv sync` from this directory. +Run this example with Python 3.12–3.14. Set `LIVEKIT_URL`, `LIVEKIT_API_KEY`, +`LIVEKIT_API_SECRET`, `OPENAI_API_KEY`, and `BRAINTRUST_API_KEY`, then run from +this directory: ```sh -export BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS=true -export BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS=true +uv sync uv run voice_agent.py dev ``` -Set `LIVEKIT_MODE` to choose the pipeline: +Connect a caller to the same LiveKit room to talk to the agent. + +## Choose a pipeline + +| `LIVEKIT_MODE` | Pipeline | +| --- | --- | +| `cascade` (default) | Speech-to-text → text model → text-to-speech | +| `half-cascade` | Realtime text model → text-to-speech | +| `realtime` | Native speech-to-speech | + +For example: ```sh LIVEKIT_MODE=realtime uv run voice_agent.py dev ``` -| `LIVEKIT_MODE` | Pipeline | Models used by this example | -| ------------------- | --------------------------------- | ------------------------------------------------------------- | -| `cascade` (default) | STT → text LLM → TTS | `gpt-4o-mini-transcribe` → `gpt-4.1-mini` → `gpt-4o-mini-tts` | -| `half-cascade` | Audio → Realtime text model → TTS | `gpt-realtime` with text output → `tts-1` | -| `realtime` | Native speech-to-speech | `gpt-realtime` | +## What to expect + +Traces appear in the `livekit-cascade` Braintrust project, with user and agent +turns, transcripts, model requests, tool calls, and available usage and timing. +Turn timing also shows the wait after the user finishes speaking. + +Audio recording is off by default. To include input and generated audio, set +these flags before starting the worker: -Agent instructions appear in each `llm_response` span's input as a `system` message, -including any instructions specific to that response. +```sh +export BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS=true +export BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS=true +``` -With `braintrust[audio]`, enabled recordings default to Opus in Ogg. Installing -the extra or setting the generic `BRAINTRUST_CAPTURE_ATTACHMENTS` flag does not -enable audio capture. +Transcripts are still logged when audio recording is off. LiveKit already +installs PyAV; recordings use Ogg/Opus when available and raw PCM otherwise. -The example enables both LiveKit and OpenAI instrumentation: +## Customize recordings + +Replace the example's `setup_livekit_agents()` call to change recording options. +For example, use 30-second WAV segments: ```python +from braintrust.audio import AudioEncoding, AudioRecordingOptions from braintrust.integrations.livekit_agents import setup_livekit_agents -from braintrust.integrations.openai import setup_openai - -setup_livekit_agents() -setup_openai() -``` -LiveKit records turns and speech; OpenAI records model requests and usage under -those operations. `auto_instrument()` also enables both integrations. - -Both audio capture flags default to false. OpenAI instrumentation also honors -these flags: `USER` enables transcription/translation input audio, and `AGENT` -enables generated output audio. This is a direction convention; OpenAI does not -identify who is speaking. With capture disabled, transcripts are still logged, -but provider audio attachments are omitted. Configure the flags before starting -the worker; changing them does not retract already queued or uploaded audio. - -LiveKit redaction is also rechecked when queued records are exported. Redacted -exports omit participant/speaker identities and provider request metadata such -as tool descriptions and schemas. Automatic provider spans retain only model -and provider attribution in metadata, alongside usage and timing metrics. - -```text -session -├── user_turn -│ └── stt -│ └── Transcription -└── agent_turn - ├── llm_response (task) - │ └── Chat Completion (llm, token usage) - ├── tool - ├── tts - └── agent_speech +setup_livekit_agents( + audio=AudioRecordingOptions( + segment_duration_seconds=30, + encoding=AudioEncoding(codec="pcm", container="wav"), + ) +) ``` -For adapters without known provider instrumentation, LiveKit emits an LLM child -from the model data exposed by the framework. It does not infer provider support -from whether a span appears. OpenAI Realtime uses a WebSocket API that the OpenAI -integration does not instrument, so LiveKit supplies its `livekit.llm_response` LLM -child. Agent instructions and model usage are available on that child. - - -STT, LLM response, and TTS task wrappers record their own start/end times. -Provider integrations own model metrics. When a client has no known integration, -LiveKit supplies a `livekit.stt`, `livekit.llm_response`, or `livekit.tts` model -child instead. OpenAI HTTP recognition/synthesis defer to OpenAI instrumentation -regardless of whether it is enabled. Native streaming STT and Realtime use -LiveKit's own clients. A native STT stream can span several turns and is traced -as a session-level operation; transcript delivery alone does not create a model -operation. With user audio capture enabled, a native STT model child includes a -WAV of the input consumed by its node, limited to the first 4 MiB of PCM per -stream. `contrib.livekit.input_audio_truncated=true` marks a partial recording; -recognition continues normally after the capture limit. This model input is -separate from the segmented session recording. Duration-only STT usage reports -do not create token counts. TTS model children record the text after LiveKit's -configured text transforms. Media model inputs identify their `operation` as -`transcribe` or `speech`. - - -LiveKit turn diagnostics are included without an OpenTelemetry exporter. User -turns expose `contrib.livekit.end_of_turn_delay`, -`contrib.livekit.transcription_delay`, and -`contrib.livekit.on_user_turn_completed_delay` in seconds when available. -`livekit.eou_wait` shows the observed interval from speech end to an accepted -turn decision. User speech spans retain native `contrib.livekit.vad_silence_duration` -and `contrib.livekit.vad_inference_duration`, also in seconds. Missing native -timing stays absent. Model timing and usage continue to belong to model children. +The capture flags above still control whether audio is recorded. diff --git a/py/pyproject.toml b/py/pyproject.toml index 6ae98f83b..40492b844 100644 --- a/py/pyproject.toml +++ b/py/pyproject.toml @@ -52,7 +52,8 @@ braintrust = "braintrust.wrappers.pytest_plugin.plugin" braintrust = "braintrust.integrations.harbor:HarborPlugin" [project.optional-dependencies] -# Opus/Ogg encoding requires Python 3.12+ (PyAV). +# Install the qualified PyAV on Python 3.12+. Encoding detects PyAV/libopus +# capability at runtime, including PyAV installed by a voice framework. audio = ["av==19.0.2; python_version>='3.12'"] cli = ["boto3", "python-dotenv", "uv>=0.13.4", "starlette", "uvicorn"] # TODO: remove the doc extra in the next major version. diff --git a/py/src/braintrust/_audio/middleware.py b/py/src/braintrust/_audio/middleware.py index 6bd89570e..41f9f27d4 100644 --- a/py/src/braintrust/_audio/middleware.py +++ b/py/src/braintrust/_audio/middleware.py @@ -97,10 +97,13 @@ def encoding_stack(config: AudioEncoding | None, *, channels: int | None = None) return create_stack(selected) except Exception as error: if config is not None: - raise ValueError("Opus/Ogg requires PyAV. Install braintrust[audio].") from error + raise ValueError( + "Opus/Ogg requires PyAV with libopus support. " + "Install braintrust[audio] or provide a compatible PyAV/FFmpeg build." + ) from error if not _warned_missing_codec: _warned_missing_codec = True logging.getLogger("braintrust").warning( - "PyAV is unavailable; recording raw PCM. Install braintrust[audio] for Opus/Ogg." + "PyAV or its libopus encoder is unavailable; recording raw PCM. Install braintrust[audio] for Opus/Ogg." ) return () diff --git a/py/src/braintrust/_audio/test_audio.py b/py/src/braintrust/_audio/test_audio.py index 2ff67c16e..f8a4372ec 100644 --- a/py/src/braintrust/_audio/test_audio.py +++ b/py/src/braintrust/_audio/test_audio.py @@ -409,7 +409,13 @@ def test_tracing_only_session_has_no_media_imports_or_audio_workers(): """ import sys from braintrust.logger import NOOP_SPAN +from braintrust.audio import AudioEncoding, AudioRecordingOptions +from braintrust._audio.config import AudioEncoding as InternalEncoding +from braintrust._audio.config import AudioRecordingOptions as InternalOptions from braintrust._audio.session import AudioSessionRecorder +assert AudioEncoding is InternalEncoding +assert AudioRecordingOptions is InternalOptions +options = AudioRecordingOptions(encoding=AudioEncoding(codec="pcm", container="wav")) from braintrust._audio import worker session = AudioSessionRecorder(NOOP_SPAN) assert session.recorder is None diff --git a/py/src/braintrust/_audio/test_codec.py b/py/src/braintrust/_audio/test_codec.py index 619f5b1a5..6da1af552 100644 --- a/py/src/braintrust/_audio/test_codec.py +++ b/py/src/braintrust/_audio/test_codec.py @@ -119,3 +119,21 @@ def unexpected_omission(meta, reason): assert sum(frame.samples for frame in frames) == segment.meta.sample_count assert frames[0].pts == 0 assert all(frame.sample_rate == 48000 for frame in frames) + + +def test_missing_libopus_warns_and_explicit_opus_fails(monkeypatch, caplog): + from braintrust._audio import middleware + + original = av.codec.Codec + + def unavailable_encoder(*args, **kwargs): + # Exercise the real PyAV missing-codec error without requiring a custom + # FFmpeg build. Encoding setup is purely local, not a provider stand-in. + return original("braintrust_missing_encoder", "w") + + monkeypatch.setattr(av.codec, "Codec", unavailable_encoder) + monkeypatch.setattr(middleware, "_warned_missing_codec", False) + assert encoding_stack(None) == () + assert "libopus" in caplog.text + with pytest.raises(ValueError, match="libopus"): + encoding_stack(AudioEncoding()) diff --git a/py/src/braintrust/audio.py b/py/src/braintrust/audio.py new file mode 100644 index 000000000..f965df029 --- /dev/null +++ b/py/src/braintrust/audio.py @@ -0,0 +1,12 @@ +"""Public audio recording configuration shared by voice integrations. + +These options choose segmentation and encoding. Capture remains opt-in through +an integration's capture options or the dedicated USER/AGENT audio environment +variables. Importing configuration does not load PyAV or a voice framework. +""" + +from ._audio.config import AudioEncoding as AudioEncoding +from ._audio.config import AudioRecordingOptions as AudioRecordingOptions + + +__all__ = ["AudioEncoding", "AudioRecordingOptions"] diff --git a/py/src/braintrust/integrations/livekit_agents/__init__.py b/py/src/braintrust/integrations/livekit_agents/__init__.py index e47209748..58677a89f 100644 --- a/py/src/braintrust/integrations/livekit_agents/__init__.py +++ b/py/src/braintrust/integrations/livekit_agents/__init__.py @@ -1,4 +1,4 @@ -from braintrust._audio.config import AudioRecordingOptions +from braintrust.audio import AudioRecordingOptions from .integration import LiveKitAgentsIntegration from .patchers import wrap_livekit_agents diff --git a/py/src/braintrust/integrations/pipecat/__init__.py b/py/src/braintrust/integrations/pipecat/__init__.py index 6b077daa2..d11c9265a 100644 --- a/py/src/braintrust/integrations/pipecat/__init__.py +++ b/py/src/braintrust/integrations/pipecat/__init__.py @@ -1,6 +1,6 @@ """Braintrust integration for Pipecat AI.""" -from braintrust._audio.config import AudioRecordingOptions +from braintrust.audio import AudioRecordingOptions from braintrust.logger import NOOP_SPAN, current_span, init_logger from .integration import PipecatIntegration @@ -41,8 +41,9 @@ def setup_pipecat( Turn containers follow Pipecat's native boundaries. Agent recordings contain transport-confirmed playback. ``audio_recording_options`` configures segmentation, encoding and shutdown - using ``braintrust._audio.config.AudioRecordingOptions``. Recordings default - to Opus/Ogg with ``braintrust[audio]`` installed, otherwise raw PCM. + using ``braintrust.audio.AudioRecordingOptions``. Recordings default + to Opus/Ogg when PyAV with libopus is available, otherwise raw PCM. + ``braintrust[audio]`` installs a pinned PyAV on Python 3.12+. """ set_default_observer_options( capture_audio_attachments=capture_audio_attachments, diff --git a/py/src/braintrust/type_tests/test_public_exports.py b/py/src/braintrust/type_tests/test_public_exports.py index 0702853f6..33557d459 100644 --- a/py/src/braintrust/type_tests/test_public_exports.py +++ b/py/src/braintrust/type_tests/test_public_exports.py @@ -19,6 +19,7 @@ wrap_openai, wrap_openrouter, ) +from braintrust.audio import AudioEncoding, AudioRecordingOptions _PUBLIC_SYMBOLS = [ @@ -37,3 +38,9 @@ def test_top_level_public_symbol(name: str, imported: object) -> None: assert callable(imported) assert callable(getattr(braintrust, name)) + + +def test_public_audio_configuration_types() -> None: + encoding: AudioEncoding = AudioEncoding(codec="pcm", container="wav") + options: AudioRecordingOptions = AudioRecordingOptions(encoding=encoding) + assert options.encoding is encoding diff --git a/py/tests/wheel/test_wheel.py b/py/tests/wheel/test_wheel.py index dac1685f4..bb51223b2 100644 --- a/py/tests/wheel/test_wheel.py +++ b/py/tests/wheel/test_wheel.py @@ -53,6 +53,14 @@ def test_ships_public_test_helpers(): from braintrust.test_helpers import init_test_logger, with_memory_logger # noqa: F401 +def test_public_audio_configuration_ships_without_optional_codecs(): + from braintrust.audio import AudioEncoding, AudioRecordingOptions + + options = AudioRecordingOptions(encoding=AudioEncoding(codec="pcm", container="wav")) + assert options.encoding.codec == "pcm" + assert "braintrust/audio.py" in WHEEL_FILES + + def test_all_modules_import(): # Optional third-party dependencies are not installed, so ImportErrors for # them are expected. A missing braintrust module means the wheel is broken.