Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 41 additions & 80 deletions examples/livekit/README.md
Original file line number Diff line number Diff line change
@@ -1,101 +1,62 @@
# LiveKit voice tracing and recording

The integration supports LiveKit 1.8.9+; this example pins 1.8.10. Set the usual LiveKit and OpenAI
credentials, plus `BRAINTRUST_API_KEY`. Use Python 3.12–3.14 and install with
`uv sync` from this directory.
Run this example with Python 3.12–3.14. Set `LIVEKIT_URL`, `LIVEKIT_API_KEY`,
`LIVEKIT_API_SECRET`, `OPENAI_API_KEY`, and `BRAINTRUST_API_KEY`, then run from
this directory:

```sh
export BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS=true
export BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS=true
uv sync
uv run voice_agent.py dev
```

Set `LIVEKIT_MODE` to choose the pipeline:
Connect a caller to the same LiveKit room to talk to the agent.

## Choose a pipeline

| `LIVEKIT_MODE` | Pipeline |
| --- | --- |
| `cascade` (default) | Speech-to-text → text model → text-to-speech |
| `half-cascade` | Realtime text model → text-to-speech |
| `realtime` | Native speech-to-speech |

For example:

```sh
LIVEKIT_MODE=realtime uv run voice_agent.py dev
```

| `LIVEKIT_MODE` | Pipeline | Models used by this example |
| ------------------- | --------------------------------- | ------------------------------------------------------------- |
| `cascade` (default) | STT → text LLM → TTS | `gpt-4o-mini-transcribe` → `gpt-4.1-mini` → `gpt-4o-mini-tts` |
| `half-cascade` | Audio → Realtime text model → TTS | `gpt-realtime` with text output → `tts-1` |
| `realtime` | Native speech-to-speech | `gpt-realtime` |
## What to expect

Traces appear in the `livekit-cascade` Braintrust project, with user and agent
turns, transcripts, model requests, tool calls, and available usage and timing.
Turn timing also shows the wait after the user finishes speaking.

Audio recording is off by default. To include input and generated audio, set
these flags before starting the worker:

Agent instructions appear in each `llm_response` span's input as a `system` message,
including any instructions specific to that response.
```sh
export BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS=true
export BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS=true
```

With `braintrust[audio]`, enabled recordings default to Opus in Ogg. Installing
the extra or setting the generic `BRAINTRUST_CAPTURE_ATTACHMENTS` flag does not
enable audio capture.
Transcripts are still logged when audio recording is off. LiveKit already
installs PyAV; recordings use Ogg/Opus when available and raw PCM otherwise.

The example enables both LiveKit and OpenAI instrumentation:
## Customize recordings

Replace the example's `setup_livekit_agents()` call to change recording options.
For example, use 30-second WAV segments:

```python
from braintrust.audio import AudioEncoding, AudioRecordingOptions
from braintrust.integrations.livekit_agents import setup_livekit_agents
from braintrust.integrations.openai import setup_openai

setup_livekit_agents()
setup_openai()
```

LiveKit records turns and speech; OpenAI records model requests and usage under
those operations. `auto_instrument()` also enables both integrations.

Both audio capture flags default to false. OpenAI instrumentation also honors
these flags: `USER` enables transcription/translation input audio, and `AGENT`
enables generated output audio. This is a direction convention; OpenAI does not
identify who is speaking. With capture disabled, transcripts are still logged,
but provider audio attachments are omitted. Configure the flags before starting
the worker; changing them does not retract already queued or uploaded audio.

LiveKit redaction is also rechecked when queued records are exported. Redacted
exports omit participant/speaker identities and provider request metadata such
as tool descriptions and schemas. Automatic provider spans retain only model
and provider attribution in metadata, alongside usage and timing metrics.

```text
session
├── user_turn
│ └── stt
│ └── Transcription
└── agent_turn
├── llm_response (task)
│ └── Chat Completion (llm, token usage)
├── tool
├── tts
└── agent_speech
setup_livekit_agents(
audio=AudioRecordingOptions(
segment_duration_seconds=30,
encoding=AudioEncoding(codec="pcm", container="wav"),
)
)
```

For adapters without known provider instrumentation, LiveKit emits an LLM child
from the model data exposed by the framework. It does not infer provider support
from whether a span appears. OpenAI Realtime uses a WebSocket API that the OpenAI
integration does not instrument, so LiveKit supplies its `livekit.llm_response` LLM
child. Agent instructions and model usage are available on that child.


STT, LLM response, and TTS task wrappers record their own start/end times.
Provider integrations own model metrics. When a client has no known integration,
LiveKit supplies a `livekit.stt`, `livekit.llm_response`, or `livekit.tts` model
child instead. OpenAI HTTP recognition/synthesis defer to OpenAI instrumentation
regardless of whether it is enabled. Native streaming STT and Realtime use
LiveKit's own clients. A native STT stream can span several turns and is traced
as a session-level operation; transcript delivery alone does not create a model
operation. With user audio capture enabled, a native STT model child includes a
WAV of the input consumed by its node, limited to the first 4 MiB of PCM per
stream. `contrib.livekit.input_audio_truncated=true` marks a partial recording;
recognition continues normally after the capture limit. This model input is
separate from the segmented session recording. Duration-only STT usage reports
do not create token counts. TTS model children record the text after LiveKit's
configured text transforms. Media model inputs identify their `operation` as
`transcribe` or `speech`.


LiveKit turn diagnostics are included without an OpenTelemetry exporter. User
turns expose `contrib.livekit.end_of_turn_delay`,
`contrib.livekit.transcription_delay`, and
`contrib.livekit.on_user_turn_completed_delay` in seconds when available.
`livekit.eou_wait` shows the observed interval from speech end to an accepted
turn decision. User speech spans retain native `contrib.livekit.vad_silence_duration`
and `contrib.livekit.vad_inference_duration`, also in seconds. Missing native
timing stays absent. Model timing and usage continue to belong to model children.
The capture flags above still control whether audio is recorded.
3 changes: 2 additions & 1 deletion py/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ braintrust = "braintrust.wrappers.pytest_plugin.plugin"
braintrust = "braintrust.integrations.harbor:HarborPlugin"

[project.optional-dependencies]
# Opus/Ogg encoding requires Python 3.12+ (PyAV).
# Install the qualified PyAV on Python 3.12+. Encoding detects PyAV/libopus
# capability at runtime, including PyAV installed by a voice framework.
audio = ["av==19.0.2; python_version>='3.12'"]
cli = ["boto3", "python-dotenv", "uv>=0.13.4", "starlette", "uvicorn"]
# TODO: remove the doc extra in the next major version.
Expand Down
7 changes: 5 additions & 2 deletions py/src/braintrust/_audio/middleware.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,13 @@ def encoding_stack(config: AudioEncoding | None, *, channels: int | None = None)
return create_stack(selected)
except Exception as error:
if config is not None:
raise ValueError("Opus/Ogg requires PyAV. Install braintrust[audio].") from error
raise ValueError(
"Opus/Ogg requires PyAV with libopus support. "
"Install braintrust[audio] or provide a compatible PyAV/FFmpeg build."
) from error
if not _warned_missing_codec:
_warned_missing_codec = True
logging.getLogger("braintrust").warning(
"PyAV is unavailable; recording raw PCM. Install braintrust[audio] for Opus/Ogg."
"PyAV or its libopus encoder is unavailable; recording raw PCM. Install braintrust[audio] for Opus/Ogg."
)
return ()
6 changes: 6 additions & 0 deletions py/src/braintrust/_audio/test_audio.py
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,13 @@ def test_tracing_only_session_has_no_media_imports_or_audio_workers():
"""
import sys
from braintrust.logger import NOOP_SPAN
from braintrust.audio import AudioEncoding, AudioRecordingOptions
from braintrust._audio.config import AudioEncoding as InternalEncoding
from braintrust._audio.config import AudioRecordingOptions as InternalOptions
from braintrust._audio.session import AudioSessionRecorder
assert AudioEncoding is InternalEncoding
assert AudioRecordingOptions is InternalOptions
options = AudioRecordingOptions(encoding=AudioEncoding(codec="pcm", container="wav"))
from braintrust._audio import worker
session = AudioSessionRecorder(NOOP_SPAN)
assert session.recorder is None
Expand Down
18 changes: 18 additions & 0 deletions py/src/braintrust/_audio/test_codec.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,3 +119,21 @@ def unexpected_omission(meta, reason):
assert sum(frame.samples for frame in frames) == segment.meta.sample_count
assert frames[0].pts == 0
assert all(frame.sample_rate == 48000 for frame in frames)


def test_missing_libopus_warns_and_explicit_opus_fails(monkeypatch, caplog):
from braintrust._audio import middleware

original = av.codec.Codec

def unavailable_encoder(*args, **kwargs):
# Exercise the real PyAV missing-codec error without requiring a custom
# FFmpeg build. Encoding setup is purely local, not a provider stand-in.
return original("braintrust_missing_encoder", "w")

monkeypatch.setattr(av.codec, "Codec", unavailable_encoder)
monkeypatch.setattr(middleware, "_warned_missing_codec", False)
assert encoding_stack(None) == ()
assert "libopus" in caplog.text
with pytest.raises(ValueError, match="libopus"):
encoding_stack(AudioEncoding())
12 changes: 12 additions & 0 deletions py/src/braintrust/audio.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
"""Public audio recording configuration shared by voice integrations.

These options choose segmentation and encoding. Capture remains opt-in through
an integration's capture options or the dedicated USER/AGENT audio environment
variables. Importing configuration does not load PyAV or a voice framework.
"""

from ._audio.config import AudioEncoding as AudioEncoding
from ._audio.config import AudioRecordingOptions as AudioRecordingOptions


__all__ = ["AudioEncoding", "AudioRecordingOptions"]
2 changes: 1 addition & 1 deletion py/src/braintrust/integrations/livekit_agents/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from braintrust._audio.config import AudioRecordingOptions
from braintrust.audio import AudioRecordingOptions

from .integration import LiveKitAgentsIntegration
from .patchers import wrap_livekit_agents
Expand Down
7 changes: 4 additions & 3 deletions py/src/braintrust/integrations/pipecat/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Braintrust integration for Pipecat AI."""

from braintrust._audio.config import AudioRecordingOptions
from braintrust.audio import AudioRecordingOptions
from braintrust.logger import NOOP_SPAN, current_span, init_logger

from .integration import PipecatIntegration
Expand Down Expand Up @@ -41,8 +41,9 @@ def setup_pipecat(
Turn containers follow Pipecat's native boundaries. Agent recordings
contain transport-confirmed playback.
``audio_recording_options`` configures segmentation, encoding and shutdown
using ``braintrust._audio.config.AudioRecordingOptions``. Recordings default
to Opus/Ogg with ``braintrust[audio]`` installed, otherwise raw PCM.
using ``braintrust.audio.AudioRecordingOptions``. Recordings default
to Opus/Ogg when PyAV with libopus is available, otherwise raw PCM.
``braintrust[audio]`` installs a pinned PyAV on Python 3.12+.
"""
set_default_observer_options(
capture_audio_attachments=capture_audio_attachments,
Expand Down
7 changes: 7 additions & 0 deletions py/src/braintrust/type_tests/test_public_exports.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
wrap_openai,
wrap_openrouter,
)
from braintrust.audio import AudioEncoding, AudioRecordingOptions


_PUBLIC_SYMBOLS = [
Expand All @@ -37,3 +38,9 @@
def test_top_level_public_symbol(name: str, imported: object) -> None:
assert callable(imported)
assert callable(getattr(braintrust, name))


def test_public_audio_configuration_types() -> None:
encoding: AudioEncoding = AudioEncoding(codec="pcm", container="wav")
options: AudioRecordingOptions = AudioRecordingOptions(encoding=encoding)
assert options.encoding is encoding
8 changes: 8 additions & 0 deletions py/tests/wheel/test_wheel.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,14 @@ def test_ships_public_test_helpers():
from braintrust.test_helpers import init_test_logger, with_memory_logger # noqa: F401


def test_public_audio_configuration_ships_without_optional_codecs():
from braintrust.audio import AudioEncoding, AudioRecordingOptions

options = AudioRecordingOptions(encoding=AudioEncoding(codec="pcm", container="wav"))
assert options.encoding.codec == "pcm"
assert "braintrust/audio.py" in WHEEL_FILES


def test_all_modules_import():
# Optional third-party dependencies are not installed, so ImportErrors for
# them are expected. A missing braintrust module means the wheel is broken.
Expand Down
Loading