Skip to content

feat(pipecat): trace GPT-Live calls and record session audio - #1135

Closed
David Elner (delner) wants to merge 1 commit into
mainfrom
prototype/pipecat-gpt-live
Closed

David Elner (delner) wants to merge 1 commit into
mainfrom
prototype/pipecat-gpt-live

Conversation

@delner

Copy link
Copy Markdown
Contributor

Adds GPT-Live support to Braintrust's Pipecat integration. Capture transcripts, user and agent audio, backend model calls, and tool execution in one trace. Usage metrics distinguish the Live model's billed duration from backend token usage, helping explain which models did the work and estimate the call's cost.

What it looks like

session                                      recording: user + agent channels
├─ pipecat.live                         llm   billed seconds, estimated cost
├─ agent_turn                          task
│  └─ transcript_received                    “How can I help?”
├─ agent_speech                        task  playback selection
├─ user_speech                         task  playback selection
├─ user_turn                           task
│  └─ transcript_received                    “Where is order 1042?”
├─ llm_response                        task
│  ├─ pipecat.live.backend_response     llm   token usage, tool request
│  └─ lookup_order                     tool  arguments and result
├─ llm_response                        task
│  └─ pipecat.live.backend_response     llm   token usage, answer
├─ agent_turn                          task
│  └─ transcript_received                    “It arrives Friday.”
└─ agent_speech                        task  playback selection

Usage

With BRAINTRUST_API_KEY set, call setup before creating your Pipecat worker:

from braintrust.integrations.pipecat import setup_pipecat

setup_pipecat(
    project_name="voice-agent",
    capture_audio_attachments=True,
)

# Construct and run your Pipecat pipeline as usual.

Install braintrust[audio] for Opus/Ogg recordings. Recording can also be enabled separately with capture_user_audio_attachments and capture_agent_audio_attachments.

Supports one OpenAILiveLLMService session per pipeline with hosted Responses delegation. Client delegation and session resets are not covered.

Design

  • Turns follow Pipecat's transcript groups. Speech and backend work remain independent where Pipecat supplies no ownership relationship.
  • Transcript timestamps describe receipt. Playback selections come from observed speech; agent audio is captured after transport output.
  • Task spans describe framework work; LLM spans own model usage, avoiding duplicate token counts.
  • Public frames drive transcript grouping and the public sender exposes tool-result submission. One guarded private incoming-event hook captures Live model events.

> 🤖 **This text was AI-generated.**

Pipecat connects directly to GPT-Live, so OpenAI SDK instrumentation cannot
capture its model execution. Add transcripts, recorded audio, usage, backend
responses, and tool execution to the Pipecat trace:

```text
session
├─ pipecat.live                         llm
├─ user_turn / agent_turn
│  └─ transcript_received
├─ user_speech / agent_speech           audio selections
└─ llm_response                        task
   ├─ pipecat.live.backend_response    llm
   └─ tool execution
```

Enable tracing and recording with
`setup_pipecat(capture_audio_attachments=True)`.

Framework frames define transcript groups; observed IDs define tool
relationships. Preserve partial output on cancellation and keep model usage
on its owning LLM span. Supports one Live session per pipeline with hosted
Responses delegation.
@delner

Copy link
Copy Markdown
Contributor Author

Replaced by #1136 after renaming the branch to feat/pipecat-gpt-live.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant