Skip to content

feat(pipecat): trace GPT-Live calls and record session audio - #1136

Merged
Abhijeet Prasad (AbhiPrasad) merged 4 commits into
mainfrom
feat/pipecat-gpt-live
Nov 18, 2026
Merged

Abhijeet Prasad (AbhiPrasad) merged 4 commits into
mainfrom
feat/pipecat-gpt-live

Conversation

@delner

Copy link
Copy Markdown
Contributor

Adds GPT-Live support to Braintrust's Pipecat integration. Capture transcripts, user and agent audio, backend model calls, and tool execution in one trace. Usage metrics distinguish the Live model's billed duration from backend token usage, helping explain which models did the work and estimate the call's cost.

What it looks like

session                                      recording: user + agent channels
├─ pipecat.live                         llm   billed seconds, estimated cost
├─ agent_turn                          task
│  └─ transcript_received                    “How can I help?”
├─ agent_speech                        task  playback selection
├─ user_speech                         task  playback selection
├─ user_turn                           task
│  └─ transcript_received                    “Where is order 1042?”
├─ llm_response                        task
│  ├─ pipecat.live.backend_response     llm   token usage, tool request
│  └─ lookup_order                     tool  arguments and result
├─ llm_response                        task
│  └─ pipecat.live.backend_response     llm   token usage, answer
├─ agent_turn                          task
│  └─ transcript_received                    “It arrives Friday.”
└─ agent_speech                        task  playback selection

Usage

With BRAINTRUST_API_KEY set, call setup before creating your Pipecat worker:

from braintrust.integrations.pipecat import setup_pipecat

setup_pipecat(
    project_name="voice-agent",
    capture_audio_attachments=True,
)

# Construct and run your Pipecat pipeline as usual.

Install braintrust[audio] for Opus/Ogg recordings. Recording can also be enabled separately with capture_user_audio_attachments and capture_agent_audio_attachments.

Supports one OpenAILiveLLMService session per pipeline with hosted Responses delegation. Client delegation and session resets are not covered.

Design

  • Turns follow Pipecat's transcript groups. Speech and backend work remain independent where Pipecat supplies no ownership relationship.
  • Transcript timestamps describe receipt. Playback selections come from observed speech; agent audio is captured after transport output.
  • Task spans describe framework work; LLM spans own model usage, avoiding duplicate token counts.
  • Public frames drive transcript grouping and the public sender exposes tool-result submission. One guarded private incoming-event hook captures Live model events.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T16:34:19.767040Z ee6e78d New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

> 🤖 **This text was AI-generated.**

Pipecat connects directly to GPT-Live, so OpenAI SDK instrumentation cannot
capture its model execution. Add transcripts, recorded audio, usage, backend
responses, and tool execution to the Pipecat trace:

```text
session
├─ pipecat.live                         llm
├─ user_turn / agent_turn
│  └─ transcript_received
├─ user_speech / agent_speech           audio selections
└─ llm_response                        task
   ├─ pipecat.live.backend_response    llm
   └─ tool execution
```

Enable tracing and recording with
`setup_pipecat(capture_audio_attachments=True)`.

Framework frames define transcript groups; observed IDs define tool
relationships. Preserve partial output on cancellation and keep model usage
on its owning LLM span. Supports one Live session per pipeline with hosted
Responses delegation.
> 🤖 **This text was AI-generated.**

Live used a separate OpenAI usage parser that dropped reported cache-write
usage:

```text
cache_write_tokens -> omitted
```

Reuse the existing OpenAI parser to preserve the standard metric:

```text
cache_write_tokens -> prompt_cache_creation_tokens
```

Keep Live hook compatibility checks in the patcher applicability gate and
let the shared composite handle patch state and idempotency. Cover either
missing hook through the existing parametrized diagnostic test.

Move the caller-context assertion outside pytest.raises so it runs after
recognition errors and cancellation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9e391dd0e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment on lines +549 to +550
if s.live is not None:
s.live.tool_finished(frame.tool_call_id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear pending submission state when a Live tool is cancelled

When a Live tool is cancelled before submitting its function_call_output, this callback only clears pending_tool_spans through tool_finished(). The call remains in pending_calls, which is cleared exclusively by tool_result_submitted(), so _finish_if_ready() cannot close the response even after backend inference and event processing finish. Consequently, the llm_response span stays open for the remainder of the conversation and is eventually marked incomplete at shutdown, inflating its duration. Handle cancellation separately so it also releases the submission requirement for the cancelled call.

Useful? React with 👍 / 👎.

> 🤖 **This text was AI-generated.**

Several tests drove private trackers and hand-built frame timelines:

```text
test -> tracker methods / observer callbacks
```

Remove those tests and their shared fixture. Keep the cassette-backed real
pipeline coverage and extend the voice session assertion to check playback
text against the synthesized text:

```text
test -> real pipeline -> recorded provider -> exported spans
```

The exact synthetic timing cases are no longer checked independently. Keep
the narrow span-export failure injection in its cassette-backed pipeline
test because provider recordings cannot trigger an instrumentation failure.

Remove the version-specific Live compatibility warning and its diagnostic
cases. Incompatible or absent hooks now skip quietly, while the existing
compatibility gate still requires both hooks.
> 🤖 **This text was AI-generated.**

Live lifecycle rules were spread across independent flags and parallel
maps. Delegation IDs accumulated for the session, and a retained tracker
kept its conversation buffers and routing history after shutdown:

```text
closed / model_closed
llm_finished / processing_finished
```

Represent session and response lifecycles with explicit phases:

```text
session: active -> draining -> closed
response: streaming -> processing -> waiting for tools -> removed
```

Keep each turn with its transcript and each pending tool's two completion
signals in one record. Combine delegation routing into one recent-history
cache using the existing retention limit, keeping active routes pinned.
Clear tracker-owned history and routing caches when pipeline cleanup ends.

Extend the existing recorded pipeline cases to check cleanup while retaining
the tracker. Full conversation transcripts and unfinished work still consume
memory until completion or shutdown; captured content is not truncated.
Abhijeet Prasad (AbhiPrasad) added a commit that referenced this pull request Nov 18, 2026
> 🤖 **This text was AI-generated.**

The follow-up commits on #1136 exposed gaps in our guidance for agents
writing SDK integrations. This PR adds a pre-review check in `AGENTS.md`
and documents the expected practices in the SDK integrations skill.

- Reuse the underlying provider’s usage parser and check the recorded
metrics.
- Gate optional hooks together, skip incompatible hooks quietly, and
rely on the shared patcher for idempotency.
- Run real framework pipelines with recorded provider traffic and assert
on exported spans. Check caller context after exceptions and
cancellation.
- Make session and response phases explicit, bound routing history, and
verify that teardown clears tracker-owned state.

The skill validator, `git diff --check`, and commit hooks passed.
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit 0778de9 into main Nov 18, 2026
98 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the feat/pipecat-gpt-live branch November 18, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants