Report native Langfuse generation metrics - #220
Conversation
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (2)
Comment |
Tiny Sweeper reviewTiny Sweeper reviewed this change across 6 lane(s) and found 2 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Ready for maintainer review Review snapshot
Completeness: Complete What changedThe review could not produce a supported behavioral summary; inspect the cited changed surface and lane details below. FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred. Findings
Resolved this pass
Before mergeNone. How this fits togetherflowchart LR
n0["collect_call_models<br/>changed<br/>7 findings"]:::flagged
n1["observation_event<br/>changed<br/>7 findings"]:::flagged
n2["generation_carries_model_from_model_started<br/>changed"]:::changed
n3["proxy"]:::impacted
n4["find"]:::impacted
n5["obs"]:::impacted
n6["build_ingestion_batch"]:::impacted
n7["new"]:::impacted
n0 -->|calls| n7
n2 -->|calls| n3
n2 -->|tests| n3
n2 -->|calls| n4
n2 -->|tests| n4
n2 -->|calls| n5
n2 -->|tests| n5
n3 -->|calls| n7
n6 -->|calls| n0
n6 -->|calls| n1
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Agent review detailscritique
security
tests
commits
description
e2e
Evidence and run details
|
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking. Approving.
$0.0106 · 193,779 in / 22,145 out · 27,430 cached (14%) · ladder/vectors, gpt-5.6-luna, deepseek-v4-flash · 492 embedded
critique: $0.0071 · 78,220 in / 4,104 out · 3,320 cached (4%) · gpt-5.6-luna, deepseek-v4-flash
security: $0.0023 · 75,708 in / 3,974 out · 21,038 cached (28%) · gpt-5.6-luna
tests: $0.0007 · 29,987 in / 5,287 out · 2,048 cached (7%) · deepseek-v4-flash
description: $0.0003 · 5,495 in / 5,404 out · 1,024 cached (19%) · deepseek-v4-flash
|
|
||
| /// First streamed output for each model call. This is the observable TTFT; | ||
| /// non-streaming calls have no first-delta timestamp to report. | ||
| fn collect_first_deltas(observations: &[AgentObservation]) -> BTreeMap<&str, u64> { |
There was a problem hiding this comment.
Scope first-delta timestamps to each call instance
collect_first_deltas keys timestamps only by call_id, even though call IDs are reused across runs/turns. If two observations contain ModelDelta { call_id: "model-1", ... } in different runs, the first run's timestamp is reused for the second run's generation whenever the second call has no earlier matching delta (and the map's entry retains the earliest timestamp). This can produce a completionStartTime outside the generation's own start/end window. Include the run identity (or another invocation identity) in the key and use the same scope when looking it up.
Additional security observation
Correlate first deltas with the specific model invocation
[RULE] invocation-scoped-correlation
collect_first_deltas keys timestamps only by call_id, which is not globally unique and can be reused across turns or runs. When a batch contains multiple invocations with the same call ID, a delta from one invocation can populate completionStartTime for another generation, producing incorrect TTFT telemetry. Key the map by the same invocation scope used to identify the corresponding ModelStarted/ModelCompleted event (at minimum the run/trace identity plus call ID), and use that scoped key when reading it in observation_event.
[RULE] unscoped-call-correlation ·
| if let AgentEvent::ModelDelta { call_id, delta, .. } = &obs.event | ||
| && (!delta.text.is_empty() || !delta.reasoning.is_empty() || delta.tool_call.is_some()) | ||
| { | ||
| first.entry(call_id.as_str()).or_insert(obs.ts_ms); |
There was a problem hiding this comment.
Scope first-delta timestamps to each call instance
The map is collected across the entire observation slice and keyed only by call_id. Call IDs are reused across turns, so a later generation can inherit an earlier invocation's first-delta timestamp, potentially reporting a completion start before its own start or after its completion. Track first deltas by the concrete call instance, using the invocation's run/turn scope together with the call ID, and use that scoped key when emitting each generation.
[RULE] invocation-scoping ·
| if let AgentEvent::ModelDelta { call_id, delta, .. } = &obs.event | ||
| && (!delta.text.is_empty() || !delta.reasoning.is_empty() || delta.tool_call.is_some()) | ||
| { | ||
| first.entry(call_id.as_str()).or_insert(obs.ts_ms); |
There was a problem hiding this comment.
Correlate first deltas with the specific model invocation
A call_id alone does not uniquely identify a model invocation. When a batch contains multiple invocations with the same call ID, a delta from one invocation can populate completionStartTime for another generation, producing incorrect TTFT telemetry. Key the map by the same invocation scope used to identify the corresponding ModelStarted/ModelCompleted event, at minimum the run or trace identity plus call ID, and use that scoped key during lookup.
[RULE] invocation-correlation ·
| // existed. | ||
| "startTime": started_at_ms.map(iso_ms).unwrap_or_else(|| timestamp.clone()), | ||
| "endTime": timestamp, | ||
| "completionStartTime": first_deltas.get(call_id.as_str()).map(|ms| iso_ms(*ms)), |
There was a problem hiding this comment.
Correlate first deltas with the specific model invocation
This lookup uses only call_id, which does not uniquely identify a model invocation in a batch containing multiple runs or reused call IDs. A delta from one invocation can populate completionStartTime for another generation, producing incorrect TTFT telemetry. Look up the first delta using the same invocation scope used to identify the corresponding ModelStarted and ModelCompleted events.
Additional critique observation
Correlate first deltas with the specific model invocation
[RULE] incorrect-call-correlation
The lookup associates a first delta with a generation using only call_id. A call ID can be reused for sequential model invocations within the same run (for example, a retry or repeated turn), so the first invocation's delta timestamp can be attached to the later invocation's ModelCompleted event. The resulting TTFT is incorrect and may precede that generation's startTime. Track deltas per invocation, using run/call identity plus an occurrence or lifecycle pairing, rather than a bare call ID.
[RULE] incorrect-correlation-key ·
| // `body["model"]`; without it Langfuse can't map pricing and every | ||
| // generation's cost is $0. | ||
| let call_models = collect_call_models(observations); | ||
| let first_deltas = collect_first_deltas(observations); |
There was a problem hiding this comment.
Scope first-delta timestamps to each call instance
The first-delta map is collected once for the entire observation slice and is keyed only by call_id. Call IDs are reused across turns, so a later generation can inherit an earlier invocation's first-delta timestamp (and potentially a timestamp after its own completion). Track first deltas per concrete call instance, using the invocation's run/turn scope together with the call ID, and look up that scoped key when emitting each generation.
[RULE] invocation-scoped-correlation ·
tinysweeper
Bot
added
the
priority: p2
1 participant
Summary
Report Langfuse's native generation fields from the durable agent journal: model, first streamed output time, token details, and provider charge. The generation's existing start/end timestamps continue to supply latency.
API Or Behavior Changes
generation-createnow carriescompletionStartTimewhen the run streamed output,usageDetailsincluding cache and reasoning tokens, andcostDetailswhen the provider supplied a charge.completionStartTimeunset; calls without a provider charge leavecostDetailsunset.Tests
cargo fmt --allcargo test -p tinyagents-harness observability::langfuse::test --lib(12 passed)Documentation
The exporter payload contract is covered by the new focused test; no user-facing documentation changed.
Summary by CodeRabbit