fix: service tier in openai chat completion (#5073)#5089
Merged
Conversation
Fixes a cost calculation bug where Anthropic prompt-cache tokens (both cache read and cache creation) were billed at standard rates even when the request used fast mode. Per Anthropic's pricing model, cache multipliers stack on top of the fast base input rate, meaning cache token costs should scale by the fast/standard input ratio rather than remaining at the standard cache rate. - Introduced `fastCacheRate`, a helper that scales a standard cache rate to the fast-mode equivalent by multiplying it by the fast/standard input price ratio. - Applied `fastCacheRate` as an early-return guard in `tieredCacheReadInputTokenRate`, `tieredCacheCreationInputTokenRate`, and `tieredCacheCreationInputAbove1hrTokenRate` so fast-mode requests use the correctly scaled cache rates. - Updated the existing fast-mode cache test (`TestComputeTextCost_FastMode_CacheStacksOnFastBase`) to reflect the correct expected values (cache tokens now scale 2× when the fast input rate is 2× the standard rate). - Added a regression test (`TestComputeTextCost_FastMode_Opus48CacheRegression`) pinning the real-world miscalculation observed with Anthropic Opus 4.8 fast mode + `cache_control`, where 44,667 5-minute cache-creation tokens were billed at the standard $6.25/MTok instead of the fast-scaled $12.50/MTok. - [x] Bug fix - [x] Core (Go) ```sh go test ./framework/modelcatalog/datasheet/... ``` The new regression test `TestComputeTextCost_FastMode_Opus48CacheRegression` will fail on the unfixed code and pass after this change. The updated `TestComputeTextCost_FastMode_CacheStacksOnFastBase` validates the general scaling behaviour. - [x] No None. - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
8 tasks
|
|
Collaborator
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Jul 10, 2026
Contributor
Confidence Score: 4/5This is close, but the streaming metadata fix should be completed before merging.
core/providers/openai/openai.go Important Files Changed
Reviews (2): Last reviewed commit: "fix: service tier in openai chat complet..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
## Summary Adds support for Anthropic's data-residency billing feature (`inference_geo`). When Anthropic serves inference in the US (`inference_geo: "us"`), a 1.1x multiplier is applied to all token and cache costs. This PR propagates the `inference_geo` field through the full response pipeline (chat, responses, streaming, passthrough) and wires it into the cost computation layer, including a new database column and UI override field. A secondary fix ensures Anthropic server-tool web search request counts (`ServerToolUse.WebSearchRequests`) are correctly forwarded and billed in both streaming and non-streaming paths, where previously the count could be lost when the terminal chunk overwrote the accumulated usage. ## Changes - **`inference_geo` propagation**: `InferenceGeo *string` added to `BifrostChatResponse`, `BifrostResponsesResponse`, and `BifrostPassthroughUsage`. The field is forwarded from `AnthropicUsage` in all conversion paths: non-streaming chat, non-streaming responses, streaming chat (captured across events and set on the final chunk), streaming responses (`message_delta`), and passthrough. - **Data-residency multiplier in cost computation**: `serviceTier` gains an `inferenceGeoUS bool` flag. `tierFromResponse` now accepts and evaluates `inferenceGeo`. `computeTextCost` applies `InferenceGeoUSMultiplier` to token/cache costs only — the flat per-search fee is intentionally excluded. - **Database migration**: `migrationAddInferenceGeoMultiplierColumn` adds `inference_geo_us_multiplier` to `TableModelPricing`. The column is included in the pricing sync update list and mapped through `convertEntryToTablePricing` / `convertTablePricingToEntry`. - **Web search billing fix**: `accumulateAnthropicResponsesUsage` now carries `ServerToolUse.WebSearchRequests` into the accumulator so the count survives the terminal-chunk overwrite on streamed Responses requests. The non-streaming chat converter also forwards the count into `CompletionTokensDetails.NumSearchQueries`. - **UI**: `inference_geo_us_multiplier` added to the custom pricing override sheet and `PricingOverridePatch` TypeScript type. - **Tests**: New unit tests cover the accumulator web-search fix, chat/responses converter forwarding of `InferenceGeo` and web search counts, multiplier application (including the no-multiplier-column safe no-op), and `tierFromResponse` geo detection (case-insensitive). ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go test ./core/providers/anthropic/... ./framework/modelcatalog/datasheet/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: send a request to an Anthropic model with data residency enabled. The response should include `inference_geo: "us"` and the computed cost should reflect the 1.1x multiplier on token/cache costs, with the per-search fee unchanged. ## Breaking changes - [ ] Yes - [x] No ## Security considerations No auth, secrets, or PII implications. The `inference_geo` value is sourced from Anthropic's API response and used only for billing computation. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary During OpenAI chat completion streaming, the `service_tier` field returned in stream chunks was being discarded, causing the final assembled response to lose priority/flex billing tier information. This PR captures `service_tier` as it appears in stream chunks and propagates it to the final response. ## Changes - Introduced a `serviceTier` variable to track the `service_tier` value echoed across streaming chunks. - After the stream completes and the final response is assembled, the captured `service_tier` is applied to the response so priority/flex billing metadata is preserved. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [ ] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test Send a streaming chat completion request to an OpenAI endpoint using a `service_tier` (e.g., `"flex"` or `"priority"`). Verify that the final streamed response includes the correct `service_tier` value. ```sh go test ./... ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations No security implications. This change only propagates a billing tier metadata field from stream chunks to the final response object. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [ ] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
TejasGhatte
force-pushed
the
07-10-cp-service-tier-chat
branch
from
5faa007 to
c898ed9
Compare
TejasGhatte
force-pushed
the
07-10-cp-inference-geo-cost
branch
from
b900b3a to
19fb627
Compare
Contributor
📝 WalkthroughWalkthroughThe OpenAI streaming handler now captures ChangesOpenAI streaming metadata propagation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
core/providers/openai/openai.go (1)
1177-1178: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a streaming regression test for terminal
service_tier. The existing passthrough usage test only covers a direct response body; add a case for the streamed path whereservice_tierappears on an intermediate or usage-only chunk and the final chunk preserves it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/providers/openai/openai.go` around lines 1177 - 1178, The streaming passthrough tests need coverage for terminal service_tier propagation. Add a regression case alongside the existing passthrough usage test that emits service_tier on an intermediate or usage-only chunk, then verifies the final streamed chunk retains the same value through the streaming response path.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@core/providers/openai/openai.go`:
- Around line 1177-1178: The streaming passthrough tests need coverage for
terminal service_tier propagation. Add a regression case alongside the existing
passthrough usage test that emits service_tier on an intermediate or usage-only
chunk, then verifies the final streamed chunk retains the same value through the
streaming response path.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e0d8494c-73b6-4104-82ac-6f71415e1aac
📒 Files selected for processing (1)
core/providers/openai/openai.go
Contributor
Merge activity
|
akshaydeo
changed the base branch from
07-10-cp-inference-geo-cost
to
graphite-base/5089
akshaydeo
dismissed
coderabbitai[bot]’s stale review
The base branch was changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Briefly explain the purpose of this PR and the problem it solves.
Changes
Type of change
Affected areas
How to test
Describe the steps to validate this change. Include commands and expected outcomes.
If adding new configs or environment variables, document them here.
Screenshots/Recordings
If UI changes, add before/after screenshots or short clips.
Breaking changes
If yes, describe impact and migration instructions.
Related issues
Link related issues and discussions. Example: Closes #123
Security considerations
Note any security implications (auth, secrets, PII, sandboxing, etc.).
Checklist
docs/contributing/README.mdand followed the guidelines