Conversation
… for Bedrock throttling
Implement two-level retry mechanism to handle AWS Bedrock throttling exceptions:
1. Application-level retry in provider.py:
- Detects retryable errors (throttling, rate limits, HTTP 429)
- Retries with exponential backoff + jitter (base 2^attempt, max 32s)
- Applied to both sync (chat) and async (achat) methods
- Fixes broken retry loop that affected all providers
2. Bedrock-specific AWS-side retry config:
- Added Config(retries={'mode': 'adaptive', 'max_attempts': 6})
- Provides client-side rate limiting complementing application retries
Fixes the Bedrock ConverseAPI ThrottlingException that occurred specifically
on sync stream calls due to rapid back-to-back requests hitting AWS quotas.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes Issue #206 where Bedrock's ConverseAPI with Claude 3.5 Sonnet was throwing throttling exceptions on sync stream calls. The fix adds retry logic with exponential backoff at both the application and AWS client levels.
The Problem
Users reported getting
ThrottlingException: Too many requests, please wait before trying againwhen making sync stream calls to Bedrock, but the same operations worked fine for async stream, sync non-stream, and async non-stream modes. This was frustrating because it seemed like only sync stream was broken.What's Actually Happening
After investigating, I found two issues:
First, the provider's retry loop was basically broken. The code had a retry mechanism in place but it was never actually retrying - it would just immediately raise an error. There was even a commented-out section suggesting someone had tried to implement retry logic for rate limits but never finished it.
Second, the Bedrock client wasn't configured for retries at all. It was using boto3's default settings, which aren't tuned for Bedrock's fairly strict per-model rate limits. So when the test harness makes four consecutive calls to the same model (async non-stream, async stream, sync non-stream, sync stream), the fourth call hits the rate limit and fails.
The Fix
I made two changes that work together:
Fixed the retry loop in provider.py - Now when a throttling or rate limit error happens, it actually retries. It uses exponential backoff so it waits longer between each retry. This affects all providers, so OpenAI, Anthropic, Vertex, and others all benefit.
Added retry config to the Bedrock client - The boto3 client now uses adaptive retry mode with up to 6 attempts. This handles AWS-side throttling gracefully.
Together, these changes make the system resilient to temporary rate limits instead of just giving up immediately.
Testing
To verify this works, run the bedrock example with proper AWS credentials. The sync stream call should no longer fail with throttling exceptions.