feat(providers): add ProviderCapabilities dataclass with unified capability detection#6233
feat(providers): add ProviderCapabilities dataclass with unified capability detection#6233tcconnally wants to merge 2 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthrough
ChangesProvider Capability Detection
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/llm.py`:
- Line 2396: The early return that bails on missing LiteLLM introspection is
happening before the get_capabilities() method call on line 2396, which prevents
the fallback logic for response_format and reasoning_effort validation from
running. Move the early return statement to after the get_capabilities() call so
that the method's built-in fallback logic can execute before any conditional
exit occurs.
- Around line 2481-2485: The current logic unconditionally sets
supports_tool_calling to True when the provider is not in the
_PROVIDERS_WITHOUT_TOOL_CALLING blocklist, which discards valid False results
from supports_function_calling(). Replace the conditional logic that assigns
True on line 2485 with AND logic that checks both conditions: that the provider
is not in _PROVIDERS_WITHOUT_TOOL_CALLING AND that
self.supports_function_calling() returns True. This ensures the function's
negative results are respected instead of being overridden.
- Around line 2487-2488: The current code at line 2487-2488 uses only a
provider-level allowlist to determine reasoning support, which incorrectly
allows reasoning_effort for non-reasoning models under reasoning-capable
providers. Replace the static allowlist check for `supports_reasoning` with a
model-aware approach using `litellm.supports_reasoning(model,
custom_llm_provider=provider)` when available, and fall back to the static
`_PROVIDERS_WITH_REASONING` allowlist check only when that function is
unavailable or returns None. This ensures that reasoning_effort is only applied
to models that actually support reasoning, not just providers that support
reasoning for some of their models.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d127da02-d2a6-48dc-bd8a-73ba44d43884
📒 Files selected for processing (1)
lib/crewai/src/crewai/llm.py
| # --- reasoning --- | ||
| supports_reasoning = provider in _PROVIDERS_WITH_REASONING |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find the file and verify its existence
find . -type f -name "llm.py" | head -20Repository: crewAIInc/crewAI
Length of output: 91
🏁 Script executed:
# Get line count to ensure lines 2487-2488 exist
wc -l lib/crewai/src/crewai/llm.py 2>/dev/null || echo "File not found in that path"Repository: crewAIInc/crewAI
Length of output: 94
🏁 Script executed:
# Read the context around lines 2487-2488
sed -n '2480,2500p' lib/crewai/src/crewai/llm.pyRepository: crewAIInc/crewAI
Length of output: 896
🏁 Script executed:
# Search for _PROVIDERS_WITH_REASONING definition
rg "_PROVIDERS_WITH_REASONING" --type py -B 5 -A 10Repository: crewAIInc/crewAI
Length of output: 1980
🏁 Script executed:
# Check if litellm module is available and inspect its structure
python3 - <<'PY'
try:
import litellm
import inspect
# Check for supports_reasoning function
if hasattr(litellm, 'utils'):
print("✓ litellm.utils exists")
if hasattr(litellm.utils, 'supports_reasoning'):
print("✓ litellm.utils.supports_reasoning exists")
sig = inspect.signature(litellm.utils.supports_reasoning)
print(f" Signature: {sig}")
else:
print("✗ litellm.utils.supports_reasoning does NOT exist")
print(" Available attributes in litellm.utils:")
attrs = [x for x in dir(litellm.utils) if 'reasoning' in x.lower()]
if attrs:
print(f" Reasoning-related: {attrs}")
else:
print(" No reasoning-related attributes found")
else:
print("✗ litellm.utils does NOT exist")
# Check litellm version and main attributes
print(f"\nlitellm version: {litellm.__version__ if hasattr(litellm, '__version__') else 'unknown'}")
reasoning_attrs = [x for x in dir(litellm) if 'reasoning' in x.lower()]
if reasoning_attrs:
print(f"Reasoning-related in litellm: {reasoning_attrs}")
except ImportError as e:
print(f"Cannot import litellm: {e}")
except Exception as e:
print(f"Error: {e}")
PYRepository: crewAIInc/crewAI
Length of output: 109
🌐 Web query:
litellm supports_reasoning function model capability detection
💡 Result:
LiteLLM provides a built-in utility function, litellm.supports_reasoning(model="<model_name>"), to detect whether a specific model supports reasoning capabilities [1][2]. Key details regarding this functionality include: How it Works: - The function returns True if the model is registered in LiteLLM's internal model registry (model_prices_and_context_window.json) as supporting reasoning, and False otherwise [1][2][3]. - This flag is critical for ensuring that reasoning-specific parameters—such as thinking blocks, reasoning effort (e.g., low, high, x-high), or related arguments—are correctly handled [1][4][5][3]. If this capability is missing or set to false, LiteLLM may raise an UnsupportedParamsError or incorrectly route requests [4][3]. Integration in LiteLLM: - Internal Routing: LiteLLM uses this utility internally to gate reasoning features [6][5]. For example, it helps determine whether to route requests to specialized reasoning APIs or to allow specific parameters like reasoning_effort [6][4][3]. - Data Source: The reasoning capability is defined in LiteLLM's model pricing and context window registry [7][8][5]. Maintenance of these entries is handled via internal model registry updates, ensuring that new models (such as new Claude or Mistral versions) are correctly identified as supporting reasoning [4][5][3]. - Additional Capability Flags: Beyond basic reasoning support, LiteLLM tracks more granular capabilities, such as supports_xhigh_reasoning_effort, supports_none_reasoning_effort, and supports_minimal_reasoning_effort, which can be accessed via litellm.get_model_info [7]. Usage Example: import litellm model_name = "anthropic/claude-3-7-sonnet-20250219" if litellm.supports_reasoning(model=model_name): print(f"{model_name} supports reasoning.") else: print(f"{model_name} does NOT support reasoning.") [1][2]
Citations:
- 1: [Feat] Add litellm.supports_reasoning() util to track if an llm supports reasoning BerriAI/litellm#9923
- 2: https://docs.litellm.ai/docs/reasoning_content
- 3: fix(model-registry): add supports_reasoning to all gpt-oss model entries BerriAI/litellm#25160
- 4: fix: added thinking and reasoning support for all claude 4+ models on bedrock BerriAI/litellm#24053
- 5: feat(mistral): Mistral Small 4 reasoning via supports_reasoning (fixes #24416) BerriAI/litellm#24461
- 6: Add routing based on if reasoning is supported or not BerriAI/litellm#21302
- 7: fix: get_model_info now returns supports_xhigh/none/minimal_reasoning_effort fields BerriAI/litellm#25099
- 8: https://docs.litellm.ai/docs/provider_registration/add_model_pricing
🌐 Web query:
litellm.supports_reasoning function signature custom_llm_provider parameter
💡 Result:
The function signature for litellm.supports_reasoning is: litellm.supports_reasoning(model: str, custom_llm_provider: Optional[str] = None) -> bool [1][2] This utility function determines whether a specific model supports reasoning capabilities (such as thinking blocks or reasoning effort parameters) [1][3]. Parameters: - model (str): The name or identifier of the model (e.g., 'anthropic/claude-3-7-sonnet-20250219') [1][3]. - custom_llm_provider (Optional[str]): An optional parameter used to explicitly specify the LLM provider (e.g., 'mistral', 'bedrock') [2][4][5]. This is particularly useful when the provider cannot be automatically inferred from the model string or when resolving provider-specific capability configurations [2][4][5]. The function returns a boolean value: True if the model supports reasoning, and False otherwise [1]. LiteLLM manages these capability flags within its internal model cost and capability registry (model_prices_and_context_window.json) [2][5].
Citations:
- 1: https://docs.litellm.ai/docs/reasoning_content
- 2: feat(mistral): Mistral Small 4 reasoning via supports_reasoning (fixes #24416) BerriAI/litellm#24461
- 3: [Feat] Add litellm.supports_reasoning() util to track if an llm supports reasoning BerriAI/litellm#9923
- 4: https://github.com/BerriAI/litellm/blob/62920a0c/litellm/litellm_core_utils/get_supported_openai_params.py
- 5: fix: added thinking and reasoning support for all claude 4+ models on bedrock BerriAI/litellm#24053
🏁 Script executed:
# Search for litellm imports and usage in the codebase to understand how it's used
rg "import litellm|from litellm" lib/crewai/src/crewai/llm.py -A 3Repository: crewAIInc/crewAI
Length of output: 970
🏁 Script executed:
# Check the context of self.model and provider to understand what information is available
sed -n '2470,2510p' lib/crewai/src/crewai/llm.pyRepository: crewAIInc/crewAI
Length of output: 1626
Make reasoning detection model-aware.
The current provider-level allowlist allows reasoning_effort through for non-reasoning models under reasoning-capable providers (e.g., OpenAI/Anthropic models without reasoning support), and blocks supported providers not in the set. Use litellm.supports_reasoning(model, custom_llm_provider=provider) when available, falling back to the static allowlist only when introspection is unavailable.
Suggested direction
# --- reasoning ---
- supports_reasoning = provider in _PROVIDERS_WITH_REASONING
+ supports_reasoning = provider in _PROVIDERS_WITH_REASONING
+ if litellm:
+ try:
+ supports_reasoning = litellm.supports_reasoning(
+ model=self.model, custom_llm_provider=provider
+ )
+ except Exception:
+ supports_reasoning = provider in _PROVIDERS_WITH_REASONING🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/llm.py` around lines 2487 - 2488, The current code at
line 2487-2488 uses only a provider-level allowlist to determine reasoning
support, which incorrectly allows reasoning_effort for non-reasoning models
under reasoning-capable providers. Replace the static allowlist check for
`supports_reasoning` with a model-aware approach using
`litellm.supports_reasoning(model, custom_llm_provider=provider)` when
available, and fall back to the static `_PROVIDERS_WITH_REASONING` allowlist
check only when that function is unavailable or returns None. This ensures that
reasoning_effort is only applied to models that actually support reasoning, not
just providers that support reasoning for some of their models.
…bility detection Adds a structured ProviderCapabilities dataclass that aggregates previously scattered capability checks (supports_response_schema, supports_function_calling, supports_stop_words) into a single object. - ProviderCapabilities: supports_response_format, supports_tool_calling, supports_reasoning, supports_streaming, supports_image_input, supports_stop_words - get_capabilities() method on LLM that uses litellm introspection with static-provider-allowlist fallbacks - Updated _validate_call_params() to use capabilities for clearer error messages including reasoning_effort validation - Static allowlists (_PROVIDERS_WITHOUT_RESPONSE_FORMAT, etc.) as fallback when litellm is unavailable Closes crewAIInc#6220
tcconnally
force-pushed
the
feat/provider-capability-detection-6220
branch
from
77cfb72 to
9ed6a84
Compare
1 participant
Summary
Adds a structured
ProviderCapabilitiesdataclass that aggregates previously scattered capability checks into a single object, addressing #6220.Problem
CrewAI currently checks provider capabilities through separate methods (
supports_response_schema(),supports_function_calling(),supports_stop_words()) and hardcoded provider lists. This means:Solution
ProviderCapabilities dataclass
LLM.get_capabilities()
Uses litellm introspection when available, falling back to static provider allowlists:
Updated validation
_validate_call_params()now uses capabilities for bothresponse_formatandreasoning_effortvalidation with clearer error messages:Static fallback allowlists
When litellm is unavailable, three frozensets provide the fallback:
_PROVIDERS_WITHOUT_RESPONSE_FORMAT: deepseek, ollama, ollama_chat, hosted_vllm_PROVIDERS_WITHOUT_TOOL_CALLING: (empty — all modern providers support tools)_PROVIDERS_WITH_REASONING: openai, azure, anthropic, bedrock, cerebrasRelated
Summary by CodeRabbit