-
Notifications
You must be signed in to change notification settings - Fork 7.6k
fix(tools): return structured JSON error data from tool exceptions #6313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
AHMEDDEV2004
wants to merge
1
commit into
crewAIInc:main
Choose a base branch
from
AHMEDDEV2004:fix/structured-tool-errors-6262
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+164
−8
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| """Structured tool error formatting for agent consumption. | ||
|
|
||
| When a tool raises an exception, the agent needs structured information | ||
| to decide whether to retry, fix its input, or skip the tool entirely. | ||
| This module provides a consistent error format across all executors. | ||
| """ | ||
|
|
||
| import json | ||
| import traceback | ||
|
|
||
| RETRYABLE_EXCEPTIONS = (TimeoutError, ConnectionError, OSError) | ||
|
|
||
|
|
||
| def format_tool_error(exception: Exception, include_traceback: bool = False) -> str: | ||
| """Format a tool execution error as structured JSON for the agent. | ||
|
|
||
| Returns a string with the "Error executing tool:" prefix (for backward | ||
| compatibility with existing parsing) followed by a JSON object containing | ||
| the exception type, message, and retryability hint. | ||
| """ | ||
| error_data = { | ||
| "error": True, | ||
| "type": type(exception).__name__, | ||
| "message": str(exception), | ||
| "retryable": isinstance(exception, RETRYABLE_EXCEPTIONS), | ||
| } | ||
| if include_traceback: | ||
| error_data["traceback"] = traceback.format_exc(limit=3) | ||
| return f"Error executing tool: {json.dumps(error_data)}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| """Tests for structured tool error formatting.""" | ||
|
|
||
| import json | ||
|
|
||
| import pytest | ||
|
|
||
| from crewai.utilities.tool_errors import RETRYABLE_EXCEPTIONS, format_tool_error | ||
|
|
||
|
|
||
| class TestFormatToolError: | ||
| """Tests for the format_tool_error utility function.""" | ||
|
|
||
| def test_returns_string_with_prefix(self): | ||
| err = ValueError("bad input") | ||
| result = format_tool_error(err) | ||
| assert result.startswith("Error executing tool: ") | ||
|
|
||
| def test_contains_valid_json_after_prefix(self): | ||
| err = ValueError("bad input") | ||
| result = format_tool_error(err) | ||
| json_str = result[len("Error executing tool: "):] | ||
| parsed = json.loads(json_str) | ||
| assert isinstance(parsed, dict) | ||
|
|
||
| def test_error_flag_is_true(self): | ||
| err = RuntimeError("something broke") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["error"] is True | ||
|
|
||
| def test_preserves_exception_type(self): | ||
| err = KeyError("missing_key") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["type"] == "KeyError" | ||
|
|
||
| def test_preserves_exception_message(self): | ||
| err = ValueError("count must be positive") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["message"] == "count must be positive" | ||
|
|
||
| def test_retryable_true_for_timeout(self): | ||
| err = TimeoutError("connection timed out") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is True | ||
|
|
||
| def test_retryable_true_for_connection_error(self): | ||
| err = ConnectionError("refused") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is True | ||
|
|
||
| def test_retryable_true_for_os_error(self): | ||
| err = OSError("disk full") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is True | ||
|
|
||
| def test_retryable_false_for_value_error(self): | ||
| err = ValueError("invalid") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is False | ||
|
|
||
| def test_retryable_false_for_type_error(self): | ||
| err = TypeError("wrong type") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is False | ||
|
|
||
| def test_retryable_false_for_key_error(self): | ||
| err = KeyError("not found") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["retryable"] is False | ||
|
|
||
| def test_no_traceback_by_default(self): | ||
| err = ValueError("test") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert "traceback" not in parsed | ||
|
|
||
| def test_traceback_included_when_requested(self): | ||
| try: | ||
| raise ValueError("deliberate error") | ||
| except ValueError as e: | ||
| result = format_tool_error(e, include_traceback=True) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert "traceback" in parsed | ||
| assert "ValueError" in parsed["traceback"] | ||
|
|
||
| def test_handles_exception_with_special_characters(self): | ||
| err = ValueError('path "C:\\Users\\test" not found') | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert 'C:\\Users\\test' in parsed["message"] | ||
|
|
||
| def test_handles_exception_with_empty_message(self): | ||
| err = RuntimeError() | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["type"] == "RuntimeError" | ||
| assert parsed["message"] == "" | ||
|
|
||
| def test_handles_custom_exception(self): | ||
| class MyToolError(Exception): | ||
| pass | ||
|
|
||
| err = MyToolError("custom failure") | ||
| result = format_tool_error(err) | ||
| parsed = json.loads(result[len("Error executing tool: "):]) | ||
| assert parsed["type"] == "MyToolError" | ||
| assert parsed["message"] == "custom failure" | ||
| assert parsed["retryable"] is False | ||
|
|
||
| def test_retryable_exceptions_tuple_contains_expected_types(self): | ||
| assert TimeoutError in RETRYABLE_EXCEPTIONS | ||
| assert ConnectionError in RETRYABLE_EXCEPTIONS | ||
| assert OSError in RETRYABLE_EXCEPTIONS |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use the passed exception traceback instead of ambient exception state.
On Line 28,
traceback.format_exc()can emitNoneType: Noneif this formatter is called outside an activeexceptblock. Format fromexception.__traceback__directly to keep traceback accurate.Proposed fix
📝 Committable suggestion
🧰 Tools
🪛 ast-grep (0.44.0)
[info] 28-28: use jsonify instead of json.dumps for JSON output
Context: json.dumps(error_data)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🤖 Prompt for AI Agents