feat(tools): add TwelveLabs video analysis tool#6336
Conversation
📝 WalkthroughWalkthroughAdds a new TwelveLabsAnalyzeTool implementation with package exports and dependency metadata, plus docs, README entries, and tests covering credential handling and analysis requests. ChangesTwelveLabs video analysis
Sequence Diagram(s)sequenceDiagram
participant CrewAIAgent as CrewAI agent
participant TwelveLabsAnalyzeTool
participant TWELVELABS_API_KEY as TWELVELABS_API_KEY env var
participant TwelveLabsSDKClient as TwelveLabs SDK client
participant TwelveLabsAPI as TwelveLabs API
CrewAIAgent->>TwelveLabsAnalyzeTool: __init__(api_key or env lookup)
TwelveLabsAnalyzeTool->>TWELVELABS_API_KEY: read api_key when needed
TWELVELABS_API_KEY-->>TwelveLabsAnalyzeTool: api_key
TwelveLabsAnalyzeTool->>TwelveLabsSDKClient: create client
CrewAIAgent->>TwelveLabsAnalyzeTool: run(prompt, video_url or video_id)
TwelveLabsAnalyzeTool->>TwelveLabsSDKClient: analyze(model_name, prompt, max_tokens, temperature, video)
TwelveLabsSDKClient->>TwelveLabsAPI: send analyze request
TwelveLabsAPI-->>TwelveLabsSDKClient: response.data
TwelveLabsSDKClient-->>TwelveLabsAnalyzeTool: response
TwelveLabsAnalyzeTool-->>CrewAIAgent: response.data or ""
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![corridor-security[bot]](https://avatars.githubusercontent.com/in/1177339?s=60&v=4){kind=small}
There was a problem hiding this comment.
Security Issues
- Server-Side Request Forgery
The newTwelveLabsAnalyzeToolaccepts a caller-suppliedvideo_urland forwards it to the TwelveLabs SDK withoutvalidate_url, allowing restricted schemes or hosts such as loopback, link-local, metadata, or private-network addresses to be submitted through the video analysis integration.
Summary: This PR adds a new TwelveLabs video-analysis tool that sends user-provided prompts and either video IDs or video URLs to an external SDK/API.
Risk: Medium risk. The new tool introduces an external URL-processing attack surface, and the added URL path lacks the repository-required SSRF validation before handing caller-controlled URLs to the integration.
Recommendations:
- Validate
video_urlwithcrewai_tools.security.safe_path.validate_urlbefore constructingVideoContext_Url, and use only the validated URL for the SDK call.
| if video_id: | ||
| analyze_kwargs["video_id"] = video_id | ||
| else: | ||
| from twelvelabs.types.video_context import VideoContext_Url |
There was a problem hiding this comment.
The video_url parameter is caller-controlled and is passed into the TwelveLabs URL context without the repository's SSRF guardrail:
from twelvelabs.types.video_context import VideoContext_Url
analyze_kwargs["video"] = VideoContext_Url(url=video_url)An attacker who can invoke this tool can supply loopback, link-local/cloud-metadata, private-network, or non-HTTP(S) URLs and cause the video analysis integration to process restricted locations.
Remediation: Import validate_url from crewai_tools.security.safe_path, call it before constructing VideoContext_Url, and pass only the validated URL to the SDK.
For more details, see the finding in Corridor.
Provide feedback: Reply with whether this is a valid vulnerability or false positive to help improve Corridor's accuracy.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
lib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.py (1)
64-66: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueOptional: validate
max_tokens >= 512.The docstring states Pegasus requires at least 512 tokens, but a caller can override
max_tokensto a smaller value, which would surface as an opaque API error inside_run. Consider validating early for a clearer message. Low priority since the default (2048) is already valid.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.py` around lines 64 - 66, Add early validation in TwelveLabsAnalyzeTool so overridden max_tokens values below 512 are rejected before _run makes the API call. Update the class field handling around model_name, max_tokens, and temperature to enforce the Pegasus minimum with a clear error message, keeping the existing default of 2048 unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.py`:
- Around line 130-142: The TwelveLabs import in the fallback path of
TwelvelabsAnalyzeTool.analyze uses an internal module path, so update the
VideoContext_Url import to the public stable SDK API exposed by
twelvelabs.types. Keep the existing analyze_kwargs logic in
TwelvelabsAnalyzeTool.analyze, but change the import used when video_id is
absent so the `video` keyword is still passed a VideoContext_Url instance while
relying on the supported import surface.
In `@lib/crewai-tools/tests/tools/twelvelabs_analyze_tool_test.py`:
- Around line 11-15: The test setup in _make_tool and the related twelvelabs
imports assume the optional SDK package exists, but patching
twelvelabs.TwelveLabs with create=True will not create a missing module. Update
the test fixture to inject fake twelvelabs package modules into sys.modules
before importing or patching TwelveLabsAnalyzeTool, and also stub
twelvelabs.types.video_context.VideoContext_Url so the VideoContext_Url-based
test can run without the real dependency. Use the existing _make_tool helper and
the test that references VideoContext_Url as the main locations to adjust.
---
Nitpick comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.py`:
- Around line 64-66: Add early validation in TwelveLabsAnalyzeTool so overridden
max_tokens values below 512 are rejected before _run makes the API call. Update
the class field handling around model_name, max_tokens, and temperature to
enforce the Pegasus minimum with a clear error message, keeping the existing
default of 2048 unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3f4b8e43-f830-439c-8e34-d2ad643a2274
📒 Files selected for processing (11)
docs/docs.jsondocs/edge/en/tools/ai-ml/twelvelabsanalyzetool.mdxlib/crewai-tools/README.mdlib/crewai-tools/pyproject.tomllib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.pylib/crewai-tools/tests/tools/twelvelabs_analyze_tool_test.pylib/crewai-tools/tool.specs.json
| if video_id: | ||
| analyze_kwargs["video_id"] = video_id | ||
| else: | ||
| from twelvelabs.types.video_context import VideoContext_Url | ||
|
|
||
| analyze_kwargs["video"] = VideoContext_Url(url=video_url) | ||
|
|
||
| try: | ||
| response = self._client.analyze(**analyze_kwargs) | ||
| except Exception as e: | ||
| raise RuntimeError(f"TwelveLabs analysis failed: {e!s}") from e | ||
|
|
||
| return response.data or "" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor
🧩 Analysis chain
🌐 Web query:
TwelveLabs Python SDK 1.2.8 client analyze method and VideoContext_Url import path twelvelabs.types.video_context
💡 Result:
In the TwelveLabs Python SDK, the analyze method is used for synchronous video analysis [1][2]. The VideoContext_Url class, which allows you to pass a video URL directly into the analysis request, is imported from twelvelabs.types [3][4]. To use these in your code, you can follow this pattern: from twelvelabs import TwelveLabs from twelvelabs.types import VideoContext_Url client = TwelveLabs(api_key="YOUR_API_KEY") # Analyze a video using a URL result = client.analyze( model_name="pegasus1.5", video=VideoContext_Url(url="YOUR_VIDEO_URL"), prompt="Describe this video.") print(result.data) Key details: 1. Method Usage: The analyze method is a synchronous call that returns results immediately [2][5]. It accepts a video parameter which should be an instance of a VideoContext type, such as VideoContext_Url (for URL-based input), VideoContext_AssetId (for previously indexed assets), or VideoContext_Base64String [3][6]. 2. Import Path: The VideoContext variants, including VideoContext_Url, are located within the twelvelabs.types module [3][7][4]. 3. Migration Note: While older versions of the SDK (e.g., v1.2.x) may have used video_id directly in the analyze method, newer implementations for the latest API versions utilize the video parameter with VideoContext objects to provide more flexibility (URL, Asset ID, etc.) [8]. Always ensure you are using the latest SDK version to maintain compatibility with the TwelveLabs API [9].
Citations:
- 1: https://docs.twelvelabs.io/sdk-reference/python/analyze-videos
- 2: https://beta.docs.twelvelabs.io/sdk-reference/python/analyze-videos/sync-analysis.mdx
- 3: https://docs.twelvelabs.io/docs/guides/analyze-videos.mdx
- 4: https://docs.twelvelabs.io/docs/get-started/quickstart/analyze-videos.mdx
- 5: https://beta.docs.twelvelabs.io/v1.3/sdk-reference/python/analyze-videos/sync-analysis
- 6: https://beta.docs.twelvelabs.io/sdk-reference/node-js/analyze-videos/sync-analysis.mdx
- 7: https://beta.docs.twelvelabs.io/docs/guides/analyze-videos/structured-responses.md
- 8: https://github.com/twelvelabs-io/twelvelabs-python/blob/main/reference.md
- 9: https://docs.twelvelabs.io/sdk-reference/python
Verify TwelveLabs SDK v1.2.8 import path and API surface.
The code uses from twelvelabs.types.video_context import VideoContext_Url, but the public TwelveLabs Python SDK (v1.2.8 and later) exposes VideoContext_Url under twelvelabs.types. The direct import path twelvelabs.types.video_context is an internal implementation detail and may break with SDK updates or fail if the internal module structure changes.
Update the import to use the public stable API:
from twelvelabs.types import VideoContext_UrlEnsure the analyze method expects the video keyword argument with the VideoContext_Url object, which is consistent with the current usage in the code.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@lib/crewai-tools/src/crewai_tools/tools/twelvelabs_analyze_tool/twelvelabs_analyze_tool.py`
around lines 130 - 142, The TwelveLabs import in the fallback path of
TwelvelabsAnalyzeTool.analyze uses an internal module path, so update the
VideoContext_Url import to the public stable SDK API exposed by
twelvelabs.types. Keep the existing analyze_kwargs logic in
TwelvelabsAnalyzeTool.analyze, but change the import used when video_id is
absent so the `video` keyword is still passed a VideoContext_Url instance while
relying on the supported import surface.
| def _make_tool() -> TwelveLabsAnalyzeTool: | ||
| """Build a tool with the SDK client stubbed so no network/SDK is needed.""" | ||
| with patch( | ||
| "twelvelabs.TwelveLabs", create=True, return_value=MagicMock() | ||
| ): |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major
🧩 Analysis chain
🌐 Web query:
In Python unittest.mock, does patch(target, create=True) create a missing module, or does the target module still need to be importable?
💡 Result:
In Python's unittest.mock, the patch(target, create=True) function does not create missing modules [1][2]. The target module must still be importable and exist for patch to successfully locate the namespace where the attribute is being replaced [1][3]. The create=True argument specifically allows you to patch attributes (such as functions, classes, or variables) that do not currently exist within an existing module or class [1][4]. When create=True is used, patch will create the missing attribute on the target object when the patched function is called and delete it again after the function exits [1][2]. If the module itself cannot be imported or found, the patch will fail with an ImportError or AttributeError before the attribute can be created [1].
Citations:
- 1: https://docs.python.org/3/library/unittest.mock.html
- 2: https://github.com/python/cpython/blob/main/Doc/library/unittest.mock.rst
- 3: https://docs.python.org/3/library/unittest.mock-examples.html
- 4: https://docs.python.org/release/3.10.19/library/unittest.mock.html
Patch strategy requires the optional SDK module to exist
The patch("twelvelabs.TwelveLabs", create=True) approach at Line 13 and Line 20 is insufficient. In Python unittest.mock, create=True only allows patching non-existent attributes on an existing object; it does not create missing modules. If the twelvelabs package is not installed, the patch target cannot be resolved during the setup phase, causing the test to crash with an AttributeError before the mock is applied.
Additionally, the test at Line 54 relies on twelvelabs.types.video_context.VideoContext_Url, which must also be stubbed to avoid import errors when the optional dependency is absent.
To ensure tests run independently of the twelvelabs package, inject fake modules directly into sys.modules:
Recommended fix
+import sys
+import types
from unittest.mock import MagicMock, patch
def _make_tool() -> TwelveLabsAnalyzeTool:
"""Build a tool with the SDK client stubbed so no network/SDK is needed."""
- with patch(
- "twelvelabs.TwelveLabs", create=True, return_value=MagicMock()
- ):
+ # Inject fake modules so imports inside the tool resolve without the actual SDK
+ fake_twelvelabs = types.ModuleType("twelvelabs")
+ fake_types = types.ModuleType("twelvelabs.types")
+ fake_video_context = types.ModuleType("twelvelabs.types.video_context")
+
+ fake_client = MagicMock()
+ fake_twelvelabs.TwelveLabs = MagicMock(return_value=fake_client)
+ fake_video_context.VideoContext_Url = MagicMock()
+
+ with patch.dict(
+ sys.modules,
+ {
+ "twelvelabs": fake_twelvelabs,
+ "twelvelabs.types": fake_types,
+ "twelvelabs.types.video_context": fake_video_context,
+ },
+ ):
return TwelveLabsAnalyzeTool(api_key="test-key")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/tests/tools/twelvelabs_analyze_tool_test.py` around lines 11
- 15, The test setup in _make_tool and the related twelvelabs imports assume the
optional SDK package exists, but patching twelvelabs.TwelveLabs with create=True
will not create a missing module. Update the test fixture to inject fake
twelvelabs package modules into sys.modules before importing or patching
TwelveLabsAnalyzeTool, and also stub
twelvelabs.types.video_context.VideoContext_Url so the VideoContext_Url-based
test can run without the real dependency. Use the existing _make_tool helper and
the test that references VideoContext_Url as the main locations to adjust.
1 participant
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
A new opt-in
TwelveLabsAnalyzeToolincrewai-toolsthat analyzes video content with TwelveLabs' Pegasus video-understanding model. Given a video — either a public URL or an already-indexedvideo_id— and a natural-language prompt, it returns a text answer generated from the video's visuals, speech, and on-screen text. Useful for video summarization, Q&A over video, content moderation, and metadata extraction.Why it helps crewAI
crewAI agents can already reason over text, images (VisionTool), and web content, but there's no first-class way to give an agent an understanding of video. This tool closes that gap, letting agents answer questions about, summarize, or moderate video content as part of a crew.
Opt-in / non-breaking
twelvelabsSDK is added as an optional dependency ([project.optional-dependencies] twelvelabs); the base package still imports without it (lazy import inside__init__, mirroringScrapegraphScrapeTooland other SDK-backed tools).crewai_tools/__init__.pyandcrewai_tools/tools/__init__.py, withpackage_dependenciesandenv_vars(TWELVELABS_API_KEY) declared the same way as sibling tools.How it was tested
lib/crewai-tools/tests/tools/twelvelabs_analyze_tool_test.py: no-network unit tests (SDK client mocked) covering API-key validation, required-input validation,video_idandvideo_urlrequest wiring, plus a live integration test gated onTWELVELABS_API_KEY(skipped without it).uv run ruff checkanduv run ruff formatclean on all changed files;uv run mypypasses on the new tool.tool.specs.jsonso the new tool's spec is included.analyzewiring end-to-end against the live API (auth,model_name,prompt,max_tokensall validated server-side).Notes
Per
CONTRIBUTING.md, this PR was prepared with AI assistance, so it should carry thellm-generatedlabel — I'm disclosing that here and will add the label.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Summary by CodeRabbit
New Features
Documentation
Tests