-
Notifications
You must be signed in to change notification settings - Fork 7.9k
feat(crewai-tools): add IBM Db2 search tool #5885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
PawanThakurIBM
wants to merge
42
commits into
crewAIInc:main
Choose a base branch
from
PawanThakurIBM:feat/db2-search-tool
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,398
−1
Open
Changes from all commits
Commits
Show all changes
42 commits
Select commit
Hold shift + click to select a range
379c9f4
feat(crewai-tools): add db2 search tool
PawanThakurIBM 02a44b9
refactor(crewai-tools): improve db2 search tool implementation
PawanThakurIBM 54f0006
feat(tools): improve DB2VectorSearchTool validation, security, and co…
PawanThakurIBM d2ac4c8
docs: add DB2SearchTool documentation
PawanThakurIBM fccb943
feat: add DB2 search tool
PawanThakurIBM 61d668b
Merge branch 'main' into feat/db2-search-tool
PawanThakurIBM a53a27a
docs: update DB2SearchTool documentation
PawanThakurIBM d14196e
fix: address CodeRabbit review feedback
PawanThakurIBM 0e1b992
fix: validate non-empty filter_by in DB2ToolSchema
PawanThakurIBM 60f9dc1
chore: trigger CodeRabbit re-review
PawanThakurIBM 468149e
feat: fortify DB2 tool; fixed JSON response shape, added input guards…
PawanThakurIBM 9655947
Merge branch 'main' into feat/db2-search-tool
priyanshu-krishnan1 f95d91d
refactor(db2): replace DB2Config with connection_string field
GeetikaChughIBM eac9fa6
refactor(db2): remove dead _setup_db2 validator and importlib import
GeetikaChughIBM 4770f20
refactor(db2): remove dead guard in _connect as _disconnect() is cal…
GeetikaChughIBM 1123815
fix(db2): tighten _validate_identifier regex. Old regex allowed leadi…
GeetikaChughIBM ab89454
fix(db2): replace __import__ with importlib.import_module in _generat…
GeetikaChughIBM 9c7026e
perf(db2): cache OpenAI client in _openai_client to avoid re-instanti…
GeetikaChughIBM d3de978
docs(db2): clarify tool description to mention embedding fallback
GeetikaChughIBM aa0cec1
docs(db2): update README supported features to clarify embedding beha…
GeetikaChughIBM 0ff4087
updated both code examples to use the correct import path and public …
GeetikaChughIBM c598d99
feat(crewai-tools): add db2 search tool
PawanThakurIBM d4c1bdd
refactor(crewai-tools): improve db2 search tool implementation
PawanThakurIBM 0ec84a5
feat(tools): improve DB2VectorSearchTool validation, security, and co…
PawanThakurIBM 0afe85f
docs: add DB2SearchTool documentation
PawanThakurIBM 8dd6b26
feat: add DB2 search tool
PawanThakurIBM 03ceb52
docs: update DB2SearchTool documentation
PawanThakurIBM d62c7d4
fix: address CodeRabbit review feedback
PawanThakurIBM c7533fe
fix: validate non-empty filter_by in DB2ToolSchema
PawanThakurIBM df32ebb
chore: trigger CodeRabbit re-review
PawanThakurIBM b22fe41
feat: fortify DB2 tool; fixed JSON response shape, added input guards…
PawanThakurIBM bf412c7
Merge branch 'crewAIInc:main' into feat/db2-search-tool
priyanshu-krishnan1 48d83eb
Merge remote-tracking branch 'prev-origin/feat/db2-search-tool' into …
GeetikaChughIBM 2583350
fix(db2): address ruff and mypy linter errors
GeetikaChughIBM f468692
style(db2): apply ruff format to db2_search_tool.py
GeetikaChughIBM 471cb00
Merge branch 'crewAIInc:main' into feat/db2-search-tool
GeetikaChughIBM a3c3c04
Merge branch 'main' into feat/db2-search-tool
lorenzejay 77f4efc
fix(db2-search-tool): address PR review comments
f77b609
Merge pull request #2 from PawanThakurIBM/feat/db2-search-tool-review…
PawanThakurIBM 0cf1309
fix(db2-search-tool): resolve default and string db2 package imports …
PawanThakurIBM 484c303
fix(db2-search-tool): export DB2VectorSearchTool and DB2ToolSchema fr…
PawanThakurIBM 4a44a21
docs(db2-search-tool): fix installation command and import path in RE…
PawanThakurIBM File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,211 @@ | ||
| --- | ||
| title: Db2 Vector Search Tool | ||
| description: Semantic vector search for CrewAI agents using IBM Db2 native VECTOR_DISTANCE capabilities. | ||
| icon: database | ||
| mode: "wide" | ||
| --- | ||
|
|
||
| # `DB2VectorSearchTool` | ||
|
|
||
| ## Description | ||
|
|
||
| Perform semantic vector similarity searches against IBM Db2 tables using the native `VECTOR_DISTANCE` function. | ||
| Supports configurable distance metrics, OpenAI or custom embeddings, metadata filtering, and result shaping. | ||
|
|
||
| ## Installation | ||
|
|
||
| ```bash | ||
| pip install ibm_db openai | ||
| ``` | ||
|
|
||
| Or with uv: | ||
|
|
||
| ```bash | ||
| uv add ibm_db openai | ||
| ``` | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| ```bash | ||
| OPENAI_API_KEY=your_openai_key # Required when using default OpenAI embeddings | ||
| DB2_CONNECTION_STRING=DATABASE=TESTDB;HOSTNAME=localhost;PORT=50000;PROTOCOL=TCPIP;UID=db2user;PWD=password; | ||
| ``` | ||
|
|
||
| ## Basic Usage | ||
|
|
||
| ```python | ||
| from crewai import Agent | ||
| from crewai_tools import DB2VectorSearchTool | ||
|
|
||
| tool = DB2VectorSearchTool( | ||
| connection_string="DATABASE=TESTDB;HOSTNAME=localhost;PORT=50000;PROTOCOL=TCPIP;UID=db2user;PWD=password;", | ||
| table_name="documents", | ||
| vector_column="embedding", | ||
| ) | ||
|
|
||
| agent = Agent( | ||
| role="Research Assistant", | ||
| goal="Find relevant information in documents", | ||
| tools=[tool], | ||
| ) | ||
| ``` | ||
|
|
||
| ## Full Semantic Search Workflow | ||
|
|
||
| ```python | ||
| import os | ||
| from dotenv import load_dotenv | ||
| from crewai import Agent, Task, Crew, Process | ||
| from crewai_tools import DB2VectorSearchTool | ||
|
|
||
| load_dotenv() | ||
|
|
||
| db2_tool = DB2VectorSearchTool( | ||
| connection_string=os.getenv("DB2_CONNECTION_STRING"), | ||
| table_name="documents", | ||
| vector_column="embedding", | ||
| return_columns=["content", "category"], | ||
| limit=3, | ||
| distance_metric="COSINE", | ||
| max_distance=0.35, | ||
| ) | ||
|
|
||
| search_agent = Agent( | ||
| role="Senior Semantic Search Agent", | ||
| goal="Find and analyse documents based on semantic search", | ||
| backstory="You are an expert research assistant who can find relevant information using semantic search in a Db2 database.", | ||
| tools=[db2_tool], | ||
| verbose=True, | ||
| ) | ||
|
|
||
| answer_agent = Agent( | ||
| role="Senior Answer Assistant", | ||
| goal="Generate answers based on retrieved context", | ||
| backstory="You are an expert assistant who generates answers from provided context.", | ||
| tools=[db2_tool], | ||
| verbose=True, | ||
| ) | ||
|
|
||
| search_task = Task( | ||
| description="""Search for relevant documents about {query}. | ||
| Include the relevant information found, vector distances, and returned fields.""", | ||
| agent=search_agent, | ||
| ) | ||
|
|
||
| answer_task = Task( | ||
| description="Given the retrieved Db2 context, generate a final answer.", | ||
| agent=answer_agent, | ||
| ) | ||
|
|
||
| crew = Crew( | ||
| agents=[search_agent, answer_agent], | ||
| tasks=[search_task, answer_task], | ||
| process=Process.sequential, | ||
| verbose=True, | ||
| ) | ||
|
|
||
| result = crew.kickoff(inputs={"query": "What is the role of X in the document?"}) | ||
| print(result) | ||
| ``` | ||
|
|
||
| ## Tool Parameters | ||
|
|
||
| | Parameter | Type | Default | Description | | ||
| |---|---|---|---| | ||
| | `connection_string` | `str` | required | Db2 connection string. Format: `DATABASE=x;HOSTNAME=x;PORT=50000;PROTOCOL=TCPIP;UID=x;PWD=x;` | | ||
| | `table_name` | `str` | `"documents"` | Table to search. Supports `schema.table` notation. | | ||
| | `vector_column` | `str` | `"embedding"` | Column storing the vector embeddings. | | ||
| | `embedding_model` | `str` | `"text-embedding-3-large"` | OpenAI model used when no custom embedding function is provided. | | ||
| | `return_columns` | `list[str]` | `["content"]` | Columns to include in each result. Must contain at least one entry. | | ||
| | `limit` | `int` | `3` | Maximum number of results (1–100). | | ||
| | `distance_metric` | `str` | `"COSINE"` | Db2 distance metric. See supported values below. | | ||
| | `max_distance` | `float \| None` | `None` | Drop results whose distance exceeds this value. | | ||
| | `custom_embedding_fn` | `Callable[[str], list[float]] \| None` | `None` | Custom embedding function. Overrides OpenAI when provided. | | ||
|
|
||
| ## Supported Distance Metrics | ||
|
|
||
| The following values map directly to the Db2 `VECTOR_DISTANCE` function: | ||
|
|
||
| - `COSINE` | ||
| - `EUCLIDEAN` | ||
| - `EUCLIDEAN_SQUARED` | ||
| - `DOT` | ||
| - `HAMMING` | ||
| - `MANHATTAN` | ||
|
|
||
| Reference: [IBM Db2 VECTOR_DISTANCE documentation](https://www.ibm.com/docs/en/db2/12.1.x?topic=functions-vector-distance) | ||
|
|
||
| ## Schema Parameters (per query) | ||
|
|
||
| | Parameter | Type | Required | Description | | ||
| |---|---|---|---| | ||
| | `query` | `str` | ✅ | The search query. | | ||
| | `filter_by` | `str \| None` | ❌ | Column name for metadata filtering. Must be paired with `filter_value`. | | ||
| | `filter_value` | `Any \| None` | ❌ | Value to filter on. Must be paired with `filter_by`. | | ||
|
|
||
| ## Return Format | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "results": [ | ||
| { | ||
| "distance": 0.1401, | ||
| "data": { | ||
| "content": "Document content here", | ||
| "category": "research" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| On error: | ||
|
|
||
| ```json | ||
| { | ||
| "success": false, | ||
| "error": "Description of what went wrong", | ||
| "error_type": "ExceptionClassName" | ||
| } | ||
| ``` | ||
|
|
||
| ## Metadata Filtering | ||
|
|
||
| ```python | ||
| result = db2_tool.run( | ||
| query="machine learning", | ||
| filter_by="category", | ||
| filter_value="research", | ||
| ) | ||
| ``` | ||
|
|
||
| `filter_by` and `filter_value` must always be provided together. Providing only one raises a validation error. | ||
|
|
||
| ## Custom Embeddings | ||
|
|
||
| Use any embedding model by supplying a `custom_embedding_fn`: | ||
|
|
||
| ```python | ||
| from sentence_transformers import SentenceTransformer | ||
| from crewai_tools import DB2VectorSearchTool | ||
|
|
||
| model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") | ||
|
|
||
| def custom_embeddings(text: str) -> list[float]: | ||
| return model.encode(text).tolist() | ||
|
|
||
| tool = DB2VectorSearchTool( | ||
| connection_string="DATABASE=TESTDB;HOSTNAME=localhost;PORT=50000;PROTOCOL=TCPIP;UID=db2user;PWD=password;", | ||
| table_name="documents", | ||
| custom_embedding_fn=custom_embeddings, | ||
| ) | ||
| ``` | ||
|
|
||
| When `custom_embedding_fn` is provided, `OPENAI_API_KEY` is not required. | ||
|
|
||
| ## Security Features | ||
|
|
||
| - SQL identifier validation (table, column names must match `^[A-Za-z][A-Za-z0-9_]*(\.[A-Za-z][A-Za-z0-9_]*)?$`) | ||
| - Parameterised SQL queries — values never interpolated into SQL strings | ||
| - Distance metric whitelist — only valid Db2 metric names accepted | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
91 changes: 91 additions & 0 deletions
91
lib/crewai-tools/src/crewai_tools/tools/db2_search_tool/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # DB2 Vector Search Tool | ||
|
|
||
| IBM DB2 Vector Search Tool for CrewAI. | ||
|
|
||
| Supports: | ||
|
|
||
| - IBM DB2 native VECTOR search | ||
| - OpenAI embeddings | ||
| - Custom embedding functions | ||
| - Metadata filtering | ||
| - Runtime dynamic imports | ||
| - Standardized CrewAI tool architecture | ||
|
|
||
| --- | ||
|
|
||
| # Installation | ||
|
|
||
| ```bash | ||
| uv add ibm_db openai | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Environment Variables | ||
|
|
||
| ```env | ||
| OPENAI_API_KEY=your_openai_key | ||
|
|
||
| DB2_CONNECTION_STRING=DATABASE=TESTDB;HOSTNAME=localhost;PORT=50000;PROTOCOL=TCPIP;UID=db2user;PWD=password; | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Example Usage | ||
|
|
||
| ```python | ||
| from crewai_tools import DB2VectorSearchTool | ||
|
|
||
| tool = DB2VectorSearchTool( | ||
| connection_string="DATABASE=TESTDB;HOSTNAME=localhost;PORT=50000;PROTOCOL=TCPIP;UID=db2user;PWD=password;", | ||
| table_name="documents", | ||
| ) | ||
|
|
||
| result = tool.run( | ||
| query="What is machine learning?", | ||
| ) | ||
|
|
||
| print(result) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Example With Metadata Filtering | ||
|
|
||
| ```python | ||
| result = tool.run( | ||
| query="AI papers", | ||
| filter_by="category", | ||
| filter_value="AI", | ||
| ) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| # Supported Features | ||
|
|
||
| - DB2 VECTOR datatype | ||
| - VECTOR_DISTANCE search | ||
| - COSINE similarity | ||
| - Metadata filtering | ||
| - Uses a custom embedding function if supplied, otherwise OpenAI embeddings | ||
|
|
||
| --- | ||
|
|
||
| # Architecture | ||
|
|
||
| This tool follows the same architecture as: | ||
|
|
||
| - QdrantVectorSearchTool | ||
| - WeaviateVectorSearchTool | ||
|
|
||
| Responsibilities: | ||
|
|
||
| - Generate query embeddings | ||
| - Perform vector similarity search | ||
| - Apply optional metadata filters | ||
| - Return normalized JSON results | ||
|
|
||
| This tool is retrieval-only. | ||
|
|
||
| Document ingestion should be handled separately. |
10 changes: 10 additions & 0 deletions
10
lib/crewai-tools/src/crewai_tools/tools/db2_search_tool/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| from crewai_tools.tools.db2_search_tool.db2_search_tool import ( | ||
| DB2ToolSchema, | ||
| DB2VectorSearchTool, | ||
| ) | ||
|
|
||
|
|
||
| __all__ = [ | ||
| "DB2ToolSchema", | ||
| "DB2VectorSearchTool", | ||
| ] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This import still fails:
DB2VectorSearchToolwas added tocrewai_tools.tools, but not the package-levelcrewai_tools/__init__.py. Please export it there and add a public-import test.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missed this one — DB2VectorSearchTool was only exported from crewai_tools.tools, not from the top-level crewai_tools package, so from crewai_tools import DB2VectorSearchTool would fail at import. Both DB2VectorSearchTool and DB2ToolSchema are now in crewai_tools/init.py and all. Added test_public_import_from_crewai_tools to cover the package-level import explicitly. All 70 tests passing.