v0.2.12#58
Conversation
…g handler for dlt loggers reuse one handler per path (#56) * fix: added retry to pipeline.py to mitigate WinError 5 "access denied", shared RotatingFileHandler cache so root and dlt loggers reuse one handler per path, avoiding rotation "file in use" errors * improve error surfacing, platform-gate the permission error to Windows, testing --------- Co-authored-by: Stran Dutton <sdutton@specterops.io>
WalkthroughThe PR moves the BHE URL into configuration, updates deployment examples, caches rotating log handlers by path, and adds bounded retries for transient pipeline filesystem failures with associated tests. ChangesBHE destination configuration
Shared rotating log handlers
Transient pipeline retries
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant BasePipeline
participant DltPipeline
participant RetryLogger
BasePipeline->>DltPipeline: run pipeline
DltPipeline-->>BasePipeline: transient filesystem failure
BasePipeline->>RetryLogger: log retry warning
BasePipeline->>BasePipeline: sleep with incremental backoff
BasePipeline->>DltPipeline: retry pipeline run
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
src/openhound/core/logging.py (2)
325-368: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueStale cached handlers on repeated
setup().If
setup()runs more than once with a differentlog_path/name(e.g. reconfiguration), the previously cached handler for the old path is removed from the loggers'handlerslists via.clear()but stays open inself._file_handlersforever — an accumulating, unclosed file descriptor per reconfiguration. Givensetup()is normally called once per process lifetime this is low-impact, but consider evicting/closing handlers for paths no longer referenced.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/logging.py` around lines 325 - 368, The setup() method in logging.py is leaving stale file handlers open when it is called multiple times with a changed log_path or name. Update setup() to evict any no-longer-used handler from self._file_handlers and close it before replacing handlers on dlt_logger and root_logger, so repeated reconfiguration does not accumulate open file descriptors. Use the setup(), self._file_handlers, self.handlers, and log_file_path symbols to locate the cleanup logic.
398-405: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueCache lookup/insert isn't atomic.
Two concurrent calls to
_get_file_handlerfor a new path could both miss the cache and each build aRotatingFileHandlerfor the same file before either is stored; the last write wins in the dict and the other handler is silently orphaned (open, unclosed, never used) — the exact multi-handler-on-one-file scenario this cache is meant to prevent. Low risk in current single-threaded setup flow, but worth athreading.Lockaround the check-and-set for defensive correctness.🔒 Optional lock-based fix
+import threading + class CustomLogger: def __init__(self, ...): ... self._file_handlers: dict[Path, "RotatingFileHandler"] = {} + self._file_handlers_lock = threading.Lock() def _get_file_handler(self, file_path: Path) -> "RotatingFileHandler": key = Path(file_path).resolve() - handler = self._file_handlers.get(key) - if handler is None: - handler = self._build_file_handler(file_path) - self._file_handlers[key] = handler - return handler + with self._file_handlers_lock: + handler = self._file_handlers.get(key) + if handler is None: + handler = self._build_file_handler(file_path) + self._file_handlers[key] = handler + return handler🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/logging.py` around lines 398 - 405, The shared handler cache in _get_file_handler is not atomic, so concurrent calls can create multiple RotatingFileHandler instances for the same path. Add a threading.Lock around the cache lookup/build/store sequence in Logging/_get_file_handler so the check-and-set is performed once per resolved Path key, and keep the existing _build_file_handler and _file_handlers behavior unchanged.src/openhound/core/pipeline.py (1)
44-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffConsider dlt's built-in
retry_loadhelper for transient load-step retries.dlt already ships
dlt.pipeline.helpers.retry_load, designed to be used withtenacity.Retryingfor retryingrun/loadon transient errors. Reimplementing bespoke retry/backoff logic here duplicates functionality dlt already provides and misses tenacity's more battle-tested jitter/backoff strategies. Worth evaluating whetherretry_loadplus a customretry_if_exceptionpredicate (checking_transient_filesystem_cause) could replace this hand-rolled loop.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/pipeline.py` around lines 44 - 91, The transient retry logic in _run is hand-rolling backoff and retry handling that dlt already provides. Replace the custom for-loop, warning, and sleep logic in Pipeline._run with dlt.pipeline.helpers.retry_load wired through tenacity.Retrying, and keep the existing transient predicate by using a custom retry_if_exception check based on _transient_filesystem_cause. Preserve the current special handling for PipelineStepFailed and PermissionError in _run, but let retry_load manage retry/backoff behavior instead of the bespoke _MAX_TRANSIENT_RETRIES path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/openhound/core/logging.py`:
- Around line 325-368: The setup() method in logging.py is leaving stale file
handlers open when it is called multiple times with a changed log_path or name.
Update setup() to evict any no-longer-used handler from self._file_handlers and
close it before replacing handlers on dlt_logger and root_logger, so repeated
reconfiguration does not accumulate open file descriptors. Use the setup(),
self._file_handlers, self.handlers, and log_file_path symbols to locate the
cleanup logic.
- Around line 398-405: The shared handler cache in _get_file_handler is not
atomic, so concurrent calls can create multiple RotatingFileHandler instances
for the same path. Add a threading.Lock around the cache lookup/build/store
sequence in Logging/_get_file_handler so the check-and-set is performed once per
resolved Path key, and keep the existing _build_file_handler and _file_handlers
behavior unchanged.
In `@src/openhound/core/pipeline.py`:
- Around line 44-91: The transient retry logic in _run is hand-rolling backoff
and retry handling that dlt already provides. Replace the custom for-loop,
warning, and sleep logic in Pipeline._run with dlt.pipeline.helpers.retry_load
wired through tenacity.Retrying, and keep the existing transient predicate by
using a custom retry_if_exception check based on _transient_filesystem_cause.
Preserve the current special handling for PipelineStepFailed and PermissionError
in _run, but let retry_load manage retry/backoff behavior instead of the bespoke
_MAX_TRANSIENT_RETRIES path.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3bc677f2-fa12-4f4e-8ff1-7629b28b3a52
📒 Files selected for processing (12)
deployments/helm/openhound/README.mddeployments/helm/values.example.yamlexample-configurations/bloodhound-enterprise/.dlt-example/config.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_github.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_jamf.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_okta.tomlexample-configurations/bloodhound-enterprise/docker-compose.ymlsrc/openhound/core/logging.pysrc/openhound/core/pipeline.pysrc/openhound/destinations/bloodhound_enterprise/destination.pytests/test_log_handlers.pytests/test_pipeline_retry.py
💤 Files with no reviewable changes (3)
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_okta.toml
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_github.toml
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_jamf.toml
…15) * Add CLI command to generate query bundle (#1) * Add CLI command to generate saved searches/query bundle * Add test for saved searches bundle generation * Added console output + help * Add antlr to dev deps * Add Cypher grammar * Fixed Cypher syntax test to work with OpenHound models + added query that should fail syntax checks * Update pyproject deps * Remove cypher test/bundle from pre-commit hook, this should only be done on the collector repos
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
tests/grammar/CypherParser.py (1)
1-20957: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winGenerated ANTLR artifact — exclude from lint, don't hand-fix.
This file is fully machine-generated (
# Generated from Cypher.g4 by ANTLR 4.13.2, matching theantlr4-python3-runtime>=4.13.2dependency). The Ruff findings (star import, mutable default class attributes, implicitOptional,inputshadowing the builtin) are all inherent to ANTLR's Python3 target codegen and will reappear identically on every regeneration — fixing them by hand provides no benefit and will be overwritten.Recommend adding
tests/grammar/**(or the specific generated*.py/*.interp/*.tokenspaths) to Ruff'sexclude/per-file-ignoresinpyproject.tomlso these recurring false positives stop surfacing in future PRs.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/grammar/CypherParser.py` around lines 1 - 20957, Exclude the machine-generated ANTLR artifacts under tests/grammar from Ruff linting in the project configuration. Update pyproject.toml to ignore the generated Python, interpreter, and token files in that directory, rather than modifying CypherParser or other generated output by hand.Source: Linters/SAST tools
src/openhound/cli/saved_search.py (1)
20-22: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove duplicated
OutputFormatenum.The
OutputFormatenum is already defined inopenhound.core.models.saved_search. Duplicating it here can lead to type-mismatch warnings and maintenance overhead. Import it from the models module instead.♻️ Proposed refactor
-class OutputFormat(str, Enum): - json = "json" - zip = "zip" -Update your imports at the top of the file to include
OutputFormat:from openhound.core.models.saved_search import QueryBundle, OutputFormat🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/cli/saved_search.py` around lines 20 - 22, Remove the local OutputFormat enum from the CLI module and import the existing OutputFormat alongside QueryBundle from openhound.core.models.saved_search. Update all references to use the imported model enum.src/openhound/core/models/saved_search.py (3)
30-33: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winUse
json.loadand specify UTF-8 encoding.Passing the file object directly to
json.loadis more memory-efficient than reading the entire file into a string withread(). Additionally, explicitly settingencoding="utf-8"prevents decoding errors on platforms where the system's default text encoding is not UTF-8 (such as Windows).♻️ Proposed fix
`@classmethod` def from_file(cls, file_path: Path) -> "SavedSearch": - with open(file_path, "r") as file_object: - json_object = json.loads(file_object.read()) + with open(file_path, "r", encoding="utf-8") as file_object: + json_object = json.load(file_object) return cls(**json_object)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/models/saved_search.py` around lines 30 - 33, Update SavedSearch.from_file to open the path with explicit UTF-8 encoding and pass the file object directly to json.load instead of calling read() and json.loads().
81-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse enum members as dictionary keys.
Since
Formatinherits fromstr, using raw strings works cleanly due to Python's equality rules. However, using the enum members explicitly is safer against potential future renaming and adheres better to enum idioms.♻️ Proposed refactor
model_choices = { - 'yaml': SavedSearchExtended, - 'json': SavedSearch, + Format.yaml: SavedSearchExtended, + Format.json: SavedSearch, }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/models/saved_search.py` around lines 81 - 84, Update the model_choices mapping in the SavedSearch model logic to use the corresponding Format enum members as keys instead of raw 'yaml' and 'json' strings, while preserving the existing SavedSearchExtended and SavedSearch value mappings.
68-71: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winUse
safe_loaddirectly on the file object and specify UTF-8 encoding.PyYAML's
safe_loadaccepts a file-like object, avoiding the need to load the entire file string into memory. Explicitly specifyingencoding="utf-8"guarantees consistent decoding across platforms.♻️ Proposed fix
`@classmethod` def from_file(cls, file_path: Path) -> "SavedSearchExtended": - with open(file_path, "r") as file_object: - yaml_object = safe_load(file_object.read()) + with open(file_path, "r", encoding="utf-8") as file_object: + yaml_object = safe_load(file_object) return cls(**yaml_object)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/openhound/core/models/saved_search.py` around lines 68 - 71, Update SavedSearchExtended.from_file to open the path with explicit UTF-8 encoding and pass the file object directly to safe_load, removing the intermediate read() call while preserving the existing cls(**yaml_object) construction.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhound/cli/saved_search.py`:
- Around line 73-74: Change saved_search.py lines 73-74 to accept a Path via
typer.Argument(file_okay=True, dir_okay=False, writable=True, ...), then update
lines 96-98 to call resolve() directly on output_path. In saved_search.py lines
92-113, update save, _to_json, and _to_zip to accept Path values; open the path
in text mode within _to_json and pass it directly to zipfile.ZipFile in _to_zip,
avoiding any pre-opened file handle.
In `@tests/test_cypher_syntax.py`:
- Around line 53-59: The test currently corrupts valid string literals by
manually stripping text after “//” before lexing. In the CypherParser setup
around validate_schema.query and CypherLexer, remove the
splitlines/uncomment_query transformation and pass the original
validate_schema.query directly to InputStream.
- Around line 59-62: Attach the existing CypherErrorListener instance to the
CypherLexer as well as the CypherParser in the test setup, ensuring lexical
errors are captured before tokenization and malformed queries cannot bypass
validation.
---
Nitpick comments:
In `@src/openhound/cli/saved_search.py`:
- Around line 20-22: Remove the local OutputFormat enum from the CLI module and
import the existing OutputFormat alongside QueryBundle from
openhound.core.models.saved_search. Update all references to use the imported
model enum.
In `@src/openhound/core/models/saved_search.py`:
- Around line 30-33: Update SavedSearch.from_file to open the path with explicit
UTF-8 encoding and pass the file object directly to json.load instead of calling
read() and json.loads().
- Around line 81-84: Update the model_choices mapping in the SavedSearch model
logic to use the corresponding Format enum members as keys instead of raw 'yaml'
and 'json' strings, while preserving the existing SavedSearchExtended and
SavedSearch value mappings.
- Around line 68-71: Update SavedSearchExtended.from_file to open the path with
explicit UTF-8 encoding and pass the file object directly to safe_load, removing
the intermediate read() call while preserving the existing cls(**yaml_object)
construction.
In `@tests/grammar/CypherParser.py`:
- Around line 1-20957: Exclude the machine-generated ANTLR artifacts under
tests/grammar from Ruff linting in the project configuration. Update
pyproject.toml to ignore the generated Python, interpreter, and token files in
that directory, rather than modifying CypherParser or other generated output by
hand.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ee709c5c-602a-4c5b-aa03-85d1d4f83768
📒 Files selected for processing (17)
.github/workflows/test.ymlpyproject.tomlsrc/openhound/cli/saved_search.pysrc/openhound/core/models/saved_search.pytests/grammar/CypherLexer.pytests/grammar/CypherListener.pytests/grammar/CypherParser.pytests/grammar/__init__.pytests/grammar/cypher/Cypher.g4tests/grammar/cypher/Cypher.interptests/grammar/cypher/Cypher.tokenstests/grammar/cypher/CypherLexer.interptests/grammar/cypher/CypherLexer.tokenstests/test_cypher_syntax.pytests/test_data/extensions/saved_searches/jamf_query_by_name_2.jsontests/test_data/extensions/saved_searches/jamf_query_by_name_error.jsontests/test_saved_searches_bundle.py
| output_path: Annotated[typer.FileTextWrite, typer.Argument( | ||
| help="Output file path for the generated saved-search bundle (including filename)")], |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | 🏗️ Heavy lift
Windows PermissionError from double-opening output file.
Using typer.FileTextWrite causes Typer to immediately open the file and hold a text-write lock. When building a ZIP bundle, _to_zip re-opens this exact same file by name using zipfile.ZipFile. On Windows, this dual exclusive lock collision triggers a PermissionError: [Errno 13] Permission denied, which contradicts the PR's goal (BHE-8837) of fixing transient access denied errors on Windows. Furthermore, it results in writing binary ZIP data into an actively bound text stream.
To fix this, change the CLI to resolve a Path without opening it, giving the model full control over the file's lifecycle and mode.
src/openhound/cli/saved_search.py#L73-L74: Change the argument type fromtyper.FileTextWritetoPath. Retain Typer's validation by usingtyper.Argument(file_okay=True, dir_okay=False, writable=True, ...).src/openhound/cli/saved_search.py#L96-L98: Update the print statement to call.resolve()directly onoutput_path, as it will now be aPathobject instead of aTextIOWrapper.src/openhound/core/models/saved_search.py#L92-L113: Updatesave,_to_json, and_to_zipto accept aPathobject. Open the file directly within_to_jsonin text mode, and pass thePathdirectly intozipfile.ZipFile.
🐛 Proposed fixes across both files
1. src/openhound/cli/saved_search.py:
- output_path: Annotated[typer.FileTextWrite, typer.Argument(
- help="Output file path for the generated saved-search bundle (including filename)")],
+ output_path: Annotated[
+ Path,
+ typer.Argument(
+ file_okay=True,
+ dir_okay=False,
+ writable=True,
+ help="Output file path for the generated saved-search bundle (including filename)",
+ ),
+ ],- console.print(
- f"[bold magenta]Output path:[/bold magenta] [italic]{Path(output_path.name).resolve()}[/italic]"
- )
+ console.print(
+ f"[bold magenta]Output path:[/bold magenta] [italic]{output_path.resolve()}[/italic]"
+ )2. src/openhound/core/models/saved_search.py:
- def _to_json(self, output_file: TextIOWrapper) -> None:
+ def _to_json(self, output_path: Path) -> None:
all_objects = [query.model_dump() for query in self.queries]
- output_file.write(json.dumps(all_objects, indent=2))
+ with open(output_path, "w", encoding="utf-8") as output_file:
+ json.dump(all_objects, output_file, indent=2)
- def _to_zip(self, output_file: TextIOWrapper) -> None:
+ def _to_zip(self, output_path: Path) -> None:
with zipfile.ZipFile(
- file=output_file.name,
+ file=output_path,
mode="w",
compression=zipfile.ZIP_DEFLATED,
compresslevel=9,
) as archive:
for query in self.queries:
archive.writestr(
zinfo_or_arcname=f"{query.name}.json",
data=query.model_dump_json().encode(),
)
- def save(self, output_file: TextIOWrapper, output_format: OutputFormat = OutputFormat.json) -> None:
+ def save(self, output_path: Path, output_format: OutputFormat = OutputFormat.json) -> None:
if output_format == OutputFormat.json:
- self._to_json(output_file)
+ self._to_json(output_path)
elif output_format == OutputFormat.zip:
- self._to_zip(output_file)
+ self._to_zip(output_path)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| output_path: Annotated[typer.FileTextWrite, typer.Argument( | |
| help="Output file path for the generated saved-search bundle (including filename)")], | |
| output_path: Annotated[ | |
| Path, | |
| typer.Argument( | |
| file_okay=True, | |
| dir_okay=False, | |
| writable=True, | |
| help="Output file path for the generated saved-search bundle (including filename)", | |
| ), | |
| ], |
| output_path: Annotated[typer.FileTextWrite, typer.Argument( | |
| help="Output file path for the generated saved-search bundle (including filename)")], | |
| console.print( | |
| f"[bold magenta]Output path:[/bold magenta] [italic]{output_path.resolve()}[/italic]" | |
| ) |
| output_path: Annotated[typer.FileTextWrite, typer.Argument( | |
| help="Output file path for the generated saved-search bundle (including filename)")], | |
| def _to_json(self, output_path: Path) -> None: | |
| all_objects = [query.model_dump() for query in self.queries] | |
| with open(output_path, "w", encoding="utf-8") as output_file: | |
| json.dump(all_objects, output_file, indent=2) | |
| def _to_zip(self, output_path: Path) -> None: | |
| with zipfile.ZipFile( | |
| file=output_path, | |
| mode="w", | |
| compression=zipfile.ZIP_DEFLATED, | |
| compresslevel=9, | |
| ) as archive: | |
| for query in self.queries: | |
| archive.writestr( | |
| zinfo_or_arcname=f"{query.name}.json", | |
| data=query.model_dump_json().encode(), | |
| ) | |
| def save(self, output_path: Path, output_format: OutputFormat = OutputFormat.json) -> None: | |
| if output_format == OutputFormat.json: | |
| self._to_json(output_path) | |
| elif output_format == OutputFormat.zip: | |
| self._to_zip(output_path) |
📍 Affects 2 files
src/openhound/cli/saved_search.py#L73-L74(this comment)src/openhound/cli/saved_search.py#L96-L98src/openhound/core/models/saved_search.py#L92-L113
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/openhound/cli/saved_search.py` around lines 73 - 74, Change
saved_search.py lines 73-74 to accept a Path via typer.Argument(file_okay=True,
dir_okay=False, writable=True, ...), then update lines 96-98 to call resolve()
directly on output_path. In saved_search.py lines 92-113, update save, _to_json,
and _to_zip to accept Path values; open the path in text mode within _to_json
and pass it directly to zipfile.ZipFile in _to_zip, avoiding any pre-opened file
handle.
| # Split query into multiple lines in order to remove line comments | ||
| lines = validate_schema.query.splitlines() | ||
| uncomment_query = "\n".join(line.split("//")[0].rstrip() for line in lines) | ||
| uncomment_query = uncomment_query.lstrip("\n") | ||
|
|
||
| # Attempt to load/parse the query using the CypherParser | ||
| lexer = CypherLexer(InputStream(uncomment_query)) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Parse the original query instead of stripping // manually.
This truncates valid string literals such as URLs containing //. The grammar already handles line comments, so pass the unmodified query to the lexer.
Proposed fix
- # Split query into multiple lines in order to remove line comments
- lines = validate_schema.query.splitlines()
- uncomment_query = "\n".join(line.split("//")[0].rstrip() for line in lines)
- uncomment_query = uncomment_query.lstrip("\n")
-
# Attempt to load/parse the query using the CypherParser
- lexer = CypherLexer(InputStream(uncomment_query))
+ lexer = CypherLexer(InputStream(validate_schema.query))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Split query into multiple lines in order to remove line comments | |
| lines = validate_schema.query.splitlines() | |
| uncomment_query = "\n".join(line.split("//")[0].rstrip() for line in lines) | |
| uncomment_query = uncomment_query.lstrip("\n") | |
| # Attempt to load/parse the query using the CypherParser | |
| lexer = CypherLexer(InputStream(uncomment_query)) | |
| # Attempt to load/parse the query using the CypherParser | |
| lexer = CypherLexer(InputStream(validate_schema.query)) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_cypher_syntax.py` around lines 53 - 59, The test currently
corrupts valid string literals by manually stripping text after “//” before
lexing. In the CypherParser setup around validate_schema.query and CypherLexer,
remove the splitlines/uncomment_query transformation and pass the original
validate_schema.query directly to InputStream.
| lexer = CypherLexer(InputStream(uncomment_query)) | ||
| stream = CommonTokenStream(lexer) | ||
| parser = CypherParser(stream) | ||
| parser.addErrorListener(CypherErrorListener()) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== tests/test_cypher_syntax.py (relevant lines) ==\n'
nl -ba tests/test_cypher_syntax.py | sed -n '1,140p'
printf '\n== Search for CypherErrorListener usage ==\n'
rg -n "CypherErrorListener|removeErrorListeners|addErrorListener|lexer.addErrorListener|parser.addErrorListener" -S .Repository: SpecterOps/OpenHound
Length of output: 250
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== tests/test_cypher_syntax.py (relevant lines) ==\n'
awk 'NR>=1 && NR<=140 { printf "%5d %s\n", NR, $0 }' tests/test_cypher_syntax.py
printf '\n== Search for CypherErrorListener usage ==\n'
rg -n "CypherErrorListener|removeErrorListeners|addErrorListener|lexer.addErrorListener|parser.addErrorListener" -S .Repository: SpecterOps/OpenHound
Length of output: 3656
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Generated grammar files ==\n'
fd -a 'CypherLexer.py' tests . || true
fd -a 'CypherParser.py' tests . || true
printf '\n== Outline CypherLexer.py ==\n'
ast-grep outline tests/.grammar/CypherLexer.py 2>/dev/null || true
printf '\n== Outline CypherParser.py ==\n'
ast-grep outline tests/.grammar/CypherParser.py 2>/dev/null || trueRepository: SpecterOps/OpenHound
Length of output: 472
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== CypherLexer.py error-related lines ==\n'
rg -n "notifyListeners|syntaxError|ErrorListener|addErrorListener|removeErrorListeners|recover|emit" tests/grammar/CypherLexer.py tests/grammar/CypherParser.py
printf '\n== CypherLexer.py top section ==\n'
awk 'NR>=1 && NR<=220 { printf "%5d %s\n", NR, $0 }' tests/grammar/CypherLexer.py
printf '\n== CypherParser.py top section ==\n'
awk 'NR>=1 && NR<=220 { printf "%5d %s\n", NR, $0 }' tests/grammar/CypherParser.pyRepository: SpecterOps/OpenHound
Length of output: 21653
tests/test_cypher_syntax.py:59-62 — Attach the same error listener to the lexer too. Lexical errors are emitted before the parser sees the input, so parser-only listeners can miss malformed characters and let bad queries slip through.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_cypher_syntax.py` around lines 59 - 62, Attach the existing
CypherErrorListener instance to the CypherLexer as well as the CypherParser in
the test setup, ensuring lexical errors are captured before tokenization and
malformed queries cannot bypass validation.
|
@coderabbitai full review |
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/openhound/core/pipeline.py`:
- Around line 60-68: Update the ParseException raise in the
ResourceExtractionError branch to explicitly suppress or chain the original
exception, matching the surrounding ConfigException handling; use “from None” if
the original err should not appear in the traceback, otherwise chain with “from
err” as appropriate.
- Around line 70-76: Update the wrapped-error branch that calls
_transient_filesystem_cause so retries occur only when sys.platform is "win32";
preserve raising the original error for non-Windows platforms and keep the
existing transient-cause handling on Windows.
In `@tests/test_log_handlers.py`:
- Around line 193-233: Update
test_root_and_dlt_loggers_share_single_file_handler cleanup to close every
handler retained in custom_logger._file_handlers before restoring global logging
state. Restore the monkeypatched RUNTIME__LOG_PATH environment explicitly before
calling logger_override.setup(), ensuring setup runs with the original
environment; retain the existing finally-based cleanup behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 724125ef-2dbe-41ee-b1e4-ad15186883a3
📒 Files selected for processing (12)
deployments/helm/openhound/README.mddeployments/helm/values.example.yamlexample-configurations/bloodhound-enterprise/.dlt-example/config.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_github.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_jamf.tomlexample-configurations/bloodhound-enterprise/.dlt-example/secrets_okta.tomlexample-configurations/bloodhound-enterprise/docker-compose.ymlsrc/openhound/core/logging.pysrc/openhound/core/pipeline.pysrc/openhound/destinations/bloodhound_enterprise/destination.pytests/test_log_handlers.pytests/test_pipeline_retry.py
💤 Files with no reviewable changes (3)
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_okta.toml
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_jamf.toml
- example-configurations/bloodhound-enterprise/.dlt-example/secrets_github.toml
| if isinstance(err.exception, ResourceExtractionError): | ||
| extract_cause: ResourceExtractionError = err.exception | ||
| raise ParseException( | ||
| pipeline_name=err.pipeline.pipeline_name, | ||
| destination=str(err.pipeline.destination), | ||
| dataset_name=err.pipeline.dataset_name, | ||
| step=extract_cause.pipe_name, | ||
| message=extract_cause.msg, | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add exception chaining to the ParseException raise.
Static analysis flags this raise for missing from err/from None. The sibling ConfigException raise just above uses from None; this one omits it, so tracebacks will show a confusing "During handling of the above exception, another exception occurred" instead of a clean chain.
Proposed fix
if isinstance(err.exception, ResourceExtractionError):
extract_cause: ResourceExtractionError = err.exception
raise ParseException(
pipeline_name=err.pipeline.pipeline_name,
destination=str(err.pipeline.destination),
dataset_name=err.pipeline.dataset_name,
step=extract_cause.pipe_name,
message=extract_cause.msg,
- )
+ ) from None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if isinstance(err.exception, ResourceExtractionError): | |
| extract_cause: ResourceExtractionError = err.exception | |
| raise ParseException( | |
| pipeline_name=err.pipeline.pipeline_name, | |
| destination=str(err.pipeline.destination), | |
| dataset_name=err.pipeline.dataset_name, | |
| step=extract_cause.pipe_name, | |
| message=extract_cause.msg, | |
| ) | |
| if isinstance(err.exception, ResourceExtractionError): | |
| extract_cause: ResourceExtractionError = err.exception | |
| raise ParseException( | |
| pipeline_name=err.pipeline.pipeline_name, | |
| destination=str(err.pipeline.destination), | |
| dataset_name=err.pipeline.dataset_name, | |
| step=extract_cause.pipe_name, | |
| message=extract_cause.msg, | |
| ) from None |
🧰 Tools
🪛 Ruff (0.15.21)
[warning] 62-68: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/openhound/core/pipeline.py` around lines 60 - 68, Update the
ParseException raise in the ResourceExtractionError branch to explicitly
suppress or chain the original exception, matching the surrounding
ConfigException handling; use “from None” if the original err should not appear
in the traceback, otherwise chain with “from err” as appropriate.
Source: Linters/SAST tools
| if _transient_filesystem_cause(err) is None: | ||
| raise | ||
| last_err = err | ||
| except PermissionError as err: | ||
| if sys.platform != "win32" or err.errno not in _TRANSIENT_RENAME_ERRNOS: | ||
| raise | ||
| last_err = err |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== pipeline.py ==\n'
sed -n '1,130p' src/openhound/core/pipeline.py
printf '\n== test_pipeline_retry.py ==\n'
sed -n '1,260p' tests/test_pipeline_retry.pyRepository: SpecterOps/OpenHound
Length of output: 7739
Gate the wrapped retry path on Windows too. _transient_filesystem_cause(err) will retry wrapped EACCES/EPERM on Linux/macOS as well, even though the bare PermissionError branch is Windows-only. That can turn real permission failures into five delayed retries with misleading warnings; if this is meant to cover the same rename quirk, add the same sys.platform == "win32" check here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/openhound/core/pipeline.py` around lines 70 - 76, Update the
wrapped-error branch that calls _transient_filesystem_cause so retries occur
only when sys.platform is "win32"; preserve raising the original error for
non-Windows platforms and keep the existing transient-cause handling on Windows.
| def test_root_and_dlt_loggers_share_single_file_handler(tmp_path, monkeypatch): | ||
| """The root and dlt loggers writing to openhound.log must share one handler | ||
| instance so the file is only opened once, which is required for rotation on | ||
| Windows where an open handle blocks renaming the file.""" | ||
| # A sibling module may set RUNTIME__LOG_PATH on import; keep our explicit base_path. | ||
| monkeypatch.delenv("RUNTIME__LOG_PATH", raising=False) | ||
| custom_logger = CustomLogger("openhound.log", base_path=str(tmp_path)) | ||
|
|
||
| try: | ||
| custom_logger.setup() | ||
|
|
||
| root_logger = logging.getLogger() | ||
| dlt_logger = logging.getLogger("dlt") | ||
|
|
||
| root_file_handlers = [ | ||
| handler | ||
| for handler in root_logger.handlers | ||
| if isinstance(handler, RotatingFileHandler) | ||
| ] | ||
| dlt_file_handlers = [ | ||
| handler | ||
| for handler in dlt_logger.handlers | ||
| if isinstance(handler, RotatingFileHandler) | ||
| ] | ||
|
|
||
| assert len(root_file_handlers) == 1, ( | ||
| "The root logger should have exactly one rotating file handler" | ||
| ) | ||
| assert len(dlt_file_handlers) == 1, ( | ||
| "The dlt logger should have exactly one rotating file handler" | ||
| ) | ||
| assert root_file_handlers[0] is dlt_file_handlers[0], ( | ||
| "Root and dlt loggers must share the same handler instance for openhound.log" | ||
| ) | ||
| assert root_file_handlers[0].baseFilename.endswith("openhound.log"), ( | ||
| "The shared handler should write to 'openhound.log'" | ||
| ) | ||
| finally: | ||
| # Restore the shared global logging state for subsequent tests. | ||
| logger_override.setup() | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Close cached file handlers and restore the environment before setup.
The locally instantiated custom_logger holds open file handlers in its _file_handlers cache. On Windows, failing to close these handlers leaves the files locked, which causes pytest to fail with a WinError 32 when it attempts to clean up tmp_path.
Additionally, calling logger_override.setup() before monkeypatch automatically restores the environment (which happens after the test function exits) means the global logger might be incorrectly re-initialized without RUNTIME__LOG_PATH. Manually undoing the monkeypatch beforehand ensures the correct state is restored.
🛠️ Proposed fix to clean up resources
finally:
+ # Undo monkeypatch before setup to ensure restored logger sees correct env vars
+ monkeypatch.undo()
# Restore the shared global logging state for subsequent tests.
logger_override.setup()
+ # Close cached handlers to release file locks (prevents WinError 32 during tmp_path cleanup)
+ for handler in custom_logger._file_handlers.values():
+ handler.close()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_root_and_dlt_loggers_share_single_file_handler(tmp_path, monkeypatch): | |
| """The root and dlt loggers writing to openhound.log must share one handler | |
| instance so the file is only opened once, which is required for rotation on | |
| Windows where an open handle blocks renaming the file.""" | |
| # A sibling module may set RUNTIME__LOG_PATH on import; keep our explicit base_path. | |
| monkeypatch.delenv("RUNTIME__LOG_PATH", raising=False) | |
| custom_logger = CustomLogger("openhound.log", base_path=str(tmp_path)) | |
| try: | |
| custom_logger.setup() | |
| root_logger = logging.getLogger() | |
| dlt_logger = logging.getLogger("dlt") | |
| root_file_handlers = [ | |
| handler | |
| for handler in root_logger.handlers | |
| if isinstance(handler, RotatingFileHandler) | |
| ] | |
| dlt_file_handlers = [ | |
| handler | |
| for handler in dlt_logger.handlers | |
| if isinstance(handler, RotatingFileHandler) | |
| ] | |
| assert len(root_file_handlers) == 1, ( | |
| "The root logger should have exactly one rotating file handler" | |
| ) | |
| assert len(dlt_file_handlers) == 1, ( | |
| "The dlt logger should have exactly one rotating file handler" | |
| ) | |
| assert root_file_handlers[0] is dlt_file_handlers[0], ( | |
| "Root and dlt loggers must share the same handler instance for openhound.log" | |
| ) | |
| assert root_file_handlers[0].baseFilename.endswith("openhound.log"), ( | |
| "The shared handler should write to 'openhound.log'" | |
| ) | |
| finally: | |
| # Restore the shared global logging state for subsequent tests. | |
| logger_override.setup() | |
| def test_root_and_dlt_loggers_share_single_file_handler(tmp_path, monkeypatch): | |
| """The root and dlt loggers writing to openhound.log must share one handler | |
| instance so the file is only opened once, which is required for rotation on | |
| Windows where an open handle blocks renaming the file.""" | |
| # A sibling module may set RUNTIME__LOG_PATH on import; keep our explicit base_path. | |
| monkeypatch.delenv("RUNTIME__LOG_PATH", raising=False) | |
| custom_logger = CustomLogger("openhound.log", base_path=str(tmp_path)) | |
| try: | |
| custom_logger.setup() | |
| root_logger = logging.getLogger() | |
| dlt_logger = logging.getLogger("dlt") | |
| root_file_handlers = [ | |
| handler | |
| for handler in root_logger.handlers | |
| if isinstance(handler, RotatingFileHandler) | |
| ] | |
| dlt_file_handlers = [ | |
| handler | |
| for handler in dlt_logger.handlers | |
| if isinstance(handler, RotatingFileHandler) | |
| ] | |
| assert len(root_file_handlers) == 1, ( | |
| "The root logger should have exactly one rotating file handler" | |
| ) | |
| assert len(dlt_file_handlers) == 1, ( | |
| "The dlt logger should have exactly one rotating file handler" | |
| ) | |
| assert root_file_handlers[0] is dlt_file_handlers[0], ( | |
| "Root and dlt loggers must share the same handler instance for openhound.log" | |
| ) | |
| assert root_file_handlers[0].baseFilename.endswith("openhound.log"), ( | |
| "The shared handler should write to 'openhound.log'" | |
| ) | |
| finally: | |
| # Undo monkeypatch before setup to ensure restored logger sees correct env vars | |
| monkeypatch.undo() | |
| # Restore the shared global logging state for subsequent tests. | |
| logger_override.setup() | |
| # Close cached handlers to release file locks (prevents WinError 32 during tmp_path cleanup) | |
| for handler in custom_logger._file_handlers.values(): | |
| handler.close() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_log_handlers.py` around lines 193 - 233, Update
test_root_and_dlt_loggers_share_single_file_handler cleanup to close every
handler retained in custom_logger._file_handlers before restoring global logging
state. Restore the monkeypatched RUNTIME__LOG_PATH environment explicitly before
calling logger_override.setup(), ensuring setup runs with the original
environment; retain the existing finally-based cleanup behavior.
3 participants
1. PR #57 by @StranDutton: fix: move non-sensitive data out of secrets.toml - BED-8838
2. PR #56 by @ktstrader : fix: added retry to pipeline.py to mitigate Windows "access denied" error, added log handler for dlt loggers reuse one handler per path - BED-8837(fix: added retry to pipeline.py to mitigate "access denied", added log handler for dlt loggers reuse one handler per path)
Important
Documentation (bloodhound-docs) update should be released at the same time (in relation to #57 OH config updates)
Summary by CodeRabbit
Configuration
Bug Fixes
Documentation