hiperai: add SecureAI integration for security and AI-governance events - #21158
juanmandelbaum wants to merge 7 commits into
Conversation
Initial release of the HiperAI SecureAI integration package. SecureAI is an enterprise AI security and governance platform that provides real-time monitoring of LLM usage, SMLTP policy enforcement, prompt injection detection, and DLP for AI workflows. This integration receives security events via HTTP endpoint and normalizes them to ECS fields including event categorization, severity mapping, GeoIP enrichment, and user agent parsing. Event types include: authentication, SMLTP violations, chat completions, DLP triggers, API usage, and audit logs.
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
|
/test |
|
@juanmandelbaum - thank you for your contributions. Appreciate it. are the basic core skills to use. I think it would help with refining the implementation and conforming it to some of our conventions. Happy to help with this as well. The |
Full quality pass using the integration-skills review/maintain workflow, in response to reviewer feedback on PR elastic#21158. - Bump format_version to 3.4.2 and kibana version floor to the new-package standard ^8.19.0 || ^9.1.0. - Fix the preserve_original_event webhook toggle: it was writing a nonexistent http_endpoint config key; now it correctly pushes into tags, matching how the pipeline gates on it. Add url var and publisher_pipeline.disable_host coupling with the forwarded tag. - Require and surface secret_header/secret_value in Fleet UI instead of hiding webhook auth behind advanced options. - Fix JSE00001 compliance (message -> event.original rename/remove pair) and remove the deprecated trailing event.original cleanup processor. - Replace the event.category append chain with a Painless script + params lookup table covering all 13 documented event types with matching event.type values; this also fixes a dead-code bug where the category conditions checked json.event_type after it had already been renamed to event.action. Fix the same class of bug in the event.outcome block. - Set ecs.version, tag every processor, guard geoip lookups on IP presence, fix Mustache double-brace vs triple-brace field interpolation, and reorder on_failure to tag failed documents for retention. - Restructure base-fields.yml to use external: ecs (correct descriptions instead of hand-written placeholders); move input.type to fields/beats.yml. - Move docs/README.md source to _dev/build/docs/README.md so the shipped file is actually rendered instead of containing raw template placeholders. - Anonymize sample_event.json and all pipeline test fixtures (RFC 5737 IPs, example.com/example.org emails). - Split the single combined pipeline test fixture into one file per event type (14 total), add test-common-config.yml to exercise the preserve_original_event branch, and add coverage for the previously untested security_incident event type and the category-fallback script branch. - Bump the ECS dependency in _dev/build/build.yml to git@v9.3.0 and fix changelog.yml''s placeholder PR link to the real PR elastic#21158. elastic-package format/lint/check/build all pass clean; test pipeline 28/28 assertions pass across 14 fixtures.
juanmandelbaum
changed the title
|
/test |
|
Thanks @qcorporation! Ran a full pass with Summary of what changed (see d2f4219 for the full diff):
|
|
/test |
|
@vera-review-bot review |
Fixes the failing dev/codeowners.Check step in Buildkite: manifest.yml declares owner.github: Hiper-AI but no matching CODEOWNERS entry existed for packages/hiperai, so it fell through to the repo-wide /packages/ fallback (@elastic/integrations-triaging) with no explicit entry for this path at all. Verified locally against the actual check (dev/codeowners.Check) rather than guessing at the format.
|
/test |
1 similar comment
|
/test |
sample_event.json was wrapped in a test-fixture style {events: [...]} array,
so the static test failed with 'field events is used as array of objects'.
Replace it with a single document taken from the login_success pipeline
test output, which also brings it in line with the current pipeline
(ecs.version, event.type, related.*). Regenerated docs/README.md.
Verified with elastic-package test (asset, static, pipeline) against
stack 8.19.0, matching CI.
|
/test |
1 similar comment
|
/test |
|
@vera-review-bot review |
🚀 Benchmarks reportTo see the full report comment with |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
Review summaryIssues found across the latest commits db1cdfd — 1 critical, 1 high, 3 medium, 4 low
Package-level:
🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills
|
Ownership: - manifest owner.github is now elastic/security-service-integrations, with a matching CODEOWNERS entry, following the convention used by the other partner security packages (bitsight, doppel, spycloud, tychon). Screenshot: - Replace the Elastic SIEM configuration screenshot with one that has the API key masked and the deployment URL redacted. The previously committed key has been rotated. - Correct the declared screenshot sizes, which claimed 1920x1080 for three images that are none of those dimensions. Input: - Pass preserve_original_event to the http_endpoint input. The toggle previously only added the tag, so event.original was never populated; the system test now shows it present. Pipeline: - Map event.severity onto the Elastic severity scale (21/47/73/99) with a painless script, matching crowdstrike, sentinel_one, m365_defender and others, instead of the 1-5 values that were not comparable with anything. - Complete the by_category fallback so every category declared in fields.yml resolves (smltp, system and analytics were missing). - Record date parsing failures in error.message rather than discarding them with ignore_failure. - Drop the processor that re-appended the forwarded tag, which overrode an operator who removed it from the Tags variable and duplicated what the stream template already does. Docs: - Rewrite the README to the structure required by the documentation guidelines, and enable docs_structure_enforced in validation.yml so elastic-package check verifies it. - Drop the Elastic Stack version compatibility claim, which can skew against the range declared in the manifest. - Correct the set up steps: the URL path was missing from the endpoint and the secret header was described as optional although it is required. Tests: - Add a system test and its docker deploy, exercising the webhook path end to end through an agent. - Regenerate sample_event.json from that system test run, replacing the hand-written one. - Add a malformed-record pipeline fixture covering unknown severity, invalid ip, actor-only records and an unmapped event type. Verified against stack 8.19.0: elastic-package check and the full test suite pass (2 asset, 30 pipeline, 1 static, 1 system).
|
Thanks @efd6 and @vera-review-bot — all review comments addressed in 4d09a4f. The API key has been rotated. The screenshot is replaced with one where the key is masked and the deployment URL is redacted. I also corrected the declared screenshot sizes, which claimed @efd6
@vera-review-bot
Not changed, flagging for your call
Verified against stack 8.19.0 to match CI: |
|
/test |
1 similar comment
|
/test |
efd6
left a comment
efd6
left a comment
There was a problem hiding this comment.
@juanmandelbaum Thanks for the summary, very helpful in reviewing.
One nit only, then LGTM
| on_failure: | ||
| - append: | ||
| field: error.message | ||
| tag: append_error_message_parse_timestamp | ||
| value: >- | ||
| Processor '{{{ _ingest.on_failure_processor_type }}}' | ||
| {{{#_ingest.on_failure_processor_tag}}}with tag '{{{ _ingest.on_failure_processor_tag }}}' {{{/_ingest.on_failure_processor_tag}}}failed with message '{{{ _ingest.on_failure_message }}}' |
There was a problem hiding this comment.
Because we have these here, I'd suggest that you have a conditional append to tags with "preserve_original_event" similar to the one in the global on_failure, but with an if: ctx.error?.message != null so that original events are retained when either the date or script processors fail.
There was a problem hiding this comment.
Done in 79b44ec — appended at the end of the processor list, with allow_duplicates: false so it does not double up with the tag the input already sets.
Verified by generating against a fixture with an unparseable timestamp: the document comes out with
"error": { "message": ["Processor 'date' with tag 'parse_timestamp' failed with message 'Failed to parse with all enclosed parsers'"] },
"tags": ["preserve_original_event"]and the rest of the document intact and correctly categorized. That fixture is not committed, for the checkErrorMessage reason in my previous comment.
The date and severity processors handle their own failures, so the pipeline keeps running and the preserve_original_event append in the global on_failure never sees them. Append the tag whenever error.message is set, so the raw event is retained for triage in those cases too. Suggested by @efd6 in review. Verified by generating a fixture with an unparseable timestamp: the document comes out with error.message set and tags [preserve_original_event], with the rest of the document intact. That fixture is not committed, because elastic-package fails any pipeline test whose document carries a non-nil error.message.
|
/test |
|
/test |
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
|
Tick the box to add this pull request to the merge queue (same as
|
|
I would propose the following commit message for the squash. @juanmandelbaum How were the test samples obtained; it is helpful to include their provenance in the commit message if issues arise. For example: derived from documentation; obtained from a live instance [with sanitization], etc. |
|
@efd6 The sample/test events ( For the squash, here's a proposed commit message with a provenance line added, and "Hiper AI" corrected (with the space): |
This branch has not been deployed
4 participants
Summary
This PR adds a new integration package for HiperAI SecureAI, an enterprise AI security and governance platform.
What does this integration do?
SecureAI monitors and secures enterprise AI/LLM usage. This integration receives security events and normalizes them to ECS for analysis in Elastic Security.
Event types collected:
Integration details
Resources
Checklist