[aws] Migrate Security Hub findings data streams to CEL via run_as_cel by kcreddy · Pull Request #20818 · elastic/integrations

kcreddy · 2026-08-19T19:32:10Z

Proposed commit message

[aws] Migrate Security Hub findings data streams to CEL via run_as_cel

Migrate the securityhub_findings and securityhub_findings_full_posture
data streams from the httpjson input to the CEL input using run_as_cel.
The existing auth.aws configuration (static keys, session token, shared
credentials, assume role, and Identity Federation / Cloud Connectors) and
the persisted collection cursor are preserved; service_name and
default_region are added so SigV4 signing targets the Security Hub
endpoint explicitly.

httpjson re-evaluates request transforms, including now(), on every
pagination page, so the hour-truncated UpdatedAt window start could
change mid-pagination and invalidate NextToken, forcing collection to
restart with HTTP 400/403 "Invalid NextToken". The CEL program pins the
window start in state once per collection interval and only appends
NextToken between pages. This fix depends on CEL state and cannot be
made in httpjson, so it lands together with the migration rather than
as a separate bugfix.

The error path clears the pagination state so a failed page does not
replay a stale NextToken on the next interval.

Add a filters variable to both data streams so operators can apply
server-side AwsSecurityFindingFilters to GetFindings. For Findings the
managed UpdatedAt filter takes precedence; for Findings Full Posture the
variable defaults to the existing RecordState/WorkflowStatus exclusions.
A cleared value renders an empty object instead of null.

Add a max_executions variable to bound pagination per interval. Findings
keeps the input default of 1000 because it resumes incrementally via its
cursor; Findings Full Posture defaults to 10000 (about one million
findings per cycle) because it re-reads the entire current posture with
no cursor. Document the cap and how to raise it.

Changes:
- Replace httpjson.yml.hbs with httpjson_as_cel.yml.hbs for both streams
- Add filters and max_executions variables to both manifests
- Document the Findings Filters setting and CEL execution cap
- Bump package version and add changelog entries

Note

Note on bugfix/enhancement separation. Per the backport policy agreed in #20040, bugfixes should land separately from enhancements so they can be cherry-picked to backport branches. That isn't possible here: the Invalid NextToken fix depends on the migration itself. httpjson re-evaluates request.transforms on every page and has no way to pin a value computed on page 1 for later pages, so the now-derived Start can drift mid-pagination. The CEL program fixes this by holding start in state and only recomputing it when neither want_more nor page.next_token is set. Making End static was the furthest httpjson could go and was already in place.

So this PR intentionally keeps both changelog entries under one version. If the NextToken fix ever needs backporting, it means backporting the migration, which would be an exception to the bugfix-only rule and would need run_as_cel support on the target line's minimum agent version.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Migrate the securityhub_findings and securityhub_findings_full_posture data streams from the legacy httpjson input with hand-rolled SigV4 signing to the CEL input using run_as_cel with native AWS authentication (auth.aws). The previous templates signed each request in an httpjson transform and re-evaluated the request body (including now()) on every pagination page. Because the UpdatedAt window and the SigV4 signature were recomputed per page, a paginated GetFindings call could invalidate its NextToken and fail. The CEL programs pin the request body once per collection interval and only append NextToken between pages, so the window and signed request stay stable across pagination. Also add a `filters` variable to both data streams so operators can apply server-side AwsSecurityFindingFilters to GetFindings and collect only matching findings. For Findings the managed UpdatedAt filter takes precedence; for Findings Full Posture the variable defaults to the existing RecordState/WorkflowStatus exclusions. These two data streams now require Elastic Agent 9.4.0+ (or 8.19.18+ on 8.x) for the httpjson-to-CEL redirector; the aws package already requires Kibana ^9.4.0. Changes: - Replace httpjson.yml.hbs with httpjson_as_cel.yml.hbs for both streams - Add filters and max_executions variables to both manifests - Document the Findings Filters setting and version requirement - Bump package version to 7.2.0 and add a changelog entry

github-actions · 2026-08-19T19:42:39Z

Elastic Docs Style Checker (Vale)

Summary: 2 suggestions found

💡 Suggestions (2): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/aws/data_stream/securityhub_findings/manifest.yml	36	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/aws/data_stream/securityhub_findings_full_posture/manifest.yml	20	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-08-19T21:58:07Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-20T05:58:52Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copilot

Pull request overview

This PR migrates the AWS Security Hub Findings and Findings Full Posture data streams from legacy httpjson (with custom SigV4 signing) to CEL execution via run_as_cel with auth.aws, aiming to stabilize pagination and enable server-side filtering.

Changes:

Switch both Security Hub findings streams to httpjson_as_cel.yml.hbs templates using run_as_cel + auth.aws.
Add filters (AwsSecurityFindingFilters) and hidden max_executions variables to both streams.
Update Security Hub docs and bump the aws package version + changelog.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/aws/manifest.yml	Bumps the aws package version to 7.3.0.
packages/aws/changelog.yml	Adds a 7.3.0 changelog entry describing the CEL migration + filters.
packages/aws/docs/securityhub.md	Documents CEL-based GetFindings collection, incremental behavior, and Findings Filters.
packages/aws/_dev/build/docs/securityhub.md	Mirrors the Security Hub documentation updates for the built docs.
packages/aws/data_stream/securityhub_findings/manifest.yml	Switches stream template to `httpjson_as_cel.yml.hbs` and adds `filters` / `max_executions`.
packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs	Removes the legacy httpjson template with hand-rolled SigV4 signing.
packages/aws/data_stream/securityhub_findings/agent/stream/httpjson_as_cel.yml.hbs	Adds CEL program for stable UpdatedAt window + pagination with native `auth.aws`.
packages/aws/data_stream/securityhub_findings_full_posture/manifest.yml	Switches stream template to `httpjson_as_cel.yml.hbs` and adds `filters` / `max_executions`.
packages/aws/data_stream/securityhub_findings_full_posture/agent/stream/httpjson.yml.hbs	Removes the legacy httpjson template with hand-rolled SigV4 signing.
packages/aws/data_stream/securityhub_findings_full_posture/agent/stream/httpjson_as_cel.yml.hbs	Adds CEL program for pagination with native `auth.aws` and configurable server-side filters.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Migrate the securityhub_findings and securityhub_findings_full_posture data streams from the legacy httpjson input with hand-rolled SigV4 signing to the CEL input using run_as_cel with native AWS authentication (auth.aws). The previous templates signed each request in an httpjson transform and re-evaluated the request body (including now()) on every pagination page. Because the UpdatedAt window and the SigV4 signature were recomputed per page, a paginated GetFindings call could invalidate its NextToken and fail. The CEL programs pin the request body once per collection interval and only append NextToken between pages, so the window and signed request stay stable across pagination. Wire the full auth.aws credential set (access keys, session token, shared credentials, and assume role) to match the other migrated AWS streams, so the Assume Role and Shared Credentials setup options the Security Hub policy template still offers are honoured rather than silently falling back to the ambient credential chain. Reset next_token on the non-2xx error path. Without this, a failure part-way through pagination left a stale NextToken in the CEL state that was replayed on every later interval; because Security Hub pagination tokens are short-lived the request kept failing and the stream stayed wedged until the agent restarted. Add a filters variable to both data streams so operators can apply server-side AwsSecurityFindingFilters to GetFindings and collect only matching findings. For Findings the managed UpdatedAt filter takes precedence; for Findings Full Posture the variable defaults to the existing RecordState/WorkflowStatus exclusions. Guard the variable so a cleared value renders an empty object instead of null. Add a max_executions variable to both streams to bound pagination per interval. Findings keeps the input default of 1000 because it resumes incrementally via its cursor; Findings Full Posture defaults to 10000 (about one million findings per cycle) because it re-reads the entire current posture with no cursor. Document the cap and how to raise it in a troubleshooting section. Changes: - Replace httpjson.yml.hbs with httpjson_as_cel.yml.hbs for both streams - Add filters and max_executions variables to both manifests - Document the Findings Filters setting and CEL execution cap - Bump package version and add a changelog entry

Copilot

🟡 Changes recommended

The new CEL templates can emit request.tracer.enabled: null when unset and the documented/manifested max_executions constraints are inconsistent with other repo streams, both of which should be reconciled before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 14/14 changed files
Comments generated: 5
Review effort level: Lite

Copilot

🟢 Approval recommended

The migration and supporting manifest/docs/test updates look consistent, with only minor documentation wording nits noted.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

packages/aws/_dev/build/docs/securityhub.md:32

Grammar: “data streams collect from the GetFindings API” reads awkwardly; use “collect data from” to make the sentence clear.
packages/aws/docs/securityhub.md:32
Grammar: “data streams collect from the GetFindings API” reads awkwardly; use “collect data from” to make the sentence clear.

Files reviewed: 14/14 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

🔵 Needs a closer look

The updated Security Hub documentation introduces/retains incorrect guidance about required credentials, minimum Agent version, and interval-based mitigation for the full-posture stream.

Review details

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

packages/aws/_dev/build/docs/securityhub.md:31

This new note about using auth.aws conflicts with (and is undermined by) the earlier documentation that says access keys are compulsory and that the module supports Elastic Agent 8.4.0. Since these streams now use run_as_cel + auth.aws (and the package manifest requires Agent >= 9.4.0), please update the Compatibility/Credentials wording so it reflects the supported credential methods and the actual minimum agent version.
packages/aws/_dev/build/docs/securityhub.md:41
The troubleshooting advice to "shorten the collection interval so each run has fewer pages to fetch" is true for the incremental Findings stream, but it doesn't reduce paging for Findings Full Posture, which re-reads the full posture each run. This guidance should distinguish between the two streams (and optionally point users to Findings Filters for Full Posture).
packages/aws/docs/securityhub.md:31
This new note about using auth.aws conflicts with (and is undermined by) the earlier documentation that says access keys are compulsory and that the module supports Elastic Agent 8.4.0. Since these streams now use run_as_cel + auth.aws (and the package manifest requires Agent >= 9.4.0), please update the Compatibility/Credentials wording so it reflects the supported credential methods and the actual minimum agent version.
packages/aws/docs/securityhub.md:41
The troubleshooting advice to "shorten the collection interval so each run has fewer pages to fetch" is true for the incremental Findings stream, but it doesn't reduce paging for Findings Full Posture, which re-reads the full posture each run. This guidance should distinguish between the two streams (and optionally point users to Findings Filters for Full Posture).

Files reviewed: 14/14 changed files
Comments generated: 0 new
Review effort level: Lite

efd6

Some of the copilot comments look valid. After that, LGTM

Copilot

🟡 Changes recommended

Both CEL requests omit the required Content-Type: application/json header.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Files reviewed: 14/14 changed files
Comments generated: 2
Review effort level: Lite

efd6 · 2026-09-22T21:26:27Z

+          ?"NextToken": state.?page.next_token,
+        }.encode_json()
+      ).do_request().as(resp, (resp.StatusCode == 200) ?


Agree. I would recommend using post_request, then you can include the content type in the request call, rather than the approach suggested by Vera.

+        ?"NextToken": state.?page.next_token,
+      }.encode_json()
+    ).do_request().as(resp, (resp.StatusCode == 200) ?


Copilot

🔵 Needs a closer look

The authentication and pagination migration is broad and warrants final human review.

Review details

Files reviewed: 14/14 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

Copilot review overview

🟢 Approval recommended

No unresolved review issues were identified.

Review effort: Lite
Findings: 2

Open (2)

This request sends an encoded POST body without a Content-Type header. The previous httpjson…
This request sends an encoded POST body without a Content-Type header. The previous httpjson…

elastic-vault-github-plugin-prod · 2026-09-22T08:11:26Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-09-22T08:12:31Z

🟢 No issues across the latest commits bdcae55.

Review summary

Issues found across earlier commits 7ca81fd — 3 high, 3 medium, 1 low

🟠 The non-200 branch does not clear next_token, so a failure part-way through pagination leaves a stale NextToken in state that is re-sent on every later interval (link)
🟠 Same stale-token problem as the findings stream: the error branch leaves next_token set, so a failed page is retried with an expired NextToken every 24 hours (link)
🟠 The new auth.aws block only wires access keys and session token, so the Assume Role and Shared Credentials options the Security Hub policy template still offers silently produce a config with no credentials (link)
🟡 The Maximum Executions description tells users to set 0 for unlimited, but the CEL input rejects a non-positive max_executions at config validation, so following the instruction stops the input from starting (link) (Outdated)
🟡 A 1000-execution cap on the full-posture sweep silently truncates collection at roughly 100k findings per cycle, and the var is hidden so operators cannot raise it (link) (Outdated)
🟡 filters is an optional yaml var rendered unguarded, so clearing the Findings Filters field puts a null under cel.state and state.?filters.orValue({}) yields null rather than {} (link)
🔵 The new note claims these data streams need agent 9.4.0+ or 8.19.18+ on 8.x, but the package already pins conditions.agent.version and conditions.kibana.version to ^9.6.0, so the 8.x path cannot exist (link) (Outdated)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.4.2 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

kcreddy · 2026-09-22T10:10:53Z

@efd6 can I get another review on this?

infra-vault-gh-plugin-prod · 2026-09-22T10:21:19Z

💚 Build Succeeded

Buildkite Build
Commit: bdcae55

History

💚 Build #49895 succeeded da9a90b
💚 Build #49418 succeeded 8cb0fb9
💚 Build #49263 succeeded 6610ef7
💚 Build #49234 succeeded 3c0c937
💚 Build #48570 succeeded ad6bfb4
💔 Build #48561 failed efe8fbb

cc @kcreddy

efd6 · 2026-09-22T21:20:58Z

+
+## Troubleshooting
+
+### "reached maximum number of CEL executions"


I was going to say that we can avoid raising this in agent health updates by using remaining_executions conditions in the CEL, but the signal is genuinely useful here, so I don't think using that would be a good idea.

efd6 · 2026-09-22T21:26:27Z

+          ?"NextToken": state.?page.next_token,
+        }.encode_json()
+      ).do_request().as(resp, (resp.StatusCode == 200) ?


Agree. I would recommend using post_request, then you can include the content type in the request call, rather than the approach suggested by Vera.

efd6 · 2026-09-22T21:27:42Z

+        }.encode_json()
+      ).do_request().as(resp, (resp.StatusCode == 200) ?
+        resp.Body.decode_json().as(body,
+          body.?NextToken.orValue("").as(next,


Suggested change

body.?NextToken.orValue("").as(next,

body.?NextToken.as(next,

but equally (possibly better), this as binding could be dropped since we're not doing anything expensive to get it.

efd6 · 2026-09-22T21:28:09Z

+          body.?NextToken.orValue("").as(next,
+            {
+              "events": body.?Findings.orValue([]).map(e, {"message": e.encode_json()}),
+              "page": (next == "") ? {} : {"next_token": next},


Suggested change

"page": (next == "") ? {} : {"next_token": next},

"page": {?"next_token": next},

or

Suggested change

"page": (next == "") ? {} : {"next_token": next},

"page": {?"next_token": body.?NextToken},

if the binding is removed.

efd6 · 2026-09-22T21:28:23Z

+            {
+              "events": body.?Findings.orValue([]).map(e, {"message": e.encode_json()}),
+              "page": (next == "") ? {} : {"next_token": next},
+              "want_more": next != "",


Suggested change

"want_more": next != "",

"want_more": next.hasValue(),

or

Suggested change

"want_more": next != "",

"want_more": has(body.NextToken),

efd6 · 2026-09-22T21:32:26Z

@@ -0,0 +1,141 @@
+config_version: 2


Suggested change

config_version: 2

efd6 · 2026-09-22T21:32:44Z

Same comments here.

kcreddy added 3 commits August 19, 2026 00:03

update PR number

9832bee

Merge remote-tracking branch 'upstream/main' into httpjson2cel-aws-sh

7ca81fd

kcreddy self-assigned this Aug 20, 2026

kcreddy marked this pull request as ready for review August 20, 2026 05:58

kcreddy requested a review from a team as a code owner August 20, 2026 05:58

Copilot AI lite review requested due to automatic review settings August 20, 2026 05:58

kcreddy requested review from a team as code owners August 20, 2026 05:58

kcreddy added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Aug 20, 2026

kcreddy added Integration:aws AWS enhancement New feature or request labels Aug 20, 2026

Copilot started reviewing on behalf of kcreddy August 20, 2026 05:59 View session

kcreddy marked this pull request as draft August 20, 2026 05:59

Copilot AI reviewed Aug 20, 2026

View reviewed changes

vera-review-bot Bot reviewed Aug 20, 2026

View reviewed changes

kcreddy marked this pull request as ready for review August 20, 2026 07:36

Merge remote-tracking branch 'upstream/main' into httpjson2cel-aws-sh

210732d

Copilot AI review requested due to automatic review settings August 25, 2026 17:53

Copilot started reviewing on behalf of kcreddy August 25, 2026 17:53 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

Comment thread packages/aws/_dev/build/docs/securityhub.md Outdated

Comment thread packages/aws/manifest.yml Outdated

Comment thread packages/aws/docs/securityhub.md Outdated

kcreddy marked this pull request as draft August 26, 2026 07:54

kcreddy added 2 commits August 26, 2026 13:29

Merge remote-tracking branch 'upstream/main' into httpjson2cel-aws-sh

81312b3

kcreddy marked this pull request as ready for review August 26, 2026 09:58

Copilot AI review requested due to automatic review settings August 26, 2026 09:58

Copilot started reviewing on behalf of kcreddy August 26, 2026 09:58 View session

Copilot AI reviewed Sep 7, 2026

View reviewed changes

efd6 reviewed Sep 7, 2026

View reviewed changes

Address review comments

9f0527e

Copilot AI review requested due to automatic review settings September 8, 2026 05:52

Copilot started reviewing on behalf of kcreddy September 8, 2026 05:53 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

kcreddy requested a review from efd6 September 8, 2026 06:27

Address review comment

6610ef7

Copilot AI review requested due to automatic review settings September 8, 2026 07:23

Copilot started reviewing on behalf of kcreddy September 8, 2026 07:23 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

efd6 reviewed Sep 9, 2026

View reviewed changes

Address copilot comments

8cb0fb9

Copilot AI review requested due to automatic review settings September 10, 2026 07:47

Copilot started reviewing on behalf of kcreddy September 10, 2026 07:48 View session

kcreddy requested a review from efd6 September 10, 2026 07:51

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into httpjson2cel-aws-sh

da9a90b

Copilot AI review requested due to automatic review settings September 17, 2026 06:24

Copilot started reviewing on behalf of kcreddy September 17, 2026 06:25 View session

Copilot AI reviewed Sep 17, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into httpjson2cel-aws-sh

bdcae55

Copilot AI review requested due to automatic review settings September 22, 2026 08:06

Copilot started reviewing on behalf of kcreddy September 22, 2026 08:06 View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

efd6 reviewed Sep 22, 2026

View reviewed changes


		## Troubleshooting

		### "reached maximum number of CEL executions"

	body.?NextToken.orValue("").as(next,
	body.?NextToken.as(next,

	"page": (next == "") ? {} : {"next_token": next},
	"page": {?"next_token": next},

	"page": (next == "") ? {} : {"next_token": next},
	"page": {?"next_token": body.?NextToken},

Conversation

kcreddy commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Uh oh!

github-actions Bot commented Aug 19, 2026

Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Suppressed comments (2)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (4)

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

elastic-vault-github-plugin-prod Bot commented Sep 22, 2026

kcreddy commented Aug 19, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Aug 19, 2026 •

edited

Loading