Skip to content

Foundation: fewer engine steps per run; health check reports failing vendor calls - #1247

Merged
scottbrumley merged 4 commits into
mainfrom
unified/fewer-playbook-steps
Nov 12, 2026
Merged

scottbrumley merged 4 commits into
mainfrom
unified/fewer-playbook-steps

Conversation

@scottbrumley

Copy link
Copy Markdown
Contributor

What this changes

Every EP_IR_NIST - AI run is one engine step per task, and on a busy tenant each step waits in the playbook queue. Most of a run was queueing, not work: one measured run spent 125 of 213 s in Product Classification's twelve tiny tasks. This cuts the Foundation steps a run takes and adds a way to see broken integrations.

Commit Change
Product classification in one task SOCFWClassifyProduct replaces the twelve-task chain inside Foundation - Product Classification_V3; same keys, same inference fallback
SOCFWHealthCheck reports failing vendor calls New check lists vendor calls that errored in the last 24 h, per vendor and command, from xsiam_socfw_ir_execution_raw (has_error = true): FAIL for a call that errored, WARN for a command no enabled instance serves
Upon Trigger calls its scripts directly Upon Trigger runs its five scripts as its own tasks instead of four wrapper sub-playbooks; extractIndicators over the whole issue (output unread) and Foundation - Error Handling_V3 (prints only) dropped
Upon Trigger without title tasks Titles are engine steps too; Upon Trigger is now start plus five script tasks

Steps per run: 52 → 16 with this PR alone; 12 with the companion soc-framework-nist-ir-ai PR (Assessment Decisions).

Tested on deathstar

  • Unit: 12 new tests for SOCFWClassifyProduct (mapped, product override, unmapped → inference, no tag, missing map, unparseable map, platform error, stale keys cleared, argument shapes); 4 new for the health check; every other script test in the pack passes except SOCCommandWrapper_test.py::test_main_production_rate_limited_soft_fails, which fails on main already.
  • Happy path, live: 81 runs on the new flow (Turla CrowdStrike + Proofpoint, Microsoft Defender, Abnormal), all completed, zero error entries, every one with all four decision rows.
  • Regression: issue context read from matching old-flow and new-flow runs of the same alert — email identical key for key; endpoint the same keys apart from fields the event itself carried; SOCFramework.Product identical, including response and responses, so vendor routing is unchanged. Category, normalization band and status identical across 18 alert types and 973 old runs. Wrapper rows by action / vendor / outcome / mode: no new combinations.
  • Unhappy paths, live: malicious-contained, false positive, benign-remediated, ambiguous and thin-evidence scenarios each produced the expected verdict and decision.
  • Noise: deathstar's own native traffic in the same 24 h (~1,600 issues): 1,516 Posture and 15 Low filtered out by the automation rule as before; the Security issues that reached the new flow completed cleanly.
  • Load: runs completed ~3× faster under a backlog (8.5/min, peak 17, vs 2.6/min) while the tenant was also closing 1,772 issues; no failures.

Deploying

  • Ships through the Package Manager like any soc-optimization-unified release; povkit installs it with the SOCFW base. Every script these playbooks call is in this pack. The old Foundation sub-playbooks stay in the pack, so nothing else that calls them changes.

  • Upload in a quiet window. On deathstar, issues that were queued while content was uploaded stayed at new and never started (twice today); issues created afterwards ran normally.

  • After the install, run !SOCFWHealthCheck and fix or disable any instance it reports failing — each failing call costs every run its wait.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VK24hKuKXVJfuVHg4Min9y

scottbrumley and others added 4 commits November 11, 2026 20:32
Foundation - Product Classification_V3 ran a chain of twelve playbook tasks
(getList, a condition, a DeleteContext and nine Set tasks). Each is its own
engine step; on a busy tenant each step waits in the queue, and one measured
run spent 125 of 213 seconds there. SOCFWClassifyProduct resolves the same
SOCFramework.Product keys in one script task, and still falls back to
SOCFWInferProductCategory when the map has no category.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VK24hKuKXVJfuVHg4Min9y
Every Universal Command call that errors is already written to
xsiam_socfw_ir_execution_raw with has_error = true, but nothing read it, so
a broken instance failed on every issue unnoticed (deathstar: CrowdStrike
Falcon 401 on cs-falcon-search-device, 119 times in a week). The health
check now lists failures from the last 24 hours per vendor and command:
FAIL for a call that errored, WARN for a command no enabled instance serves,
each with the last error and what to do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VK24hKuKXVJfuVHg4Min9y
Foundation - Upon Trigger V3 called four sub-playbooks that each wrapped one
script (Data Integrity, Normalize Artifacts, Enrichment, Case Sync). Every
sub-playbook adds its own engine steps, and on a busy tenant each step waits
in the queue. Upon Trigger now runs SOCFWClassifyProduct, SOCNormalizeFromList,
SOCEnrichFromList and SOCFWCaseLedgerWrite as tasks of its own.

Dropped on the way: extractIndicators over the whole issue (nothing reads its
output) and Foundation - Error Handling_V3 (it only printed errors).

Context is unchanged: the platform already passed every SOCFramework.Product
key (response, responses, Inferred) out of the separate-context sub-playbook,
so vendor routing is the same; verified by reading the issue context of
matching old and new runs.

The sub-playbooks stay in the pack for anything else that calls them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VK24hKuKXVJfuVHg4Min9y
Each title is an engine step that does nothing; on a busy tenant one waited
9.4 s in the queue. Upon Trigger is now start, then its five script tasks in
a straight line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VK24hKuKXVJfuVHg4Min9y
@scottbrumley
scottbrumley force-pushed the unified/fewer-playbook-steps branch from 6ce8257 to 6cc623b Compare November 12, 2026 01:38
@scottbrumley
scottbrumley merged commit 2c801db into main Nov 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

version:patch Bug fix or hotfix → x.x.N

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant