Skip to content

fix(skills): repair update-rules skill family per full audit#214

Merged
yasirhamza merged 1 commit into
mainfrom
fix/update-rules-skill-audit
Jun 10, 2026
Merged

fix(skills): repair update-rules skill family per full audit#214
yasirhamza merged 1 commit into
mainfrom
fix/update-rules-skill-audit

Conversation

@yasirhamza

Copy link
Copy Markdown
Owner

Summary

A full audit of the update-rules skill family (12 skill files + the update-rules-e2e workflow) found 12 issues — broken paths, an invalid IOC category taught by example, impossible tool instructions, a stale validator service list, an always-true filter bug, and month-old baked-in workflow state. This PR fixes all of them, plus additional findings from the two-reviewer cycle.

Changes

update-rules.md (dispatcher)

  • Sigma repo layout corrected: production rules live in service dirs at the submodule root + staging/<service>/ — the referenced rules/production/ and rules/staging/ never existed; repo path is third-party/android-sigma-rules/
  • Missing MALWAREBAZAAR_API_KEY no longer aborts a full sweep — abusech is skipped and recorded as a feed FAILURE, remaining feeds continue (matches e2e behavior)
  • Tool note: WebFetch cannot POST or set headers; Auth-Key / GITHUB_TOKEN calls need curl
  • Step 6.5 cross-dedup explicitly covers Rule-Author-emitted ioc_data, not just ingester candidates
  • Dropped stale [Phase 4 of #117] commit-message tag

update-rules-author.md

  • TROJAN example replaced with MALWARE + the actual ioc-entry-schema.json category enum documented (no TROJAN category exists; the example was teaching a schema violation)
  • New mandatory section on the device-posture severity cap: SeverityCapPolicy clamps device_posture findings to medium at runtime, so declaring high/critical on posture rules is dead text (root cause of the recurring "pipeline proposes high" annoyance)
  • Generic lookup rules corrected to 001–004 (APK-hash 004 was omitted)
  • threat_research added to allowed sources (present in allowed-sources.json); familyNamefamily per schema

update-rules-validate.md

  • Gate 1 service check now reads the rule-schema.json enum + taxonomy status: active instead of a stale 5-service hardcode that would falsely fail valid receiver_audit/accessibility_audit/appops_audit rules
  • Gate 5 self-review documented as inline (subagents cannot spawn agents)
  • Gate count corrected to six; IOC required fields aligned with the actual schema

update-rules-ingest-abusech.md

  • "WebFetch to POST with Auth-Key header" replaced with working Bash + curl invocations (ThreatFox JSON POST, MalwareBazaar form-encoded POST)

update-rules-e2e.workflow.js (now git-tracked — was ignored, so the pipeline's orchestration was unversioned)

  • Live state is now REQUIRED via args (today, next_id, rule_index, tracked_threat_names, cursors, discover_cursors, since) with a usage-error return — replaces baked-in constants that were a month stale and pointed next_id at the retired androdr-084
  • Fixed always-true || true in the source-SIR filter: validators now receive only their candidate's SIRs (with fallback), restoring Gate 2 precision
  • Null validator results surface as failed candidates instead of silently vanishing; null IOC validation gets a logged fallback
  • De-hardcoded the author prompt's service list (taxonomy is the source of truth); schema-accurate IOC field checks; Array.isArray guard

Verification

  • Workflow body + meta parse cleanly (node --check on the harness-wrapped body)
  • Grep confirms zero residual stale strings (rules/production, 5-gate, familyName, TROJAN, WebFetch to POST, retired IDs/dates) across all 13 files
  • Discover helper test suite: 19/19 pass
  • Two independent review agents (spec-compliance + adversarial fact-checker) verified every factual claim against repo ground truth; all their findings are addressed in this PR

No app code touched — changes are limited to .claude/ skill/workflow files.

🤖 Generated with Claude Code

Audit of the 12 update-rules skills + e2e workflow found 12 issues;
this fixes all of them plus reviewer findings:

Dispatcher (update-rules.md):
- correct sigma repo layout: service dirs at root + staging/<service>/
  (rules/production and rules/staging never existed); submodule path is
  third-party/android-sigma-rules
- degraded-run policy for missing MALWAREBAZAAR_API_KEY (skip feed,
  record FAILURE, continue) instead of aborting the sweep
- WebFetch cannot POST or set headers — note curl requirement for
  Auth-Key / GITHUB_TOKEN calls
- Step 6.5 cross-dedup now covers Rule-Author-emitted ioc_data too
- drop stale "[Phase 4 of #117]" commit-message tag

Author (update-rules-author.md):
- TROJAN is not a valid category — example fixed to MALWARE, enum from
  ioc-entry-schema.json documented (bankers/RATs -> MALWARE)
- document the runtime device-posture severity cap (SeverityCapPolicy
  clamps to medium) — stop proposing high/critical for posture rules
- generic lookup rules are 001-004 (004 APK hash was omitted)
- allowed sources now include threat_research; familyName -> family

Validator (update-rules-validate.md):
- Gate 1 service check now reads rule-schema.json enum + taxonomy
  active status instead of a stale 5-service hardcode
- Gate 5 self-review runs inline (subagents cannot spawn agents)
- six-gate count corrected; required IOC fields per actual schema

abuse.ch ingester: Bash + curl invocations (ThreatFox JSON POST,
MalwareBazaar form POST) replace impossible WebFetch-POST instructions

e2e workflow (now git-tracked):
- live state required via args (today/next_id/rule_index/cursors/...)
  instead of stale baked-in constants (month-old cursors, retired
  androdr-084 next_id, hardcoded since-date)
- fix always-true source_sirs filter (|| true) so validators get only
  their candidate's SIRs; Array.isArray guard
- null validator results surface as failed candidates instead of
  silently vanishing; IOC-validation fallback object
- de-hardcode author service list; schema-accurate IOC field checks

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yasirhamza yasirhamza merged commit e0f7ea9 into main Jun 10, 2026
9 checks passed
@yasirhamza yasirhamza deleted the fix/update-rules-skill-audit branch June 10, 2026 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant