High-value, low-noise, security-centric logging for custom software vendors building for Government and Enterprise SOC teams, designed to pair directly with our open-source security log normaliser and Australian privacy gateway, Bower.
Tip
Don't build a log normaliser, PII redactor, tamper-evident queue, or Sentinel arrival prover from scratch.
Use jusso-dev/Bower (Docs · Integration Guide) alongside this repo. Bower runs Collect → Select → Arrange → Deliver → Prove on-host: default-deny security policies, deterministic checksum-validated Australian PII/secret redaction (TFN, Medicare, IHI, ABN/ACN, CRN, keys/JWTs), a hash-chained SQLite WAL ledger, multi-cloud AWS/GCP/SQL Server normalisation, and bower evidence run signed Sentinel arrival bundles.
Aligned to OWASP (Logging Cheat Sheet, Application Logging Vocabulary, ASVS v4.0.3 V7 & v5.0 V16, Top 10 A09) and ACSC / ASD (Information Security Manual System Monitoring, Best Practices for Event Logging and Threat Detection, and Priority Logs for SIEM Ingestion), complete with Azure, AWS, GCP & Cross-Cloud federation guides, Diagram-as-Code architecture diagrams with official cloud icons, zero-dependency CLI scripts, drop-in language wrappers, starter SIEM detections, and portable AI coding agent skills.
- microsoft-security-logging-guide: What to log. Practical, product-by-product guidance on getting high-value security logs from Microsoft products into Microsoft Sentinel and Microsoft Defender XDR.
- soc-ready-logging (you are here): How to instrument custom applications. OWASP and ACSC ISM schemas, CLI validators, language wrappers, starter detections, and AI agent skills for high-value, low-noise security logging.
- Bower: How to normalise, redact, and ship logs. Self-hosted security telemetry normaliser and Australian privacy gateway with default-deny selection, host-side PII redaction, tamper-evident buffering, and signed Sentinel arrival proof.
- Operate on the logs in Microsoft Defender XDR and Microsoft Sentinel:
- blueboard: Shared board where AI agents and SOC analysts work incidents together across Microsoft Defender XDR and Microsoft Sentinel, with human-approved containment and eradication.
- crosswire: Open-source SOC kit for GitHub Copilot and Squad agents that hunt in Microsoft Defender XDR and Microsoft Sentinel, cross-compare threat intelligence, and propose gated responses for human approval.
Featured Open-Source Normaliser & Privacy Gateway: Bower
Need an open-source, self-hosted gateway that enforces default-deny security selection, host-side Australian PII redaction, tamper-evident buffering, multi-cloud normalisation, and signed Sentinel arrival proof out of the box?
Pair this repository with jusso-dev/Bower (Docs · Integration Guide):
- Collect → Select → Arrange → Deliver → Prove: Default-deny versioned policies reject noise before persistence; only approved semantic security events pass.
- Australian-first host-side Privacy & Secret Protection Engine: Deterministic checksum validation for TFN, Medicare, IHI, ABN/ACN, payment cards plus CRN, JWTs, cloud keys, and PEM secrets before data leaves the host, quarantining on failure and emitting
sensitive_data_detectedto your SOC. - Multi-cloud & legacy on one contract: Normalises custom apps (Bower SDK +
AGENTS.md), legacy SQL Server audit tables, Docker containers, Windows Event Logs, AWS (GuardDuty, Security Hub, high-value CloudTrail via SQS/Firehose with no stored cloud keys), and GCP (SCC & Admin Audit via Pub/Sub using WIF only, key files refused) into Microsoft Sentinel (Logs Ingestion API / AMA spool), S3, or Splunk HEC / HTTP NDJSON. - Tamper-evident ledger & IRAP/ATO proof: Hash-chained SQLite WAL queue ledger, DCR generator & drift checks, plus
bower evidence runto verify a live canary inside Sentinel and sign an ACSC ISM / Essential Eight evidence bundle.
Government SOC teams spend far too much time fighting custom applications that either:
- Flood the SIEM with 95% noise (
/healthz, static assets,DEBUGtraces, per-request JWT checks), inflating 12-month retention costs and burning out analysts; or - Miss the fundamentals (no UTC timestamps, proxy IPs instead of client IPs, free-text strings, missing authz/privilege/export events, or leaked Bearer tokens and citizen PII).
This toolkit gives vendors and government buyers one clean contract and automated tooling + AI agent skills to get it right on the first try.
Option A: Ship & Normalise Through Bower (Recommended for Gov & Sentinel)
Let Bower handle default-deny noise rejection, host-side Australian PII redaction (TFN, Medicare, IHI, ABN/ACN, CRN), tamper-evident queuing, multi-cloud AWS/GCP/SQL Server normalisation, and signed Sentinel arrival proof (bower evidence run):
git clone https://github.com/jusso-dev/Bower.git
cd Bower
# Run Bower locally (selects, redacts, queues & spools JSONL for Sentinel/AMA)
BOWER_SINGLE_TRUST_DOMAIN=true \
BOWER_QUEUE_PATH=./artifacts/bower.db \
BOWER_POLICY_DIRECTORY=./policies/default \
BOWER_OUTPUT=ama-spool \
BOWER_AMA_SPOOL_PATH=./artifacts/spool \
dotnet run --project src/Bower.CollectorSee docs/12-normalising-and-shipping-with-bower.md and the Bower Getting Started Guide for SDK, Azure Logs Ingestion API, S3, Splunk HEC, AWS bower-cloud, and GCP Pub/Sub setups.
Install the 4 portable SKILL.md skills into Claude Code, Codex, Cursor, Antigravity, GitHub Copilot, or OpenCode:
git clone https://github.com/jusso-dev/soc-ready-logging.git
cd soc-ready-logging
# Install into your vendor repository (.agents/skills, .claude/skills, .cursor/skills)
./scripts/install-skills.sh --target all-project --project-dir /path/to/your-appThen ask your coding agent inside your application repo:
- "Use
soc-logging-auditto audit this repo's logging for OWASP and ACSC SOC readiness." - "Use
soc-logging-implementto wire the two-stream SecurityLogger (and Bower integration) across all P1 security boundaries." - "Use
soc-logging-verifyto test redaction, CWE-117 injection safety, and verify Signal-to-Noise >= 90." - "Use
soc-handoff-packto generate our Day-1 Government SOC Handoff Pack, Bower evidence checklist, and starter KQL/Sigma detections."
(Using a chat assistant without skill loading? Copy-paste from templates/vendor-agent-prompt-pack.md.)
All scripts use Python 3.9+ standard library only (no pip install), so they work in restricted or air-gapped government build pipelines:
# 1. Validate a log capture against schema, OWASP taxonomy, secret/PII redaction, & CWE-117 safety
python3 scripts/validate_logs.py schemas/examples/valid-events.ndjson --strict
# 2. Score Signal-to-Noise quality (0-100) + SIEM GB estimate per 1M events
python3 scripts/score_signal_noise.py schemas/examples/valid-events.ndjson --min-score 90
# 3. Audit an application codebase for secret/PII log leaks, noise, & missing canonical events
python3 scripts/audit_repo_logging.py /path/to/your-app --markdown
# 4. Generate synthetic attack & heartbeat NDJSON events for UAT SIEM testing
python3 scripts/generate_test_events.py --app-id gov-grants-portal --scenario all --out tmp/uat-test.ndjson
# 5. Generate a tailored Day-1 Government SOC Handoff Pack
python3 scripts/generate_soc_handoff.py --app-id gov-grants-portal --app-name "Grants Portal" --from-ndjson tmp/uat-test.ndjson --out docs/SOC-HANDOFF-PACK.mdVendor hosting and agency SOC SIEMs frequently span different clouds. Dedicated guides and official-icon diagrams cover each cloud and every inter-cloud path using zero static secrets (OIDC / Workload Identity Federation / Managed Identity) and filtering before egress:
| Cloud / Scenario | Implementation Guide | Architecture Diagram |
|---|---|---|
| Microsoft Azure & Sentinel | 08: Azure & Sentinel Guide | cloud-azure-sentinel.png |
| Amazon Web Services (AWS) | 09: AWS & Security Lake Guide | cloud-aws-security-lake.png |
| Google Cloud Platform (GCP) | 10: GCP & Google SecOps Guide | cloud-gcp-secops.png |
| Cross-Cloud & Hybrid Federation | 11: Cross-Cloud Log Federation | cross-cloud-log-federation.png |
| Directory | What's Inside |
|---|---|
docs/ |
12 practical guides: Why SOC Logs Fail, Taxonomy & Schema, High-Value / Low-Noise, OWASP & ACSC Mapping, PII Redaction & Safety, SOC Handoff Playbook, Government RFP Clauses, Azure, AWS, GCP, Cross-Cloud Federation, and Normalising with Bower. |
schemas/ |
Strict Draft 2020-12 security-event.schema.json, 32-event machine-readable event-catalog.json (mapped to MITRE ATT&CK, OWASP ASVS, and ACSC ISM), plus golden & anti-pattern NDJSON fixtures. |
skills/ |
4 portable Agent Skills (soc-logging-audit, soc-logging-implement, soc-logging-verify, soc-handoff-pack) with progressive-disclosure reference sheets. |
scripts/ |
Zero-dependency Python 3 validation, signal-to-noise scoring, static repo auditing, synthetic attack generation, SOC handoff generation, and Bash skill installer. |
examples/ |
Drop-in, zero-bloat SecurityLogger reference wrappers in TypeScript, Python, C# / .NET, Go, and Java (all with emitHeartbeat parity). |
collectors/ |
Drop-in starter configs for OpenTelemetry Collector, Fluent Bit / ECS FireLens, Azure Monitor DCR (ARM), AWS CloudWatch/Firehose/S3 WORM (Terraform), and GCP Logging Sink + Locked Bucket (Terraform). |
detections/ |
Day-1 starter SOC detection rules in Sigma, Microsoft Sentinel KQL, Splunk SPL, and Google SecOps (Chronicle) YARA-L. |
diagrams/ |
Diagram-as-Code generate_diagrams.py (Python diagrams, consistent Helvetica font, official Azure/AWS/GCP icons) + all 8 rendered PNG diagrams. |
templates/ |
Fill-in-the-blank SOC Handoff Pack, UAT/ATO Acceptance Checklist, Vendor AI Prompt Pack, plus SECURITY.md. |
- OWASP: Logging Cheat Sheet, Application Logging Vocabulary Cheat Sheet, ASVS v4.0.3 (V7) & v5.0 (V16), Top 10 A09.
- ACSC / ASD: Information Security Manual (ISM) Guidelines for System Monitoring (
ISM-0580,ISM-0585,ISM-0988,ISM-1228,ISM-1405,ISM-1815,ISM-1988), Best Practices for Event Logging and Threat Detection, Priority Logs for SIEM Ingestion, and Essential Eight MFA/Privileged Access visibility. (Always verify exact ISM control wording against the live release at cyber.gov.au/ism.) - Open-Source Privacy Gateway & Normaliser: Pair this repo with jusso-dev/Bower (guide) for host-side Australian PII redaction, default-deny policy enforcement, tamper-evident queueing, AWS/GCP/SQL Server/WEC normalisation, and
bower evidence runsigned Sentinel arrival proof.
MIT (see LICENSE). Free for vendors, government agencies, integrators, and SOC teams to adopt, embed in RFPs, and adapt.




