Nightly Audit Soak Test #63
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Nightly Audit Soak Test | |
| # Long-duration sustained-load test for the v0.1.25.20 audit-write path. | |
| # Runs a failure flood (401 unauth + 403 authenticated + 400 malformed) | |
| # against a real Testcontainers Redis to catch the class of failures that | |
| # only surface under hours-scale traffic: slow memory leaks, Redis | |
| # pool exhaustion, audit-index bloat, p99 latency climb on the admin-plane | |
| # hot path, lost counter increments under sustained contention. | |
| # | |
| # Test: AuditFailureSoakIntegrationTest (@Tag("soak")) | |
| # Invariants asserted after 10 minutes at ~500 ops/s: | |
| # AS1. JVM heap end/start ratio < 2.0 (no runaway leak) | |
| # AS2. 401 mean latency < 100ms (audit write on hot path — regression guard) | |
| # AS3. written + error + sampled-out == total_requests (no lost increments) | |
| # AS4. audit:logs:_all cardinality <= written (no orphan-ZADD bug) | |
| # AS5. Network-error rate < 1% (no Redis-pool exhaustion) | |
| # | |
| # Excluded from PR CI — the soak profile has surefire groups=soak and the | |
| # default build excludes that group. Run locally with: | |
| # mvn test -Psoak --file cycles-admin-service/pom.xml \ | |
| # -pl cycles-admin-service-api -am \ | |
| # -Dtest=AuditFailureSoakIntegrationTest | |
| # | |
| # Parameters tuned for ubuntu-latest (2 vCPU, 7 GB RAM). Longer or | |
| # higher-rps soaks can be triggered manually via workflow_dispatch with | |
| # larger inputs — surefire forkedProcessTimeoutInSeconds in the pom | |
| # caps long runs at ~75 minutes; bump there if you need longer. | |
| on: | |
| schedule: | |
| - cron: '30 6 * * *' # 06:30 UTC daily (parallels cycles-server soak) | |
| workflow_dispatch: | |
| inputs: | |
| duration_minutes: | |
| description: 'Soak duration in minutes (default 10)' | |
| required: false | |
| default: '10' | |
| target_rps: | |
| description: 'Target throughput req/s (default 500)' | |
| required: false | |
| default: '500' | |
| permissions: read-all | |
| jobs: | |
| audit-soak: | |
| name: Sustained failure-flood stability | |
| runs-on: ubuntu-latest | |
| # Timeout = soak duration + generous overhead. 90 min covers default | |
| # 10-min run, 30-min dispatch variants, and up to ~60-min deep soaks. | |
| timeout-minutes: 90 | |
| permissions: | |
| contents: read | |
| steps: | |
| - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 | |
| - name: Set up JDK 21 | |
| uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 | |
| with: | |
| distribution: temurin | |
| java-version: 21 | |
| cache: maven | |
| - name: Run audit soak | |
| run: > | |
| mvn -B test | |
| --file cycles-admin-service/pom.xml | |
| -pl cycles-admin-service-api -am | |
| -Psoak | |
| -Dsoak.duration.minutes=${{ github.event.inputs.duration_minutes || '10' }} | |
| -Dsoak.target.rps=${{ github.event.inputs.target_rps || '500' }} | |
| -Dtest=AuditFailureSoakIntegrationTest | |
| -Dsurefire.failIfNoSpecifiedTests=false | |
| - name: Upload surefire reports on failure | |
| if: failure() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: surefire-reports-audit-soak | |
| path: cycles-admin-service/cycles-admin-service-api/target/surefire-reports/ | |
| retention-days: 14 |