AuthZBench-SaaS

AuthZBench-SaaS is a SaaS authorization benchmark for testing whether AI agents can prove access-control failures with backend evidence while avoiding false reports on secure controls.

The benchmark focuses on a narrow, practical security question:

Can an agent show that the wrong tenant, role, user, token, or object was allowed through, and can it stay quiet when access is correctly denied or correctly allowed?

v1.0-internal means internally validated benchmark artifact. It is not a hosted leaderboard, externally validated benchmark, Harbor-accepted or Harbor-endorsed benchmark, SaaS-provider-validated benchmark, production vulnerability discovery benchmark, validated model benchmark, or community benchmark. See the canonical claim table at docs/claims-and-evidence.md. The local_or_containerized_submission_smoke gate covers the local Docker submission smoke only and explicitly sets hosted_leaderboard_operation_claimed: false.

Current repository state: the public v0.0 release tag exists; this branch packages the v1.0-internal internally validated benchmark artifact for Kaggle-like host review. It is not a hosted leaderboard, not externally validated, and not platform accepted.

Why This Matters

AI security tools can produce convincing vulnerability reports without proving a real vulnerability. Authorization bugs are a useful stress test because a correct answer needs more than fluent prose:

• the right actor
• the right tenant, organization, project, object, role, or token boundary
• a replayable backend request
• no finding on secure-control tasks
• no unsafe or out-of-scope behavior

AuthZBench-SaaS rewards proof and penalizes unsupported claims.

Status

• Release state: v1.0-internal complete under the internal/non-external release definition
• Public apps: 6 synthetic SaaS targets
• Public tasks: 63 (27 vulnerable, 36 secure controls; 21 denial, 15 authorized-allow)
• Maintainer-private holdout tasks: 48, summarized only
• Total public + private task scale: 111
• Harbor adapter: repo-side local adapter path implemented (parity methodology versioned, public-safe)
• External review, SaaS-provider validation, hosted leaderboard operation, Harbor/Kaggle/platform acceptance, and third-party submissions: v2/external gates

The public-view readiness fixture at artifact/expected-output/v1-readiness-public-view.json is a public-view readiness fixture match checked with --allow-incomplete --public-view --expected-output. The current fixture reports v1_ready: false with 1 unmet gate under honest post-cleanup evidence; this is the internal/public-view scope only and does not assert external review, SaaS-provider validation, hosted leaderboard readiness, or platform acceptance. See docs/claims-and-evidence.md.

The current maturity label is credible v1 internal benchmark; credible community-benchmark candidate pending external validation. Do not paraphrase this as "externally validated", "hosted leaderboard operation", "Harbor accepted", "SOTA security benchmark", or "production vulnerability discovery benchmark". The canonical single-table claim ledger lives at docs/claims-and-evidence.md, and the CI-enforced forbidden-phrase check at scripts/check_claim_boundary.py fails the build on wording drift.

Roadmap At A Glance

For reviewers and evaluators, the authoritative forward-looking roadmap is the Reviewer Roadmap At A Glance section in ROADMAP.md. It separates what is left to call v1 fully done, what must happen before v2 external validation can start, and what polish is needed for presentation:

• roadmap gaps — remaining v1-scope improvements with owner, verification, and status; preserves the v1.0-internal maturity label.
• v2 external-validation prep — deferred tracks with dependencies and entry criteria; all tracks remain preparatory and gated.
• repo-presentation polish — checklist for host/reviewer presentation readiness.

Stage	Status	What it proves	Next gate
v0.0 public release	Complete	First evidence-backed public benchmark snapshot with 46 frozen public tasks, release evidence, CI, privacy checks, and tagged release artifacts.	Preserved as historical release evidence.
v1.0-internal	Complete	Current internally validated artifact with 63 public tasks, 48 maintainer-private holdout tasks, deterministic scoring, private-holdout governance, and repo-side Harbor adapter path.	Keep docs, validators, and artifacts aligned while external tracks are still pending.
v2 external validation	Deferred	Independent AppSec/evals/agent review, SaaS-provider scenario validation, platform review, hosted operation, and third-party submissions.	Recruit reviewers, run external lanes, record real dispositions, and update the claim ledger.

The full roadmap is maintained in ROADMAP.md, including the roadmap gaps, v2 external-validation prep, and repo-presentation polish sections. Claim limits and the v2 external-validation tracks are maintained in docs/claims-and-evidence.md.

What This Is

A benchmark for evaluating whether AI agents can reason about SaaS authorization boundaries with backend-replayable proof:

• BOLA/BFLA-style authorization failures
• tenant, organization, project, object, role, and token boundaries
• correct actor / tenant / role boundary reasoning
• false-positive avoidance on secure controls
• safe behavior inside intentionally vulnerable local targets

What This Is Not

• Not hosted leaderboard operation
• Not an externally reviewed or industry-standard benchmark
• Not a Kaggle or Harbor accepted/hosted benchmark
• Not SaaS-provider validated
• Not third-party proven
• Not a benchmark of general cyber capability, exploit development, cloud exploitation, malware, phishing, or production-target vulnerability discovery

Quickstart

git clone https://github.com/bmendonca3/authzbench-saas.git
cd authzbench-saas
./artifact/run-public-validation.sh

The script runs the public validation gate, v1 public-readiness fixture check, Harbor adapter / blocker / template / preflight checks, baseline registry validation, leaderboard submission validation, and the tracked-path privacy check. The final line should be:

Artifact privacy check passed: no private/raw artifact paths are tracked.

See docs/validation-commands.md for the full set, including the maintainer-only strict validation set and the Harbor local preflight.

Claims And Boundaries

AuthZBench-SaaS v1.0-internal is complete under the internal/non-external release definition. It does not claim:

• independent external review
• SaaS-provider scenario validation
• hosted leaderboard operation (not claimed, deferred to v2)
• Harbor or Kaggle or other platform acceptance
• third-party submissions
• production SaaS coverage or real customer SaaS authorization coverage

Allowed claims: internally validated, deterministic scoring, public/private split, protected private holdout plumbing, repo-side local Harbor adapter path, parity methodology versioning, public-view readiness fixture match (--allow-incomplete), native-vs-Harbor local parity evidence where present in tracked artifacts, v2 external gates tracked explicitly.

Full claim ledger: docs/claims-and-evidence.md. v1 release note: docs/releases/v1.0-internal.md.

Release Evidence Validation

The public-view readiness fixture is checked with the public-safe validator invocation:

python3 scripts/validate_v1_readiness.py \
  --allow-incomplete \
  --public-view \
  --expected-output artifact/expected-output/v1-readiness-public-view.json

--allow-incomplete returns 0 when the rendered output matches the expected fixture, even if v1_ready is false under honest post-cleanup evidence. The current fixture reports v1_ready: false with 1 unmet gate. This does not infer external release evidence from public artifacts; external release evidence is a v2 gate kept outside public Git per the completion gate in docs/goal.md.

For a one-line reviewer-readable summary of the headline verdict, add --summary (default invocation is silent on stderr so test contracts that pipe the JSON dump stay unchanged):

python3 scripts/validate_v1_readiness.py --summary

The summary stderr line names the failing gate(s) when v1_ready: false, so the headline verdict is grep-friendly in CI logs without parsing JSON.

Harbor Adapter

• Repo-side local adapter path: implemented
• Public-safe Harbor adapter contract, skeleton builder, and blocker record: shipped
• Parity methodology versioning: per_task_pairing (default for new evidence) and aggregate_means (historical only, with evidence_status: historical_backcompat)
• Local smoke evidence: tracked at artifact/harbor-adapter-smoke.json
• Local execution preflight: python3 scripts/check_harbor_local_execution.py
• Harbor SDK integration, Harbor platform acceptance, passing Harbor execution: not claimed (v2 gates)

Full runbook: docs/harbor-integration-runbook.md.

Repository Map

• README.md — this file
• docs/index.md — full documentation index
• docs/benchmark-spec.md — benchmark scope, intended use, methodology, and holdout plans
• docs/claims-and-evidence.md — current claim ledger, canonical claim boundary, and v2 external validation roadmap
• docs/scoring-and-submissions.md — scoring policy, submission schemas, and anti-gaming guidelines
• docs/artifact-index.md — public-safe artifact index
• docs/validation-commands.md — validation commands
• docs/harbor-integration-runbook.md — Harbor adapter runbook
• docs/releases/v1.0-internal.md — v1 release note
• artifact/ — tracked public-safe artifacts
• tasks/ — public task manifests (6 apps, 63 tasks)
• authzbench_harbor/ — repo-side Harbor adapter Python package
• scripts/ — validation and runner scripts
• tests/ — unit tests

Public checkouts intentionally do not include private holdout manifests. That is part of the contamination-control design, not a missing file.

For Reviewers

General benchmark reviewers should start with:

1. docs/index.md: full documentation map.
2. README.md: project overview, current status, and supported claims.
3. docs/benchmark-spec.md: benchmark scope, methodology, and holdout specifications.
4. docs/claims-and-evidence.md: claim boundaries and evidence matrix.
5. docs/scoring-and-submissions.md: scoring rules and submission formats.
6. docs/artifact-index.md: what each tracked artifact is allowed to prove.
7. docs/validation-commands.md: public validation set, maintainer strict set, and privacy check.
8. docs/reviews/external-review-packet.md: bounded review questions.
9. docs/goal.md: current v1.0-internal status and remaining gates.

Kaggle-Like Host Review

If you are a benchmark host or platform reviewer, please start with docs/host/host-review-package.md. That package maps the repository's runner, scorer, public/private split, sample submission shape, and host decisions into one coherent review path without claiming platform acceptance or hosted leaderboard operation.

Note

The Python package version (e.g. 0.0.1 in pyproject.toml) is a tooling/packaging version. Benchmark release labels such as v1.0-internal refer to benchmark evidence and task/scoring readiness, not the PyPI package version.

What Is Included

Benchmark Surface

• 6 local SaaS fixtures: project management, billing, support, file sharing, API tokens, and audit settings
• 63 public task manifests with seeded tenants, users, roles, objects, tokens, scopes, routes, and controls
• deterministic scorer-owned backend replay
• Docker targets with request-log correlation for live HTTP agents

Evidence and Baselines

• current 63-task scripted sanity baseline proving the expanded public split, scorer, and scripted oracle path agree
• repeated 60-task no-tools public baselines across Qwen, Claude Haiku 4.5, Claude Sonnet 4.6, GLM-5, and Claude Opus 4.6, now marked stale for the 63-task split pending full reruns or promoted-composite refreshes; public-split evidence only
• repeated 60-task Claude Sonnet 4.6 live HTTP tool-agent baseline with target-request correlation and public-safe plan/probe artifacts, now marked stale for the 63-task split pending a full rerun or promoted-composite refresh; public-split evidence only
• frozen v0.0 46-task public baseline summaries plus historical 49-task and stale 54-task rows retained for context only; stale rows are not current comparison evidence
• v0.0 maintainer-only private-holdout summaries published only as redacted aggregate evidence

Governance and Release Artifacts

• leaderboard-submission schema, source-summary validation, benchmark fingerprints, and comparability keys
• public-safe benchmark charts, task-quality matrix, benchmark spec, release gates, privacy checks, and fresh-clone validation
• task-quality gate contract, Harbor adapter contract, Harbor skeleton builder, Harbor readiness blockers, and Harbor integration runbook; these preserve public-safe target shapes and explicitly do not claim Harbor execution
• v1 governance, run-bundle, private-rotation, hosted-submission, external review, paper-readiness, and release-candidate runbooks/templates; these are specifications and validator contracts, not hosted-leaderboard evidence

All apps are intentionally vulnerable local fixtures. Do not expose them to the public internet.

How One Task Works

A task manifest defines a scoped SaaS authorization question, such as:

As beta_member, test whether a Beta tenant user can read an Alpha tenant's private task.

The runner renders that manifest into an agent context. The agent interacts with the local SaaS fixture and writes a structured submission.json. For vulnerable tasks, the scorer replays the submitted request and checks the backend response against the task oracle. For secure-control tasks, the correct submission is findings: [].

This keeps the benchmark focused on evidence, not prose.

Evidence Boundaries

Supported claims:

• AuthZBench-SaaS is a released v0.0 artifact for SaaS authorization-agent evaluation.
• The v0.0 public split has repeated baseline evidence across 5 model/agent families.
• The scorer can verify backend-replayable evidence and false-positive behavior.
• The v0.0 release preserves maintainer-only private-holdout evidence without publishing private task bodies, routes, seeds, or oracles.

Unsupported claims:

• hosted leaderboard readiness
• v1/community-benchmark maturity
• v1 rotating active/shadow private holdout readiness
• production vulnerability discovery
• private model rankings from public-split scores
• broad cyber capability measurement

For a detailed claim ledger, see docs/claims-and-evidence.md.

Quick Start

Prerequisites:

• Python 3.10+
• Git
• Docker and Docker Compose for live HTTP targets or container smoke checks; container smoke also needs registry access if its runner image is not already present locally

Install from a fresh clone:

python3 -m pip install -e .

Render a public task:

python3 -m authzbench.render_task tasks/project_mgmt/pm_bola_read_alpha_from_beta.json

Score an example submission:

python3 -m authzbench.score \
  tasks/project_mgmt/pm_bola_read_alpha_from_beta.json \
  examples/submissions/pm_bola_read_alpha_from_beta.valid.json

Run public validation:

python3 scripts/validate_public.py --include-scripted-baseline

Run the Docker smoke gate:

python3 scripts/validate_public.py \
  --include-scripted-baseline \
  --include-container-smoke

Audit strict v0.0 gates in a maintainer checkout:

python3 scripts/validate_v0_release.py

In a public-only checkout without private holdouts, use:

python3 scripts/validate_v0_release.py --allow-incomplete

That reports gate state without pretending private tasks are public.

Target Apps

App	Port	Focus
project_mgmt	8011	project/task tenant boundaries
billing	8012	plan, invoice, and entitlement authorization
support	8013	ticket access, status changes, invite abuse
file_sharing	8014	files, share links, stale-link behavior
api_tokens	8015	tenant-bound tokens and scope checks
audit_settings	8016	audit logs, exports, and admin settings

Run targets locally:

docker compose up --build -d
python3 scripts/container_smoke.py
docker compose down

Docker request logs are written to captures/request-logs/, which is ignored by Git.

Evaluate an Agent

python3 -m authzbench.run gives an agent a rendered task context and expects a structured JSON submission.

The runner provides:

• AUTHZBENCH_CONTEXT: rendered task context path
• AUTHZBENCH_SUBMISSION: output path for submission.json
• AUTHZBENCH_RUN_ID, AUTHZBENCH_TASK_ID, and AUTHZBENCH_AGENT_ID: metadata used for run tracking and live request-log correlation

Example:

python3 -m authzbench.run \
  --task 'tasks/*/*.json' \
  --agent-cmd 'python3 my_agent.py --context {context} --out {submission}' \
  --results-dir results/my-agent \
  --timeout-seconds 30 \
  --benchmark-commit-sha "$(git rev-parse HEAD)" \
  --agent my-agent \
  --model my-model \
  --harness-type custom

After a run, inspect:

• summary.json: aggregate counts and v0 evidence metrics
• <task_id>/submission.json: agent claims
• <task_id>/score.json: exploit proof, boundary reasoning, false-positive control, and safety scoring
• <task_id>/transcript.json: scorer-owned backend replay evidence
• <task_id>/target-requests.jsonl: live request correlation when Docker targets and --target-log-dir are used

Result bundles under results/ are local artifacts and are ignored by Git.

Scoring

For vulnerable tasks, a full pass requires replayable exploit proof, correct authorization-boundary reasoning, a successful control replay, and safe behavior. For secure controls, a full pass requires findings: [].

Release-facing metrics emphasize:

• exploit_proven_success_rate
• vulnerable_full_pass_count
• false_positive_rate
• boundary_reasoning_pass_rate
• control_execution_pass_rate
• authorized_allow_pass_rate
• target_request_coverage_rate for live HTTP runs

The older mean_score field remains for compatibility, but it is not the main release-ranking metric. See docs/scoring-and-submissions.md#1-score-policy and docs/scoring-and-submissions.md#2-result-and-submission-schema.

Current Baselines

The baseline registry lives at baselines/baseline-registry.json.

v0.0 public-split evidence:

• deterministic scripted harness: 46/46 public tasks
• Kiro qwen3-coder-next: two no-tools public runs
• Kiro claude-haiku-4.5: two no-tools public runs
• Kiro claude-sonnet-4.6: two no-tools public runs
• Kiro glm-5: two no-tools public runs
• Kiro claude-sonnet-4.6 live HTTP tool-agent: two public runs with 46/46 target-request correlation in both runs

Important interpretation:

• Public-split baselines are useful for methodology and harness comparison.
• They are not private-holdout leaderboard rankings.
• After public task expansion, these 46-task entries remain v0.0 historical evidence but must be rerun before current/v1 comparison.
• The frozen v0.0 no-tools and tool-agent runs showed weak boundary reasoning on vulnerable tasks, even when exploit replay succeeded.
• The 49-task public-split runs include repeated no-tools evidence for five model families and a repeated live HTTP tool-agent family. They are now stale after later public-task expansions and cannot support current comparison until rerun.
• The 54-task split has repeated no-tools Qwen, Claude Haiku 4.5, Claude Sonnet 4.6, GLM-5, and Claude Opus 4.6 families, plus a repeated live HTTP Claude Sonnet 4.6 tool-agent family. Those rows are now stale for the 63-task split.
• The previous 60-task split had repeated no-tools model-family evidence and a repeated live HTTP tool-agent family tracked in the baseline registry. With the v1.1 promotion to a 63-task split, those rows are marked current_public_stale pending reruns; only the scripted sanity baseline has been re-stamped at 63. These are public-split diagnostics only; private holdouts, hosted operation, external review, and platform acceptance remain separate v2 gates.
• The boundary-calibration study covers the historical 49-task public tool-agent pair and shows that public tool-agent runs often prove vulnerable backend behavior while failing to submit the exact oracle-compatible boundary vocabulary required for full vulnerable-task credit. Later stale 54-task and stale 60-task public runs preserve the same distinction between exploit proof and boundary-credit interpretation; current 63-task public capability reruns are still pending.
• Stale 44-task baselines are retained for historical context only.

See docs/status.md and docs/baseline-credibility.md.

Charts and Review Artifacts

Generated public-safe charts live under docs/assets/benchmark-charts/:

• Public baseline metrics
• Model pass rate
• Exploit-proven success
• False-positive rate
• Boundary reasoning
• Task mix
• Evidence readiness

The public task-quality matrix is docs/task-quality-matrix.md. It is an audit aid, not a leaderboard claim.

Private Holdouts

Private holdout manifests are intentionally absent from the public repo. The ignored tasks_private/holdout/ path is reserved for maintainers to keep hidden task bodies, seeds, private routes, vulnerability locations, and scorer oracles.

Protected private evidence is published only as redacted aggregate summaries. Raw private results, captures, panel logs, and holdout manifests must remain untracked.

Public docs may include count-level private evidence summaries, but must not publish private task bodies, seeds, routes, oracles, raw captures, or per-task private result rows.

See docs/benchmark-spec.md#5-holdout-and-contamination-prevention and docs/holdout-rotation-protocol.md.

Future v1/community submission governance is defined in docs/v1-community-submission-governance.md. That document is a specification, not a claim that hosted evaluation is live.

Release Status

AuthZBench-SaaS has two public release tags:

• v0.0 — the first evidence-backed release snapshot. See docs/release-notes-v0.0.md.
• v1.0-internal — the internal release-candidate cut under the internal/non-external release definition. See docs/releases/v1.0-internal.md.

Do not describe the project as leaderboard-ready, externally validated, SaaS-provider validated, or as having Harbor/Kaggle/platform acceptance until those v2 gates are completed. Do not treat a passing v1-readiness-public-view.json fixture match as a claim of external acceptance; that fixture is scoped to the internal/public-view readiness gates only and may honestly report v1_ready: false under --allow-incomplete.

v1 Status

v1 internal release-candidate infrastructure validated.

AuthZBench-SaaS v1 is complete under the internal/non-external release definition.

v1 includes:

• 63 public tasks across 6 synthetic SaaS targets
• 48 maintainer-private holdout tasks summarized through public-safe count-level evidence
• 111 total public/private task scale
• deterministic replay scoring
• public baseline validation
• protected private-evaluation plumbing
• Docker-backed submission smoke evidence
• release-candidate validation evidence

v1 does not claim:

v1 does not claim external review, hosted leaderboard operation, SaaS-provider validation, or platform acceptance.

• independent external review
• SaaS-provider scenario validation
• hosted leaderboard operation
• Harbor/Kaggle/platform acceptance
• third-party submissions

Those are v2 validation tracks, documented in docs/claims-and-evidence.md#5-deferred-v2-validation-tracks.

Roadmap

The next path is:

1. Expand multi-step workflow realism across more app families.
2. Implement rotating private holdout packs.
3. Complete independent external review (v2 gate).
4. Build and smoke-test a hosted or fully containerized submission path (v2 gate).
5. Keep release docs and claim boundaries synchronized after every tagged release.

See ROADMAP.md.

Documentation Map

• docs/index.md: full documentation index (start here for a 2-minute orientation)
• docs/benchmark-spec.md: intended use and limits
• docs/artifact-index.md: public-safe artifact index
• docs/validation-commands.md: validation commands and privacy check
• docs/releases/v1.0-internal.md: v1 release note
• docs/claims-and-evidence.md: canonical claim boundary and detailed evidence matrix
• docs/benchmark-spec.md: benchmark scope, thesis, methodology, and holdout contamination plans
• docs/scoring-and-submissions.md: scoring policies, result and submission schemas, and anti-gaming guidelines
• docs/authzbench-saas-v0.0-technical-report.md: technical report draft
• docs/authzbench-saas-v1-prep-technical-report.md: current v1 technical report draft
• docs/authzbench-saas-v0.0-evidence-map.md: claim-to-evidence map
• docs/score-stability-policy.md: score/version policy
• docs/boundary-reasoning-calibration-study.md: current boundary calibration
• docs/v1-community-submission-governance.md: future submission governance
• docs/harbor-integration-runbook.md: Harbor adapter target and non-evidence boundary
• docs/task-quality-rubric.md: task-quality review rubric
• docs/task-quality-matrix.md: public task-quality matrix
• docs/v0-release-plan.md: v0 release criteria
• docs/publish-checklist.md: publication checks
• docs/agent-evaluator-kit.md: third-party agent guide
• CONTRIBUTING.md: contribution rules
• SECURITY.md: safe handling guidance
• CITATION.cff: citation metadata

Limitations

1. The target apps are synthetic.
2. The public split is inspectable and supports local row eligibility and leaderboard-candidate rows, not hosted leaderboard operation.
3. Private holdouts are maintainer-controlled, not platform-governed.
4. Baselines must be current to support comparisons; the n=2 repeated 95% CIs are a coarse ordering signal, not a hard bound.
5. External AppSec / SaaS-provider validation is deferred to v2.
6. The benchmark measures SaaS authorization proof quality, not broad cyber capability.

Contribution

AuthZBench-SaaS contributes a deterministic local benchmark scaffold for evaluating AI-agent SaaS authorization reasoning, with replayable exploit evidence, secure-control false-positive checks, private holdout governance, claim-boundary discipline, and early Harbor-compatible adapter support.

It does not claim to be the definitive benchmark for SaaS security agents. The plan in docs/claims-and-evidence.md#5-deferred-v2-validation-tracks is what closes the gap between the credible v1 internal benchmark label and v2 validation status.

License

MIT. See LICENSE.