feat(governance): Kravalg-gated multi-repo IAM/OIDC governance stack (#77)#78
Open
dmytrocraft wants to merge 39 commits into
Open
feat(governance): Kravalg-gated multi-repo IAM/OIDC governance stack (#77)#78dmytrocraft wants to merge 39 commits into
dmytrocraft wants to merge 39 commits into
Conversation
Refactor _pulumi_secrets_alias_conditions to the repo/env-parametric
signature (repo, env, *, include_platform_bootstrap) so the governance
service-repo loop (include_platform_bootstrap=False) gets ONLY
alias/pulumi-{repo}-{env}-secrets and can never decrypt the platform
master key (AWS-SRE-1, FR3). The single-repo bootstrap entrypoint passes
include_platform_bootstrap=True to retain alias/pulumi-platform-bootstrap-{env}
for its own backend. The pre-refactor alias/pulumi-*-{env}-secrets
wildcard is replaced by the repo-scoped alias either way.
Add the additive BootstrapSettings.secrets_alias_for_repo(repo, env)
convenience (explicit env) and re-express pulumi_secrets_alias_name_for_repo
in terms of it. Helper layer only; no component wiring yet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntext Capture the golden parity fixture of the current single-repo bootstrap-infrastructure render (deploy/config-read/triage role names, per-role trust JSON, backend/read-only/apply policy JSON incl. the _automation_policy_documents apply branch, and the Repository tag) under tests/unit/fixtures + tests/unit/test_governance_parity.py. This is the NFR6 gating artifact for E1.S2-E1.S4b (FEASIBILITY-4). Extend the existing _BootstrapBuildContext with repo/project fields (single source of truth; project = sanitize_bucket_component(repo) full slug, AWS-SRE-4 — no parallel _RepoCiContext) and thread repo/project through _ci_role_name, _repo_subject, _deployment_role_subjects, _state_bucket_resources, _pulumi_backend_policy_document, _role_policy_documents, _role_specs, _create_role (now reading context.repo/context.project), and _create_roles. GitHubCiBootstrap builds one context from settings.repo. Defaults derive from settings so the single-repo render stays byte-identical to the golden fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…1.S3)
Add repo:str|None to CiConfigurationArgs (None preserves single-repo
behaviour byte-for-byte, NFR6) and thread it through the CI-config helpers
so secret IDs, trust subjects and the config-read Allow ARN become
repo-scoped when the governance loop supplies an explicit repo.
Harden the secret-leaking-read surface (FR22, D4, SECURITY-5, §5.2a/5.3/5.4):
- config-read policy gains a second DenySecretLeakingReads statement that
EXCLUDES secretsmanager:GetSecretValue (Deny would void the role's own
repo-scoped Allow) but denies kms:Decrypt + the other leaky reads.
- read-only policy (preview/drift) replaces the single-action Deny with the
FULL DenySecretLeakingReads block INCLUDING secretsmanager:GetSecretValue
and kms:Decrypt (these roles never decrypt a Pulumi secrets key).
- apply role gains a surgical DenySecretLeakingReadsApply managed policy with
NotResource carve-out for its own /{project}/ci/* secrets and WITHOUT
kms:Decrypt (apply legitimately decrypts its key, bounded by the §5.2 alias
condition). Not attached to the pulumi-backend policy, which keeps its
kms:Decrypt Allow and carries no Deny.
Every new statement is Effect=Deny, so it is CrossGuard-exempt and make
test-policy stays green. The golden parity fixture is regenerated for ONLY
the read-only Deny and the new apply Deny doc; role names, trust JSON,
backend policy, Repository tag and config-read role names stay byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ject override
Create pulumi/infra/governance.py part 1 (Story 1.4 / E1.S4a): the per-repo
half of the multi-repo governance stack. Adds GovernanceStackArgs (incl.
oidc_provider_arn consumed by .get() and an injectable region so policy/secret
ARNs render under the mock account/region), and the RepoGovernance
ComponentResource (type token bootstrap:governance:RepoGovernance) that wires,
for one managed *-infrastructure repo:
- per-repo Pulumi state bucket + replica via PulumiStateBuckets (single-repo
catalog, FR5);
- per-repo KMS key + alias via PulumiSecretsKeys (FR6);
- per-repo CI-config secret(s) + config-read role(s) via CiConfiguration with
the repo override (FR4);
- the preview/apply/drift deploy trio via the lifted _create_roles/_role_specs
(FR2/FR3), repo-scoped to THIS repo's bucket ARN + KMS alias only.
Security-load-bearing pieces:
- _governance_apply_subjects(settings, repo) returns ONLY
[repo:org/repo:environment:governance] for the apply role on BOTH the test
and prod stacks (SECURITY-2, §5.1a), binding the IAM trust to the
@Kravalg-gated GitHub environment; preview/drift keep their existing subjects.
- _governance_backend_policy_document renders the §5.2 deploy/backend policy:
region-pinned KMS Resource (no region wildcard, SECURITY-KMS-medium) and the
repo's own alias ONLY -- the pulumi-platform-bootstrap alias is gone
(AWS-SRE-1, FR3). No Effect:Allow wildcard outside the CrossGuard-exempt
allowlist (FR23). All ARNs interpolate {account_id}/{region}, never a literal
891377212104/eu-central-1 (FEAS-3).
- _governance_payloads mirrors ci_bootstrap._payloads WITHOUT the
operations-triage requirement (§3.5); backend URL and secrets provider are
repo-scoped.
Helpers are decomposed to keep complexity well under the ruff/xenon thresholds.
Tests (tests/unit/test_governance_repo_component.py) cover the positive,
negative, and edge cases for one synthetic repo under the conftest Pulumi mocks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the stack half of governance.py (Story 1.5 / E1.S4b): GovernanceStack (type token bootstrap:governance:GovernanceStack) consumes its account's GitHub OIDC provider by pinned ARN via OpenIdConnectProvider.get (zero create branch, AWS-SRE-2/FR7), raising if oidc_provider_arn is unset so it can never fall back to a create that would race or mutate the bootstrap stack's state. Assert the live account against the injectable, per-stack expected_account_id (D1): the literal lives only in stack config, so the injectable seam lets tests drive both the match (mock account 123456789012 -> proceeds) and mismatch (-> ValueError) branches under the Pulumi mocks. region stays injectable so ARNs/secrets providers render under the mock region with no hardcoded eu-central-1/account literal in component code. Loop one RepoGovernance per catalog repo (suffix via _repo_suffix), then register the stable per-repo outputs map (§3.4: stateBucketName, stateBackendUrl, secretsAlias, secretsProvider, deploymentRoleArns, configReadRoleArns, ciConfigSecretIds, githubVariables) plus sorted managedRepositories and the consumed oidcProviderArn. Export GovernanceStack, GovernanceStackArgs and RepoGovernance from pulumi/infra/__init__.py. Tests: a 2-repo synthetic catalog yields exactly 3N deploy roles, zero OpenIdConnectProvider create (only .get adopt), both account-assertion branches, the unset-ARN raise, and the §3.4 output shape / repo isolation / parametricity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…AM/OIDC governance) Research, brief, PRD, architecture, epics, implementation-readiness, and a Well-Architected review for the Kravalg-gated multi-repo governance stack. Two-account model (test 891377212104 / prod 933245420672). Planned with BMAD, adversarially critiqued (PASS), kept under specs/ per repo convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t/prod)
Add the thin pulumi/governance Pulumi project (E1.S5) in parity with
pulumi/github-ci-bootstrap: a name=governance manifest, per-stack
test/prod config pinning each stack to its OWN account (test ->
891377212104, prod -> 933245420672) in eu-central-1, consuming that
account's OIDC provider ARN by .get() and its awskms:// secrets
provider, a non-discovered example stack, requirements.txt, and a thin
__main__.py that resolves BootstrapSettings + ManagedRepositoryCatalog
from config, builds GovernanceStack(GovernanceStackArgs(...,
expected_account_id=cfg.get("awsAccountId"),
oidc_provider_arn=cfg.get("githubOidcProviderArn"), region=...)) and
exports perRepo/oidcProviderArn/managedRepositories.
Account-number literals live only in the per-stack config (never in
*.py). The repositoryCatalogPath points at ../repositories.governance.json
(added in E1.S6); bootstrap-infrastructure is intentionally never listed
to avoid double-managing its IAM roles. A focused scaffold test asserts
the manifest name, two-account pinning, awskms:// provider, example
non-discovery, secret-safe stack files, and the entrypoint wiring without
breaking the existing 25 structural tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… truth Add pulumi/repositories.governance.json as the active governance catalog (user-service-infrastructure only; bootstrap-infrastructure deliberately EXCLUDED to avoid double-managing the same IAM roles/secret it already owns via github-ci-bootstrap). project is the full slug user-service-infrastructure to match the canonical naming source. Validated by the existing repositories.schema.json with no schema change; auto-discovered by validate_repository_catalogs.py via its repositories*.json glob. Add scripts/governance_paths.py holding GOVERNANCE_PATH_GLOBS, the single authoritative copy of the architecture 7.1 / CODEOWNERS \@Kravalg glob set covering ALL credential-bearing / trust-or-scope-altering code. The tuple is byte- and order-equal to the architecture list so the later CODEOWNERS drift test stays green. paths_touch_governance(files) uses fnmatch over repo-relative paths (directory globs match the dir and its subtree), and a --files-stdin CLI prints governance_touched=true|false for the intake scope step and the governance runner's server-side recompute. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend scripts/validate_repository_catalogs.py with a separate governance fanout (AWS-SRE-6) so repositories.governance.json validates with governance- specific counts without relaxing the shared deployment-catalog guard: - Detect catalog kind by filename (repositories.governance.json -> governance, else deployment/central); governance fanout creates NO central-stack resources and scales linearly per repo: iamRoles = 3 trio + 4 config-read + 1 replication = 8; managedPolicies = 7 (apply role); secrets = 4 (one per CI suffix). PER_ENVIRONMENT_FANOUT / CENTRAL_STACK_FANOUT are untouched, so repositories.bootstrap.json keeps its unchanged deployment fanout. - Add an account-quota headroom report (1000 roles, 1500 managed policies, 10 managed-policies-per-role) that flags the apply role's fixed 7/10 per-role managed-policy usage. - Add a governance unique-project guard (Section 4) and a role-name-length guard (FEASIBILITY-4): reject any repo whose sanitize_bucket_component(name) + '-prod-preview' would exceed 64 chars at validation time, not apply time. - Governance catalogs print governance fanout + quota headroom with their own thresholds (--max-governance-*), kept separate from the deployment guard. Tests cover governance fanout within thresholds, linear single-repo scaling, duplicate-project rejection, over-64-char prod-preview rejection, the sanitize /role-name helpers, and that the deployment fanout stays unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Governance component tests iterated the session-scoped pulumi_mocks.resources and matched the first resource by Sid/type, so they passed in isolation but failed when prior tests populated the shared list. Snapshot the resource count before building and scan only resources created by the test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vidence The Well-Architected repository-fanout evidence applies deployment thresholds to every discovered catalog. The new repositories.governance.json intentionally exceeds them (preview/apply/drift trio + config-read + replication roles per repo) and is validated separately by validate_repository_catalogs.py with a governance-specific fanout, so it is excluded here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add .github/CODEOWNERS scoping every governance/IAM/policy/trust-or-scope glob to sole approver @Kravalg, with no catch-all '*' line so unrelated paths stay unowned (D5, no over-scoping). The @Kravalg glob block is derived from and kept byte/set-equal to scripts/governance_paths.GOVERNANCE_PATH_GLOBS (the single source of truth, SECURITY-4). Add tests/unit/test_codeowners.py asserting: drift set-equality both directions between the parsed @Kravalg globs and GOVERNANCE_PATH_GLOBS; every governance glob resolves to @Kravalg (case-insensitive); unrelated docs/app/test paths resolve to no owner; no catch-all rule; and .github/CODEOWNERS itself is owned by @Kravalg (FR10, D5, SECURITY-4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…harden PR rule Add GOVERNANCE_ENVIRONMENT, governance_environment_payload (reuses the shared protected-reviewer helper: sole @Kravalg, prevent_self_review, protected-branch-only), and governance_environment_verification_blockers to _github_repository_controls.py. Wire configure_github_repository_controls.py to emit payloads[governanceEnvironment] in dry-run and apply branches, PUT repos/{repo}/environments/governance on apply, and add a governance blocker plus reviewer/env reporting to _verify_applied_controls. Harden default_pull_request_rule(): dismiss_stale_reviews_on_push=True and require_last_push_approval=True so any push after approval dismisses the stale code-owner review and re-requires fresh @Kravalg approval, binding approval to the exact approved diff (SECURITY-3, architecture 7.6). FR11. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add .github/workflows/pulumi-governance.yml: a trusted, dedicated apply
path keyed ONLY to repository_dispatch type pulumi-governance-command
(no workflow_dispatch, closing the SECURITY-1 manual-escalation gap).
The preflight job (no id-token) treats every client_payload field as
untrusted: it re-validates PR number / head SHA / open-not-merged /
same-repo head, resolves client_payload.comment_id to its author via
gh api .../issues/comments/{id} and requires login==Kravalg
(case-insensitive), and recomputes governance scope server-side from the
verified head SHA changed files through scripts/governance_paths.py,
rejecting non-governance requests so a non-governance PR cannot mint
governance credentials.
Both governance_test_apply and governance_prod_apply declare a static
environment: governance with PULUMI_DIR=pulumi/governance and run
make pulumi-up-plan (saved plan, never direct up), so the @Kravalg
reviewer gate fires before either the test or prod apply mutates AWS
(SECURITY-2). prod apply needs test apply + test post-apply drift,
preserving test->prod ordering (FR15). A final always() status job posts
the byte-identical context "Governance Apply" to the approved head SHA
with state from the prod apply outcome, providing the
success-before-merge gate (FEASIBILITY-1).
Add tests/pulumi/test_governance_runner_workflow.py: a parsed-YAML
workflow-lint suite asserting the dedicated trigger, absence of
workflow_dispatch, preflight re-auth + Kravalg assertion + server-side
scope recompute, both apply jobs env-gated on governance with the
governance PULUMI_DIR and stacks, no direct pulumi-up, prod-needs-test
ordering, and the head-SHA status post with the exact context string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unner status-post
Append the "Governance Apply" context to REQUIRED_STATUS_CHECKS so the
@Kravalg-approved governance apply becomes a success-before-merge gate. The
governance runner fires on repository_dispatch and cannot natively report a
check, so it explicitly posts this byte-identical context to the approved head
SHA via gh api .../statuses/{head_sha}; the required check resolves only because
the tuple entry and the runner status-post agree (FR15, FEASIBILITY-1,
architecture §7.5). Tests assert both tuple membership and the runner's posted
context, plus no regression to the existing required-check set. The doc/spec
required-check contract (ci-guardrails, ci-quality-gates, operating-evidence,
readiness report, external-control evidence) is synced so the cross-cutting
doc-sync contract test stays green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add KRAVALG_LOGIN and extend author_is_authorized with keyword-only author_login/governance_touched/action params: governance up is accepted only when the comment author login is @Kravalg (case-insensitive); all other paths (plan, non-governance) keep the existing AUTHORIZED_ASSOCIATIONS rule. Add --author-login and --governance-touched CLI flags and thread command.action plus the flags through build_outputs. This intake gate is defense-in-depth; the governance runner's server-side re-auth (E1.S8) is the hard control. New tests/unit/test_pr_comment_gate.py covers the full decision matrix (E3.S1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Detect governance-touching changes step to the pulumi-pr-commands.yml
intake that lists the PR's changed files (gh api .../pulls/{n}/files
--paginate) and pipes them through scripts/governance_paths.py
--files-stdin to compute governance_touched, then passes --author-login and
--governance-touched into the parse/authorize step. When governance is
touched the intake dispatches the dedicated event type
pulumi-governance-command (to pulumi-governance.yml); otherwise it keeps
pulumi-pr-command for the existing runner. Both dispatches carry comment_id,
pull_request_number, and head_sha so the governance runner re-resolves the
author and recomputes scope server-side (E3.S2, FR14, architecture 7.2/7.4).
The intake author/path gate is defense-in-depth; the governance runner's
server-side re-auth is the hard control. Adds a parsed-YAML structural test
asserting the scope step, the parse-step inputs, the conditional event-type
dispatch, and comment_id in client_payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E3.S3: add a workflow-graph regression test pinning that the monolithic pulumi-pr-command-runner.yml listens ONLY for the pulumi-pr-command repository_dispatch event (never pulumi-governance-command), owns no governance apply job or environment: governance gate, and keeps every prod_* job gated on test_* success (FR15 success-before-merge). Also asserts governance up is handled exclusively by pulumi-governance.yml. Routing to the governance apply path is done at the intake via the dedicated event type (E3.S2), so the runner needs no code change; this story is the tests-only regression guard against that split eroding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the in-repo template assets for the managed service repo
user-service-infrastructure under pulumi/user-service-infrastructure/
(E5.S1): the Pulumi project skeleton (manifest, test/prod/example stacks,
thin __main__.py), plus repo-local AGENTS.md and README.
The scaffold CONSUMES, never creates, the governance-provided state bucket
(s3://pulumi-user-service-infrastructure-{env}-state), the KMS secrets alias
(alias/pulumi-user-service-infrastructure-{env}-secrets), and the
GitHubCi{Preview,Apply,Drift}/ConfigRead-user-service-infrastructure-*
deploy/config-read roles; it creates no IAM roles or OIDC trust for itself.
Two accounts (test 891377212104 / prod 933245420672), region eu-central-1,
account literals only in stack config. No static AWS keys, no
AdministratorAccess, no inline secrets/passphrase.
Tests assert STRUCTURE ONLY (no pulumi preview, which is blocked until the
operator applies governance + sets repo vars): asset presence, governance
backend/alias references, no self-created IAM, two-account model, example
stack non-discovery, and byte-equal name parity of the referenced governance
role names against the rendered governance helper output (single source of
truth). The nested scaffold is not picked up by top-level Pulumi project or
catalog discovery; tests/unit and tests/pulumi stay all-passing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the in-repo self-deploy.yml template asset (E5.S2) for the managed user-service-infrastructure repo: PR-comment driven preflight -> test plan -> test apply (saved-plan) -> prod, with AWS credentials sourced ONLY via GitHub OIDC role assumption through ./.github/actions/load-aws-ci-env (config-role-arn from vars.AWS_TEST_CI_CONFIG_ROLE_ARN = GitHubCiConfigRead-user-service- infrastructure-test; role-to-assume = the loaded apply-role GitHubCiApply-user-service-infrastructure-test). Applies route through the IaC-only saved-plan path (make pulumi-up-plan), never direct make pulumi-up; no static AWS keys and no admin-managed-policy grant. The scaffold consumes the governance-owned roles, it never creates IAM/OIDC trust for itself. Add tests/pulumi/test_self_deploy_template.py: structure-only lint (no pulumi preview) asserting OIDC-only role assumption with id-token: write on every credential job, saved-plan apply path, the test->prod ordering, a gitleaks-style no-secret/no-static-key/no-admin scan, and a NAME-PARITY test pinning every GitHubCi*-user-service-infrastructure-* role name the template references to the names the governance helpers render (single source of truth) — a user-service short-name reference fails the test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Red-team proved a non-@Kravalg reviewer could approve changes to scripts that run inside the governance apply with AWS credentials, because these paths were outside GOVERNANCE_PATH_GLOBS and therefore tripped neither CODEOWNERS nor the server-side governance_touched recompute: - scripts/_pulumi_command_support.py (imported by run_pulumi_command.py) - scripts/_script_support.py (imported by run_pulumi_command.py) - scripts/prepare_policy_pack.py (builds the CrossGuard policy pack) Add all three to GOVERNANCE_PATH_GLOBS and the matching @Kravalg lines to .github/CODEOWNERS, keeping the CODEOWNERS<->globs drift-equality test green. Also pre-register the new merge-gate workflow glob .github/workflows/governance-apply-status.yml (file added in the next commit) so the two lists stay set-equal at every commit. Defensive (FIX C): normalize each input path with posixpath.normpath before fnmatch so a traversal-style path such as pulumi/governance/../infra/x.py resolves to pulumi/infra/x.py and matches the correct glob instead of spuriously matching pulumi/governance/*. Tests: prove the three apply-context support paths and the status workflow now return True from paths_touch_governance, and that a ..-containing path resolves to its true target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
"Governance Apply" is a GLOBAL required status check but was only ever
posted by the governance runner, which fires only on governance-touching
PRs. Consequence (red-team CRITICAL): NON-governance PRs never received a
"Governance Apply" status and were permanently unmergeable.
Add .github/workflows/governance-apply-status.yml: on every PR head SHA
(opened/synchronize/reopened/ready_for_review) it recomputes governance
scope from the changed-file LIST and posts a terminal/placeholder status:
- non-governance -> state=success (gate not required), required check met
- governance -> state=pending until the runner posts success after
the gated test+prod apply
CRITICAL SAFETY: pull_request_target gives the GITHUB_TOKEN
statuses: write even for fork PRs, but the workflow NEVER checks out or
executes PR-head code. It checks out only the BASE repo (no ref => base
branch) and reads the PR's changed-file list as DATA via the API. No
id-token, no AWS. PR_NUMBER/HEAD_SHA flow through env (no inline
interpolation in run:). Concurrency is keyed per head SHA. actionlint +
yamllint clean.
The workflow's own glob is already Kravalg-gated (registered in the prior
commit) so a reviewer cannot weaken the merge-gate semantics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (B2)
Red-team CRITICAL+HIGH: the governance runner could strand the
"Governance Apply" required check pending.
(1) The preflight emitted head_sha/pull_request_number to GITHUB_OUTPUT
only at the END, after the auth/scope/head-moved exit-1 checks. A
rejected governance PR therefore left the status job with empty
inputs and the required check stuck pending. Move the emission to
occur right after FORMAT validation (numeric PR + 40-char lowercase
SHA) but BEFORE any auth/scope/head-moved rejection, so a rejection
still leaves the outputs populated. (A malformed SHA still exits
early — a status cannot be posted to a bad SHA anyway.)
(2) The status job mapped skipped/cancelled/'' to 'pending', leaving the
check stuck. Make the result mapping terminal:
success) state=success ;; *) state=failure ;;
so failure/skipped/cancelled/empty all post state=failure with
"Governance apply rejected or did not complete." The post is skipped
only when head_sha is genuinely empty (un-postable malformed input).
The success path is unchanged: state=success only when the prod apply
succeeded, posted to the approved head SHA (success-before-merge).
Tests: preflight emits outputs before the auth/scope rejections; status
job is if:always() and maps every non-success result to failure with no
pending branch; the post guard is only-empty-head_sha. actionlint +
yamllint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s authoritative (C) Defense-in-depth clarification (red-team MEDIUM, doc-only): expand the intake "Detect governance-touching changes" comment to state explicitly that the issue_comment intake governance_touched flag is advisory and NEVER the security boundary. The authoritative control is the governance runner's server-side recompute (pulumi-governance.yml preflight), which re-resolves the comment author to @Kravalg and recomputes the predicate from the verified head SHA before any credential-minting job; if the two disagree, the runner wins. The companion path-normalization half of FIX C (posixpath.normpath before fnmatch in scripts/governance_paths.py, with its ..-traversal test) landed in the missing-glob commit since it is the same file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the PR-A / PR-B / create-repo / PR-C onboarding sequence for a new X-infrastructure service in AGENTS.md, each step tagged CODE or OPERATOR, with the gated /pulumi test up and /pulumi prod up commands, the config-only repositories.governance.json contract, two-account (test 891377212104 / prod 933245420672) isolation, @Kravalg sole-approver gating, the self-deploy hand-off, and the Governance Apply required-check semantics. Add tests/pulumi/test_onboarding_docs.py asserting the four-step sequence, CODE/OPERATOR labels, governance command and catalog references, and the preserved BMAD/BMALPH specs/ phrases. The existing required specs/ phrases are untouched (structural test still passes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add docs/governance-stack.md: a design overview (two-account 891377212104/933245420672 separate-account isolation, per-account OIDC provider consumed-not-created, sole approver @Kravalg, IaC-only saved-plan applies, config-only onboarding) plus the architecture §10.2 operator runbook. Each operator action is tagged OPERATOR: one-time local bootstrap apply (hardware-MFA, test stack then prod stack with exact pulumi -C pulumi/governance commands + per-account secrets providers), per-account OIDC-ARN pin before first apply (stack raises if unset), cost-anomaly ARN-matches-stack verification (no prod repoint off 933245420672), protected governance environment + branch protection via configure_github_repository_controls.py --apply, repo variables from githubVariables, user-service-infrastructure create + scaffold push, gated real applies via @Kravalg PR comments, and audited break-glass. Add tests/pulumi/test_governance_account_model.py: assert no account-number literal is hardcoded as a runtime value in any pulumi/infra/*.py (AST-based, docstrings exempt), the test stack config pins 891377212104 and the prod config pins 933245420672 (each its own account only), and that costAnomalyMonitorArn (if present) matches its stack. Add tests/pulumi/test_governance_runbook_docs.py: assert the runbook is linked from AGENTS.md and enumerates each OPERATOR step (incl. per-account OIDC pin, cost-anomaly-matches-stack, break-glass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover _governance_backend_url/_governance_secrets_provider override-None defaults, the non-test/prod payload fallback, the repo-settings rescope branch, the normpath empty/dot guard, and the catalog validator's malformed-item/non-list/quota-alias branches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add infra to import-linter root_packages and a forbidden contract keeping infra.governance free of the policy pack and Pulumi app layer (11 contracts kept, 0 broken). Add an AST guard for the scripts half, which is intentionally not part of the import-linter graph. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![qltysh[bot]](https://avatars.githubusercontent.com/in/890766?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 68 files
Confidence score: 1/5
- The highest-risk issue is in
.github/workflows/pulumi-governance.yml: the runner ignores requested command/environment and always executes apply stages, so a/pulumi ... planor test invocation could still mutate AWS or promote to prod—fix command/target routing and add a guard test before merging. pulumi/governance/__main__.py(wrong default project directory) plus payload drift inpulumi/infra/governance.py(PULUMI_PREVIEW_STACKSdropped for prod) can generate invalid CI inputs and break downstream workflows unexpectedly—restore the expected directory/payload contract and run an end-to-end governance dispatch check.- The credential flow appears under-specified in
specs/issue-77-multi-repo-governance/architecture.md, and current assumptions about bootstrap CI-config roles may not hold for governance apply; merged as-is, apply jobs may fail or run with incorrect access—document and implement the exact role/credential path and validate it in the target repo workflow. - Two runtime-failure paths remain:
scripts/validate_repository_catalogs.pycan miss overlong IAM role names (passing precheck but failing on apply), andpulumi/infra/ci_bootstrap.pydenieskms:Decryptfor preview/drift roles (breaking stacks with encrypted secrets); additionally,.github/workflows/governance-apply-status.ymlcan fail to post required status checks ongh apierrors—tighten validation limits, correct IAM policy precedence, and add retry/error handling for status posting before merge.
Architecture diagram
sequenceDiagram
participant User as GitHub User
participant GH as GitHub API/Workflows
participant Intake as PR Command Intake
participant ScopeDetect as Scope Detector
participant AuthGate as Author Gate
participant GovRunner as Governance Runner
participant GovWF as Governance Workflow
participant Preflight as Preflight Job
participant GovPlan as Governance Plan Job
participant GovApply as Governance Apply Job
participant GovStatus as Governance Status
participant StatusWF as Governance Apply Status
participant AWS as AWS (IAM/OIDC/KMS/S3)
participant Pulumi as Pulumi Governance Stack
participant Config as Config Catalog
Note over User,Config: Multi-Repo Governance IAM/OIDC Stack
User->>GH: Opens PR with governance changes
GH->>Intake: pull_request_target trigger
Intake->>ScopeDetect: Compute governance_touched from PR files
Note over ScopeDetect: Uses governance_paths.py (single source of truth)
ScopeDetect-->>Intake: governance_touched=true
Intake->>AuthGate: Parse comment with --author-login, --governance-touched
alt Governance + up command
AuthGate->>AuthGate: Authorized only if Kravalg
else Plan command or non-governance
AuthGate->>AuthGate: Uses existing association rule
end
AuthGate-->>Intake: authorized status
alt Governance PR + authorized
Intake->>GH: repository_dispatch(pulumi-governance-command)
Note over Intake,GH: Non-governance PRs keep existing pulumi-pr-command
end
GH->>GovRunner: pulumi-governance-command received
GovRunner->>GovWF: Start workflow with untrusted client_payload
GovWF->>Preflight: Re-derive trust server-side
Note over Preflight: All client_payload treated as untrusted
Preflight->>Preflight: Validate PR number, head SHA, PR is open, same-repo
Preflight->>Preflight: Resolve comment author from comment_id
alt Author not Kravalg
Preflight-->>GovWF: Reject with comment
GovWF-->>GH: Exit early
end
Preflight->>Preflight: Recompute scope from head SHA files
alt Not governance scope
Preflight-->>GovWF: Reject with comment
GovWF-->>GH: Exit early
end
Preflight-->>GovWF: head_sha, pull_request_number
GovWF->>GovPlan: Governance Test Plan (aws-actions/configure-aws-credentials via OIDC)
Note over GovPlan: Uses test preview role
GovPlan->>AWS: Assume preview role
GovPlan->>Pulumi: make pulumi-plan (gov stack, test env)
Pulumi->>Config: Read repositories.governance.json
Pulumi->>AWS: Get OIDC provider by ARN
Note over Pulumi: .get() only, never create
Pulumi->>AWS: Assert account = 891377212104 (test)
Pulumi->>Pulumi: For each repo in catalog...
Pulumi->>AWS: Create state bucket + KMS key
Pulumi->>AWS: Create preview/apply/drift roles
Note over AWS: Apply role trusts only environment:governance
Pulumi->>AWS: Create config-read role + CI secret
Note over Pulumi: Secret-read Deny on preview/drift roles
Pulumi-->>GovPlan: Plan saved as artifact
alt Environment gate
GovWF->>GovApply: Job runs under environment: governance
Note over GovApply: Sole reviewer @Kravalg, prevent_self_review
end
GovApply->>AWS: Assume apply role via OIDC (using saved plan)
GovApply->>Pulumi: make pulumi-up-plan (test stack)
Pulumi->>AWS: Apply per-repo state bucket, KMS, roles, secrets
Pulumi-->>GovApply: Test apply complete
GovApply->>AWS: Assume apply role (prod stack)
Pulumi->>AWS: Assert account = 933245420672 (prod)
Pulumi->>Pulumi: Loop over same repos, prod environment
Pulumi->>AWS: Create prod resources per repo
Pulumi-->>GovApply: Prod apply complete
GovApply->>GovStatus: Post Governance Apply success to head SHA
GovStatus->>GH: statuses/{head_sha} (context: Governance Apply)
Note over GovStatus: Always() job — posts success/failure from prod result
par Always-reportable status
Note over StatusWF: Pull_request_target trigger for EVERY PR
StatusWF->>ScopeDetect: Recompute scope from PR files
alt Governance touched
StatusWF->>GH: Post pending Governance Apply status
Note over StatusWF: Placeholder until governor runner completes
else Non-governance
StatusWF->>GH: Post success Governance Apply status
Note over StatusWF: Required check satisfied immediately
end
end
Note over AWS: Each managed repo gets isolated state bucket, KMS key, and roles
Note over AWS: Repo A cannot access Repo B's resources
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…+ governance contract The import-linter root_packages gained 'infra' and an 11th contract (infra.governance isolation) in the governance closeout; update the pinned structural assertion to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Requires human review: Auto-approval blocked by 9 unresolved issues from previous reviews.
Re-trigger cubic
"Governance Apply" was registered as a GLOBAL required status check, which gated EVERY bootstrap PR and required reconciling the live `main` ruleset (code expected 25 checks; the live ruleset enforces 24 — drift that broke the required "Test Account Evidence" check on this branch). A global required check is the wrong mechanism for the governance gate. The gate's real security is the protected `governance` GitHub Environment plus `.github/CODEOWNERS` (@Kravalg review), both already in place. Remove the "Governance Apply" entry so REQUIRED_STATUS_CHECKS matches the live ruleset again (24 checks). The governance runner still posts the "Governance Apply" commit status — now purely INFORMATIONAL, not a required check. Update the unit tests accordingly: drop the membership and required-check↔ runner binding assertions; keep the assertion that the runner posts the informational status; assert the required set is exactly the 24 contexts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
governance-apply-status.yml existed only to satisfy the global "Governance Apply" required check for non-governance PRs (it posted success/pending to every PR head via pull_request_target). With "Governance Apply" no longer a required check, this workflow is unnecessary. Delete the workflow and its structural test. Drop `/.github/workflows/governance-apply-status.yml` from GOVERNANCE_PATH_GLOBS and the byte-equal @Kravalg line in .github/CODEOWNERS (the drift test keeps the two lists set-equal), and remove the governance_paths test case that asserted that specific glob is in scope. The dedicated env-gated runner (pulumi-governance.yml) and its server-side re-auth + informational status-post are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert the required-check doc/evidence bumps that accompanied registering "Governance Apply" as a global required check: - external-control-evidence + readiness report + ci-guardrails + ci-quality-gates + operating-evidence: drop the "Governance Apply" required-check row/sentence and restore the documented set to all 24 ending at "Test Account Evidence" (requiredStatusCheckCount 25 -> 24). - governance-stack.md + AGENTS.md: reword "Governance Apply" from a required merge check to an informational commit status. The governance merge gate is CODEOWNERS (@Kravalg review) plus the protected `governance` environment (@Kravalg approves the apply); @Kravalg merges after the gated test+prod apply succeeds. Keeps the doc-sync contract test and the doc-presence tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
0 issues found across 14 files (changes from recent commits).
Requires human review: Auto-approval blocked by 8 unresolved issues from previous reviews.
Re-trigger cubic
…t decrypt The read-only policy (preview/drift) carried a kms:Decrypt Deny on Resource *, but those roles also carry the pulumi-backend policy whose alias-scoped Allow grants kms:Decrypt on the repo's own secrets key. An explicit Deny wins over the Allow, so `pulumi preview`/drift could not decrypt the stack's encrypted config on any KMS-secrets stack. Remove kms:Decrypt from _READ_ONLY_SECRET_DENY_ACTIONS (the alias-scoped Allow is the boundary; the broad Deny was redundant and harmful). secretsmanager:GetSecretValue and the other leak actions stay denied. config-read keeps its kms:Decrypt Deny (it never decrypts a Pulumi key). Regenerate the golden parity fixture and update §5.3/D4 + the deny tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ength
The repository-catalog validator checked only `{project}-prod-preview` against
the 64-char IAM limit, omitting the role-name prefixes. The actual longest
rendered role is the config-read role `GitHubCiConfigRead-{project}-prod-preview`
(19 + project + 13), so a catalog could validate yet produce a >64-char role
that fails at apply. Compute the guard against every rendered role name (the
preview/apply/drift trio per stack and the per-suffix config-read roles) and
reject if any exceeds 64. Add a test for a name the old guard accepted (40 chars
-> 72-char config-read role) that the corrected guard rejects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ject pulumi_dir flows into each managed repo's generated CI-config payload, and the managed *-infrastructure repo hosts its Pulumi project at the repo-root pulumi/ directory (architecture §8.1), not at pulumi/governance (which exists only in this bootstrap repo). The wrong default would point downstream self-deploy at a non-existent pulumi/governance dir. Default pulumiDir to "pulumi"; add tests on the entrypoint default and the rendered payload PULUMI_DIR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three governance.py fixes: - F6: the prod "prod" CI payload omitted PULUMI_PREVIEW_STACKS, diverging from ci_bootstrap._payloads. Add it so the saved-plan apply targets the same stack set the preview produced. - F7: _governance_apply_subjects took an unused settings arg (the subject is fully determined by repository + the fixed governance env). Remove it and update callers + tests. - F8: annotate _repo_settings's return as BootstrapSettings so it is not inferred as DataclassInstance (radarlint S5886). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…book `pulumi stack init --secrets-provider` persists the provider into the stack config, so the subsequent preview/up no longer need the PULUMI_SECRETS_PROVIDER env prefix. Tidy the Step 1 runbook commands and note the persistence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…st path
F1 — the governance runner ran every job unconditionally, so `/pulumi test plan`
performed an apply and `/pulumi test up` promoted to prod. Thread the parsed
command (plan|up) and target (test|prod) from the intake through client_payload,
re-derive and allowlist-validate them server-side in preflight (never trust the
payload blindly), expose them as preflight outputs, and gate the jobs with `if:`
guards mirroring the established runner: `plan` reaches no apply; `test up` runs
test apply+drift but no prod job; `prod up` runs test then prod. The commit
status now reflects the stage the request was meant to reach (a plan/test-up is
no longer reported as a failed prod apply).
F2 — the apply jobs run under `environment: governance`, so their OIDC token
carries sub=environment:governance, which the bootstrap test/prod CI-config and
deploy roles do NOT trust — the apply credential path was broken. The apply jobs
now assume a dedicated per-account governance automation role
(vars.AWS_GOVERNANCE_{TEST,PROD}_APPLY_ROLE_ARN, trust=environment:governance)
directly via OIDC, with the governance program's own backend/secrets provider
sourced from repo variables. The plan/drift jobs (no environment) keep the
existing branch-ref CI-config path. The role + its least-privilege apply policy
are operator-created in the privileged bootstrap (documented in the runbook +
architecture §7.2; repo variables in github-actions-secrets.md). Add structural
tests asserting the env-trusted role assumption and the command/target gating.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ty flagged max(..., key=len) return as not provably str (invalid-return-type, not in the gate ignore list). Use an explicit loop reduction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Generalizes the one-time
pulumi/github-ci-bootstrapproject into a reusable, Kravalg-gated multi-repo IAM/OIDC governance stack.bootstrap-infrastructurenow owns — in a separate, privileged Pulumi project — every IAM role, S3 state bucket, KMS key, and GitHub OIDC trust required by any managed*-infrastructurerepo. Onboarding a new service infra repo becomes config + GitHub-PR-comment driven, never a manualAdministratorAccessaction.Closes #77. Stacked on
codex/issue59-github-ci-bootstrap-stack(#60), which it extends.What this delivers
Generic governance component (
pulumi/infra/governance.py,pulumi/governance/):GovernanceStack+RepoGovernancedrive a per-repo loop overpulumi/repositories.governance.json. For each managed repo they provision: preview/apply/drift OIDC deploy roles, a least-privilege deploy policy scoped to that repo's state bucket + KMS alias only, a config-read role + CI-config secret, an S3 state bucket, and a KMS key. Adding a repo is config-only.github-ci-bootstrapoutput stays byte-identical (NFR6, enforced by a golden parity fixture).891377212104, prod stack →933245420672(separate-account blast-radius isolation). Component code is account-parametric (no hardcoded account literals; per-stack assertion)..get(), never created). Each stack consumes its own account's provider.Least privilege (red-team verified):
Effect: Allowwildcard on any deploy role. Repo A cannot touch repo B's bucket/KMS; no service repo can decrypt the platform-bootstrap key.Denyon read-only/config-read roles; surgicalDenyon the apply role (keeps onlykms:Decrypton its own key +GetSecretValueon its own CI secret).Kravalg-only governance gate:
.github/CODEOWNERSscopes the governance/IAM/policy/trust paths (incl. apply-context helpers) to@Kravalg; the glob set is kept byte-equal toscripts/governance_paths.pyby a drift test.governanceGitHub Environment (sole reviewer@Kravalg,prevent_self_review) + hardened branch rule (dismiss_stale_reviews_on_push,require_last_push_approval).uprequires author==Kravalg; non-governance PRs keep the existing association rule.@dmytrocraftopens PRs and may runplan.pulumi-governance.ymlrunner with server-side re-authorization (re-resolves comment author + recomputes scope, allclient_payloaduntrusted), staticenvironment: governance-gated test→prod applies, and a terminalGovernance Applycommit status (success-before-merge). IaC-only apply (saved-plan; humans neverpulumi upin CI).pull_request_targetstatus setter postsGovernance Applyfor every PR head SHA (success for non-governance, pending for governance) so the required check never strands a PR.user-service-infrastructurescaffold: in-repo template assets (pulumi/user-service-infrastructure/) + aself-deploy.ymltemplate that assumes only the governance-provided OIDC roles (name-parity asserted against the rendered role names; no static keys, no admin access).Docs:
AGENTS.mdmulti-repo onboarding flow (CODE/OPERATOR labeled) +docs/governance-stack.mdoperator runbook. Planning record underspecs/issue-77-multi-repo-governance/(incl. a Well-Architected review).Method
Planned with the BMAD method (research → brief → PRD → architecture → epics/stories → readiness, PASS), adversarially critiqued (3 planning lenses + a 5-lens red-team of the gating surface — all criticals fixed: missing apply-context globs, the
Governance Applyrequired-check lifecycle, the preflight-failure deadlock), implemented as 24 forward-safe stories with TDD, then driven through the full quality gate stack.How Has This Been Tested?
All via the container gate stack:
make test-unit— 100% unit branch coverage (pulumi/*,scripts/*).make test-policy— 100%; CrossGuardiam-no-wildcardsclean on governance roles.make test-integration— 100%.make test-coverage— combined 100%.make test-architecture— import-linter 11 contracts kept, 0 broken (newinfra.governance ↛ {policy, app}contract + AST guard forscripts).make test-ty— clean.make test-mutation— clean. Structural + actionlint + yamllint — clean.Operator-only follow-up (requires AWS admin / GitHub org-admin — out of this PR)
All code/IaC/docs/tests are delivered and green; the following need your credentials and are enumerated in
docs/governance-stack.md:pulumi -C pulumi/governance up --stack testthen--stack prod, each with its account'sawskms://secrets provider.pulumi/governance/Pulumi.{test,prod}.yamlbefore the first apply (the stack raises if unset).costAnomalyMonitorArnmatches each stack's account (test→891…, prod→933…); do not repoint prod off933245420672.governanceenvironment + branch protection viascripts/configure_github_repository_controls.py --apply.githubVariablesoutput.user-service-infrastructureinVilnaCRM-Organd push the scaffold.@Kravalgcomments/pulumi test upthen/pulumi prod up).@Kravalgis unavailable.Types of changes
🤖 Generated with Claude Code
Summary by cubic
Build a Kravalg-gated, multi-repo IAM/OIDC governance stack that provisions per‑repo state buckets, KMS keys, and GitHub OIDC roles, with a dedicated governance runner that applies saved plans and posts an informational “Governance Apply” status. The runner now honors plan/test/prod targets and uses dedicated env‑trusted apply roles for safer applies.
Bug Fixes
kms:Decryptfrom the deny list so preview/drift can decrypt repo secrets; config-read/apply deny rules stay scoped.AWS_GOVERNANCE_{TEST,PROD}_APPLY_ROLE_ARNand use governance backend/secrets from repo vars; plan/drift keep the CI-config path. Status now reflects the intended stage.GitHubCiConfigRead-*) and fail early on overflow; explicitly returnstrfor longest-name detection to satisfy typing.PULUMI_DIRtopulumi; prod CI payload includesPULUMI_PREVIEW_STACKSfor plan/apply parity.PULUMI_SECRETS_PROVIDERafterpulumi stack init(provider persists in stack config).Migration
pulumi/governance/Pulumi.{test,prod}.yaml, then applypulumi/governance(test→prod) with the per‑accountawskms://provider.governanceenvironment and branch protection viascripts/configure_github_repository_controls.py --apply.AWS_GOVERNANCE_TEST_APPLY_ROLE_ARN,AWS_GOVERNANCE_PROD_APPLY_ROLE_ARNand backend/secrets provider URLs) from the governancegithubVariablesoutput, then run gated applies by@Kravalg(/pulumi test up,/pulumi prod up).Written for commit 4b97403. Summary will update on new commits.