🟠 Estate CI-health backlog: public repos with independently-red CI

After the allow-list fix (27 repos) and billing (separate), the remaining stuck PRs are blocked by each repo's own red CI — real build/test/governance failures, not the dependency bump. Rebasing won't help; each needs per-repo repair (home: hypatia ci-health-sweep, #461).

Sampled failures: bofig (Build and test, Hypatia) · project-wharf (Analyze rust/actions, build, workflow linter) · double-track-browser (TypeScript Tests, language-policy) · coq-jr (language-policy, Well-Known, Validate Security Files).

Repos with stuck PRs (per-repo CI repair needed):

Note: ~33 of the burn-cut PRs opened this session are also gated on these same repo-CI issues (or billing).

docs(status): session handover snapshot 2026-06-20

docs(status): session handover snapshot 2026-06-20 (#517)

docs(audits): record central actions/cache SHA corruption + #394 repa…

📋 Status — 2026-06-21: the Hypatia-lane root cause is a central `actions/cache` SHA corruption (already fixed & merged)

Actioning the CI-health findings surfaced 2026-06-20/21. I have hypatia + standards write access (the reporting session did not, so it could only diagnose). Cross-ref nextgen-typing#69 ("out of scope — central").

(1) ROOT CAUSE — central `actions/cache` SHA corruption — ✅ FIXED & MERGED

The estate-wide scan / Hypatia Neurosymbolic Analysis failure at "Prepare all required actions":

Unable to resolve action actions/cache@d4373f267a887d77f9eb0683a479ec60b1fe5b2b
(unable to find version d4373f267a887d77f9eb0683a479ec60b1fe5b2b)

…is, as diagnosed, not in any consumer workflow. It was pinned once, centrally, in both Hypatia reusables in standards:

.github/workflows/hypatia-scan-reusable.yml
.github/workflows/governance-reusable.yml

d4373f… resolves to nothing upstream — it's a corruption of v4.2.2's real commit d4323d4….

It was already repaired and merged before I reached it: standards#394 (merged 2026-06-21 10:52Z, commit d72fe5a) re-pinned both reusables to the genuine v4.2.0 commit 1bd1e32a…, preserving the # v4.2.0 comment.

Independently verified this session via git ls-remote https://github.com/actions/cache:

SHA	upstream ref	resolves?
`d4373f…` (the corrupt pin)	(none)	✗ bogus
`1bd1e32a…` (the repair)	`refs/tags/v4.2.0`	✓
`0057852b…` ("most common")	`v4` + `v4.3.0`	✓
`27d5ce7f…` (used across hypatia)	`main` + `v5` + `v5.0.5`	✓

git grep d4373f… across standards + hypatia → 0 matches. hypatia's own workflows are clean (they pin actions/cache to the valid 27d5ce7f… # v5.0.5).

⚠️ Propagation caveat — necessary but not yet sufficient

Consumers pin these reusables by standards commit SHA, not @main (hypatia uses @5eb28d7d…; most repos @861b5e91…). The repair landed as a new standards HEAD (d72fe5a), so a consumer still pinned at a pre-#394 SHA keeps dereferencing the broken cache pin until it is re-enrolled to d72fe5a+. A consumer's lane therefore only goes green after re-enrollment (gitbot-fleet enroll-repos) — not automatically.

(2) `governance / Check Workflow Staleness` red — EXPECTED drift, not a new defect

standards/scripts/check-workflow-staleness.sh fails any consumer whose pinned reusable SHA ≠ current standards HEAD ("Workflow pins Hypatia reusable before cache/baseline-delay fix. Refresh to current standards SHA."). Because #394 advanced HEAD to d72fe5a, every consumer is stale by definition right now — this is precisely the signal that the re-enrollment pass is pending, the same root cause as (1) viewed from the propagation side.

Remediation = gitbot-fleet enroll-repos re-pin of consumers to d72fe5a+. That's out of scope for standards/hypatia (consumer repos and gitbot-fleet aren't in my access), so per the task I'm recording it here as expected post-#394 drift.

(3) `nextgen-databases`, pre-existing & repo-internal — recorded (out of scope to fix)

Both live in nextgen-databases (not standards/hypatia), so flagging with grounded fixes rather than touching them:

K9 pedigree — verisimdb/connectors/test-infra/deploy.k9.ncl → "Pedigree block missing 'name'". Confirmed against the schema: in standards/k9-svc/pedigree.ncl, Metadata.name | String is the only metadata field with no default, so it is mandatory. Fix: add metadata.name; per the canonical k9-svc/pandoc/container/deploy.k9.ncl sample, ideally also metadata.version + validation.pedigree_version and a leash level trust_level/security_level ∈ 'Kennel | 'Yard | 'Hunt (a shell-running deploy.k9.ncl is 'Hunt).
governance / Trusted-base reduction policy red — per standards/docs/TRUSTED-BASE-REDUCTION-POLICY.adoc + scripts/check-trusted-base.sh: an undocumented soundness-relevant escape hatch in a proof-bearing file. Disposition is per-repo — discharge, budget (// TRUSTED:), axiom (// AXIOM:), or a dated debt entry in nextgen-databases' docs/proof-debt.md.

Durable record

Full audit with the verification table + propagation analysis: standards/docs/audits/audit-hypatia-cache-sha-corruption-2026-06-21.adoc (+ .a2ml companion), filed as draft standards#396.

Net. The Hypatia-lane root cause (1) is fixed & merged (standards#394), independently verified. (2) is expected post-fix staleness drift awaiting a gitbot-fleet enroll-repos re-pin to d72fe5a+ (out of my scope). (3) is two pre-existing nextgen-databases-internal items for that repo's maintainers. Nothing here required (or received) an unverified fix.

Generated by Claude Code

▶️ Remaining action — propagate the fix via gitbot-fleet `enroll-repos`

The central root cause is fixed + merged (standards#394). The only step left to turn the estate green is to re-pin stale consumers — they pin the reusables by standards commit SHA (not @main), so they keep dereferencing the old workflow and show Check Workflow Staleness red until refreshed.

Target pin: current standards/main HEAD = 4ddc926 (contains actions/cache@1bd1e32a… # v4.2.0; re-resolve to current HEAD at run time).

Operation (same as the 2026-05-27 orphan re-pin sweep — Contents API + auto-merge squash):

Run gitbot-fleet enroll-repos to re-pin each stale consumer's .github/workflows:
- hypatia-scan.yml → hypatia-scan-reusable.yml@4ddc926
- governance.yml → governance-reusable.yml@4ddc926
Prioritise the observed-failing repos: nextgen-databases, KnotTheory.jl, nextgen-typing, plus the repo list above.
Verify on 2–3 samples: scan lane green + staleness check green; post the result here.

Not covered by this sweep (separate per-repo fixes in nextgen-databases):

K9 pedigree — verisimdb/connectors/test-infra/deploy.k9.ncl needs metadata.name (+ version/pedigree_version + trust_level 'Hunt).
governance / trusted-base red — undocumented escape hatch → entry in that repo's docs/proof-debt.md.

Note: this step cannot be driven from a session scoped only to hypatia/standards (which is where the diagnosis + audit #396 were done). It needs a session/agent scoped to gitbot-fleet (and the consumer repos). Cross-ref nextgen-typing#69.

Generated by Claude Code

⛔ enroll-repos re-pin is blocked — the propagation actuator (.git-private-farm Actions) is down

Picked this up from a session scoped to gitbot-fleet + .git-private-farm. The re-pin is fully scoped and ready, but it cannot run: the farm workflow that performs the Contents-API + auto-merge-squash sweep is failing at the infrastructure level. Reporting before any half-done state is created.

Target pin (confirmed)

standards/main HEAD = 4ddc926 (carries actions/cache@1bd1e32a… # v4.2.0, the #394 repair). stapeln + panoply are already pinned here — proof the target resolves.

Estate consumers enumerated (GitHub code search, both reusables)

SHA-stale = pinned SHA ≠ 4ddc926, excluding @main auto-trackers (already get the fix) and the 2 already at HEAD:

Reusable	refs	`@main` (skip)	already `4ddc926`	SHA-stale to bump
`governance-reusable.yml`	225	62	2	~143
`hypatia-scan-reusable.yml`	197	0	2	~195

Old-SHA clusters (one farm run keys on one OLD_SHA, so this is ~17 dispatches):

governance: 5a93d9d5→102, 861b5e91→30, d72fe5a→4, 5eb28d7d→2, 3b3549e2→2, +3 singletons (echidna, ephapax, valence-shell)
hypatia-scan: 5a93d9d5→102, 915139d7→54, 97df762→16 (the 2026-05-27 orphan SHA still live), 6cd37728→12, 5eb28d7d→5, d72fe5a→3, +3 singletons (bunsenite, eclexia, vcl-ut)

Priority repos located: nextgen-typing both reusables @5a93d9d5; nextgen-databases governance @main (no re-pin) + hypatia-scan @915139d7; KnotTheory.jl both @5a93d9d5 (⚠️ this one was missed by paginated code-search and only found by targeted lookup — code-search pagination is lossy, so the final sweep needs a per-repo verification pass to catch stragglers).

The blocker — farm GitHub Actions are not allocating runners

I fired one validation dispatch into .git-private-farm sha-bump-propagate.yml (workflow_dispatch, nextgen-typing hypatia-scan 5a93d9d5→4ddc926). It failed in 6 s with no runner assigned (runner_id: 0, run 27914137962) — before any step ran. Inputs passed every validation gate; this is not an input/logic failure.

It is not isolated. Every .git-private-farm Actions run today dies the same way:

14+ Propagate (repository_dispatch) runs — all failure in 7–19 s
scheduled Well-Known Standards, Estate CI-deadlock Audit, RustSec Audit, Forge Drift Detection — failure in 4–5 s
OSSF Scorecard — startup_failure in 0 s

GitHub-hosted (ubuntu-latest) jobs failing in <20 s with no runner / startup_failure across unrelated workflows = Actions are administratively or financially blocked on the private farm repo (spending-limit / billing exhaustion, or Actions disabled). This is the "billing (separate)" blocker already noted on this issue — and it sits upstream of the re-pin: the estate's own propagation automation has been failing against it all day.

Net

0 consumers re-pinned, 0 verified. The single sample failed on infra, not on the change.
The farm is the only estate-wide write path (consumers are out of this session's repo scope), so the sweep is blocked until farm Actions billing/runners are restored.
Once restored, the sweep is ~17 workflow_dispatch calls into sha-bump-propagate.yml (one per reusable × old-SHA, all new_sha=4ddc926) + a re-search pass for stragglers + sample verification. I can re-fire immediately on the word that Actions are live again.

Out-of-scope items unchanged (K9 pedigree on nextgen-databases, trusted-base proof-debt) — not part of this sweep.

Posted by Claude Code (gitbot-fleet session). No changes were pushed; this is a blocker report.

Generated by Claude Code

ci: refresh standards reusable pins to current HEAD (d7c2271) (#74)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#52)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#37)

@Head

✅ 3 priority repos re-pinned + admin-merged — and the real `scan` blocker is a fresh Hypatia compile regression

Did these directly (owner-admin token), not via the farm: .git-private-farm Actions are still billing-blocked (every run dies <20s with no runner). bag-of-actions offers no propagation path (Zig/Elixir continuation runtime; its only workflows are codeql + push-email-notify).

Target corrected — pinned to current HEAD, not `4ddc926`

Re-resolved standards/main HEAD at run time (as the task instructs): it has advanced to d7c22711 (#432, 2026‑06‑26), well past 4ddc926 (06‑21). The samples were already at d135b05b (#416, 06‑24 — post‑cache‑fix), so their cache lane was already resolved; the live-open items were staleness + the newer reusable fixes (#424 --exit-zero, #429 GITHUB_TOKEN). All re-pinned to d7c22711.

Done (all merged)

repo	PR	reusables → `d7c2271`	extra	`Check Workflow Staleness`
nextgen-typing	#74 ✅	hypatia-scan, governance, scorecard	—	pins all @Head
nextgen-databases	#52 ✅	hypatia-scan, governance, scorecard	governance `@main`→pinned; removed retired `scorecard-enforcer.yml`	🟢 success
KnotTheory.jl	#37 ✅	hypatia-scan, governance, scorecard	—	🟢 success

Notes: I bumped scorecard-reusable too — the staleness checker fails on any of {scorecard, hypatia-scan, governance} pin ≠ HEAD, not just the two named. KnotTheory.jl was missing from code-search enumeration (the index is lossy) — found by direct lookup.

Verification

governance / Check Workflow Staleness → success on nextgen-databases + KnotTheory.jl (post-merge main).
In CI, actions/cache@27d5ce7f… resolves fine → the central cache corruption is fully closed.

🔴 `scan / Hypatia Neurosymbolic Analysis` still RED — estate-wide — and it is NOT the pin

The reusable git clones hypatia.git and runs mix escript.build; hypatia main no longer compiles as of #545 (7df53b1b, 06‑26 20:29):

== Compilation error in file lib/rules/structural_drift.ex ==
** (CompileError) undefined variable "real_subdirs"

lib/rules/structural_drift.ex:1019 still references real_subdirs, but fix(scanner): make SD022 path-drift rule precise + fix real doc/workflow drift #545 renamed that binding to real_basenames (src_dir_index/1, line 963). real_subdirs is now defined nowhere → escript.build fails → the scan job dies at "Build Hypatia scanner" on every repo that calls the reusable.
Latent second defect: line 1004 regex has one capture group, but line 1006 destructures [_, prefix, dir] (two) → FunctionClauseError once the compile error is fixed.

I did not push a fix — there's no Elixir/mix in this session to compile-verify it, and you're actively iterating on this file (the bash fallback can't substitute). Flagging for a tested fix. Until hypatia main compiles, no repo's scan lane can go green — this is the real "scan green" blocker now (cache is closed).

Still open

Full estate re-pin (~200 consumers): needs either .git-private-farm Actions billing restored (farm sweep), or the same direct-token approach I used here, repo-by-repo. Happy to run the rest the same way on your go.
Hypatia main compile fix (fix(scanner): make SD022 path-drift rule precise + fix real doc/workflow drift #545 follow-up) to clear the estate-wide scan red.

Posted by Claude Code (gitbot-fleet session).

Generated by Claude Code

fix(scanner): repair SD022 compile error breaking estate-wide Hypatia…

ci: refresh standards reusable pins to current HEAD (d7c2271) (#93)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#16)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#58)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#19)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#70)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#12)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#30)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#238)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#20)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#19)

ci: refresh standards reusable pins to current HEAD (d7c2271) (#166)

Status (2026-09-30 issue sweep): reviewed and kept open as a live meta-tracker. Feature/RFC issues were consolidated into the roadmap #886. Root-cause fixes in flight: #883 (scanner build + npx precision), #888 (workflow_audit path anchoring, 0-AI-MANIFEST.deed), standards#1084 (harden-runner in standards-only workflows). Ledger: dev-notes/audits/hypatia-issue-sweep/2026-09-30.md.

Navigation Menu

Repository navigation

🟠 Estate CI-health backlog: public repos with independently-red CI #464

📋 Status — 2026-06-21: the Hypatia-lane root cause is a central `actions/cache` SHA corruption (already fixed & merged)

(1) ROOT CAUSE — central `actions/cache` SHA corruption — ✅ FIXED & MERGED

⚠️ Propagation caveat — necessary but not yet sufficient

(2) `governance / Check Workflow Staleness` red — EXPECTED drift, not a new defect

(3) `nextgen-databases`, pre-existing & repo-internal — recorded (out of scope to fix)

Durable record

▶️ Remaining action — propagate the fix via gitbot-fleet `enroll-repos`

⛔ enroll-repos re-pin is blocked — the propagation actuator (.git-private-farm Actions) is down

Target pin (confirmed)

Estate consumers enumerated (GitHub code search, both reusables)

The blocker — farm GitHub Actions are not allocating runners

Net

✅ 3 priority repos re-pinned + admin-merged — and the real `scan` blocker is a fresh Hypatia compile regression

Target corrected — pinned to current HEAD, not `4ddc926`

Done (all merged)

Verification

🔴 `scan / Hypatia Neurosymbolic Analysis` still RED — estate-wide — and it is NOT the pin

Still open

143 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

🟠 Estate CI-health backlog: public repos with independently-red CI #464

Description

Activity

hyperpolymath commented on Jun 21, 2026

📋 Status — 2026-06-21: the Hypatia-lane root cause is a central actions/cache SHA corruption (already fixed & merged)

(1) ROOT CAUSE — central actions/cache SHA corruption — ✅ FIXED & MERGED

⚠️ Propagation caveat — necessary but not yet sufficient

(2) governance / Check Workflow Staleness red — EXPECTED drift, not a new defect

(3) nextgen-databases, pre-existing & repo-internal — recorded (out of scope to fix)

Durable record

hyperpolymath commented on Jun 21, 2026

▶️ Remaining action — propagate the fix via gitbot-fleet enroll-repos

hyperpolymath commented on Jun 26, 2026

⛔ enroll-repos re-pin is blocked — the propagation actuator (.git-private-farm Actions) is down

Target pin (confirmed)

Estate consumers enumerated (GitHub code search, both reusables)

The blocker — farm GitHub Actions are not allocating runners

Net

hyperpolymath commented on Jun 27, 2026

✅ 3 priority repos re-pinned + admin-merged — and the real scan blocker is a fresh Hypatia compile regression

Target corrected — pinned to current HEAD, not 4ddc926

Done (all merged)

Verification

🔴 scan / Hypatia Neurosymbolic Analysis still RED — estate-wide — and it is NOT the pin

Still open

143 remaining items

hyperpolymath commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

📋 Status — 2026-06-21: the Hypatia-lane root cause is a central `actions/cache` SHA corruption (already fixed & merged)

(1) ROOT CAUSE — central `actions/cache` SHA corruption — ✅ FIXED & MERGED

(2) `governance / Check Workflow Staleness` red — EXPECTED drift, not a new defect

(3) `nextgen-databases`, pre-existing & repo-internal — recorded (out of scope to fix)

▶️ Remaining action — propagate the fix via gitbot-fleet `enroll-repos`

✅ 3 priority repos re-pinned + admin-merged — and the real `scan` blocker is a fresh Hypatia compile regression

Target corrected — pinned to current HEAD, not `4ddc926`

🔴 `scan / Hypatia Neurosymbolic Analysis` still RED — estate-wide — and it is NOT the pin