From c6db69fbfe86a5354f2ae7ed8ea5118a4d5577db Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 06:56:36 +0000 Subject: [PATCH 1/6] feat(pm/fleet-write): the run_rerun_failed row, its fences and its validator battery One relay op re-runs only the failed jobs of one ci.yml pull-request run, once: run_id alone, alone in its stroke, actions write minted for that stroke through the existing stroke-scoped output, and four fences on the run itself (workflow, event, once, failed) that the executor judges. REFUSED_PATH_FAMILIES carves out exactly runs/{digits}/rerun-failed-jobs. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- scripts/pm/fleet-write/ops.mjs | 251 +++++++++++++++++++++++++--- scripts/pm/fleet-write/validate.mjs | 148 ++++++++++++++-- 2 files changed, 359 insertions(+), 40 deletions(-) diff --git a/scripts/pm/fleet-write/ops.mjs b/scripts/pm/fleet-write/ops.mjs index 33d690ac58c..0bf473d1f8b 100644 --- a/scripts/pm/fleet-write/ops.mjs +++ b/scripts/pm/fleet-write/ops.mjs @@ -30,13 +30,14 @@ * * An op is only what this table names. There is no row for a merge of a pull, * a review submission, a direct branch or ref write (`/git/**`, `/branches/**`, - * `/contents/**`), a workflow-file write, a run cancel / re-run / delete, a - * cache or artifact write, a release, or any organization / administration - * endpoint, so a payload naming one is refused at the validator with zero - * writes — the `--self-test` of `validate.mjs` pins that every path spelled - * here stays clear of those families and that the table's one `PUT` - * (`pr_update_branch`, below) reaches no whole-set replace of labels or - * assignees. + * `/contents/**`), a workflow-file write, a run cancel or delete, a whole-run + * or single-job re-run, a cache or artifact write, a release, or any + * organization / administration endpoint, so a payload naming one is refused + * at the validator with zero writes — the `--self-test` of `validate.mjs` pins + * that every path spelled here stays clear of those families and that the + * table's one `PUT` (`pr_update_branch`, below) reaches no whole-set replace + * of labels or assignees. The one re-run the table has re-runs a run's FAILED + * jobs alone, once, on a closed list of workflows (`run_rerun_failed`, below). * * ## The one row that moves a branch — `pr_update_branch` * @@ -152,8 +153,9 @@ * the maintainer's own ruling, comment 6104822731 (「22708 A」), so this grant * is not a drift. The grant also reaches workflow-file writes through * `/contents/**` and `/git/**`; `REFUSED_PATH_FAMILIES` refuses both for every - * row. The one workflow-run dispatch (`workflow_dispatch`, below) - * is `actions: write`, the one row spending it; the sender gate's + * row. The workflow-run dispatch (`workflow_dispatch`) and the failed-jobs + * re-run (`run_rerun_failed`), both below, are `actions: write`, the two rows + * spending it; the sender gate's * `GET /repos/{o}/{r}/collaborators/{login}/permission` is `metadata: read`. * `PERMISSIONS` is that set and nothing wider. The workflow declares the five * grants every stroke gets as literal `permission-*` inputs on @@ -178,11 +180,63 @@ * value for every other stroke, and the mint action skips an empty * `permission-*` input, so every other stroke's token is what it was before * this row. `REFUSED_PATH_FAMILIES` refuses every other `actions/*` path, and - * `validate.mjs --self-test` pins that the dispatch call is the only one the - * table reaches. The dispatch answers 204 with no body, so the run it starts + * `validate.mjs --self-test` pins that the dispatch call and + * `run_rerun_failed`'s re-run call (below) are the only two the table reaches. + * The dispatch answers 204 with no body, so the run it starts * is read back by the SEAT (`dispatch.mjs`): the newest `workflow_dispatch` * run of that file created since the dispatch. * + * ## The second row whose token carries `actions: write` — `run_rerun_failed` + * + * `POST /repos/{repo}/actions/runs/{run_id}/rerun-failed-jobs` starts a new + * ATTEMPT of a completed run in which the jobs that failed, and the jobs that + * need them, run again, and the jobs that passed are carried over as they + * were. Measured on run 39713789453's attempt 2: the passed jobs were listed + * in it at once as `completed`, keeping attempt 1's start times; the failed + * shard was listed at once and started two seconds later; and the aggregate + * job that needs every shard was listed only when its need finished, 16 + * minutes later. It spends the `actions: write` `workflow_dispatch` spends, so + * it is fenced the same way. These are the six fences: + * + * - the WORKFLOW: the run's `path` is one of `RERUN_FAILED_WORKFLOWS`, a + * closed list matched whole and joined only by a PR that adds the path + * here — never a request-time input, since the payload carries no key + * that names a workflow; + * - the EVENT: the run's `event` is `pull_request` — a push, merge-group, + * schedule, release or `workflow_dispatch` run is refused; + * - ONCE: the run's `run_attempt` is 1. A second failure is real, and a + * re-send after a landing meets attempt 2 here and is refused; + * - FAILED: the run is `completed` with `conclusion` `failure`; + * - ALONE: a stroke carrying it carries exactly that one action (`alone`), + * so the wider token serves one request; + * - the GRANT: `actions: write` is minted for that stroke alone, through the + * `STROKE_SCOPED_PERMISSIONS` output `workflow_dispatch` already uses — the + * validator writes `permission_actions=write` for either op's stroke and an + * empty value for every other, so the workflow names neither op. + * + * The payload is `run_id` and nothing else: no workflow, attempt, job or + * debug-logging key exists for it, and the POST carries no body. The first + * four fences are facts about the RUN, not the request, so the EXECUTOR judges + * them (`RERUN_FAILED_FENCES`) on `GET /repos/{repo}/actions/runs/{run_id}`, + * read immediately before the POST. A run that changed between the seat's + * request and the relay's run, for example re-run by a person meanwhile, is + * refused THERE with zero writes. The seat's wrapper + * (`scripts/pm/rerun-failed.mjs`) reads the same four before it dispatches. + * That is an early refusal naming the fence that fails, and it never replaces + * the executor's. + * + * The 201 the POST answers is an acceptance, not a landing. The executor reads + * the run again until it shows `run_attempt` 2, then that attempt's job list, + * and the action LANDS only on what `rerunFailedReadBack` judges there. Every + * job that concluded `failure` in attempt 1 and is listed in attempt 2 must be + * `queued` or `in_progress`, and at least one must be. A failed job not yet + * listed is NAMED, not failed: a job is created when its needs finish + * (measured above). A read-back that never sees attempt 2 inside its window is + * a FAILED action, never a success. `REFUSED_PATH_FAMILIES` carves out exactly + * `runs/{digits}/rerun-failed-jobs`; the whole-run `rerun`, a job's `rerun`, + * cancel, delete, logs, caches, artifacts, secrets and variables all stay + * refused, and `validate.mjs --self-test` pins each. + * * ## A workflow's inputs are CLOSED per file — `WORKFLOW_DISPATCH_INPUTS` * * An input picks which of a workflow's lanes the run takes, so the inputs are @@ -224,7 +278,8 @@ * endpoint) needs `issues: write` on the repository the card is in AND on the * one it moves to, so a transfer stroke's token is minted for both. That is * the only widening of the token's repository reach the relay makes (the one - * per-stroke widening of its permissions is `workflow_dispatch`'s, above), and it is fenced four ways: the target is + * per-stroke widening of its permissions is the `actions` grant of + * `workflow_dispatch` and `run_rerun_failed`, above), and it is fenced four ways: the target is * a closed enum — `TRANSFER_TARGETS`, the fleet's governed-repository roster * reused from `check-governed-merges.mjs`, never a second roster and never * free text; the target is never the source; a stroke carrying a transfer @@ -314,9 +369,138 @@ export function dispatchInputsOnTheWire(declared) { } /** The one ref a `workflow_dispatch` may start a run at. ⛔ Never a free string. */ export const WORKFLOW_DISPATCH_REF = 'main'; -/** The shape of the one `actions/*` path the table reaches — the dispatch call, nothing beside it. */ +/** The shape of the dispatch call — one of the two `actions/*` paths the table reaches. */ export const WORKFLOW_DISPATCH_PATH_SHAPE = /\/actions\/workflows\/[^/]+\/dispatches$/; +/** + * The workflows a `run_rerun_failed` may re-run the failed jobs of — the run's + * own `path`, as `GET /actions/runs/{id}` answers it, matched WHOLE (measured + * on run 39713789453: `.github/workflows/ci.yml`, no ref suffix). Joined only + * by a PR that adds the path here; ⛔ never a key of the payload. + * `validate.mjs --self-test` pins every entry to a workflow on disk that runs + * on `pull_request`. + */ +export const RERUN_FAILED_WORKFLOWS = Object.freeze(['.github/workflows/ci.yml']); +/** The one event whose run a `run_rerun_failed` may re-run: a pull request's. */ +export const RERUN_FAILED_EVENT = 'pull_request'; +/** The one attempt it may re-run — once: a second failure is real. The read-back wants the attempt after it. */ +export const RERUN_FAILED_ATTEMPT = 1; +/** The state the run must be in: completed, and failed. */ +export const RERUN_FAILED_STATUS = 'completed'; +export const RERUN_FAILED_CONCLUSION = 'failure'; +/** The job states the read-back accepts for a job attempt 1 failed — the ruling's two, nothing beside them. */ +export const RERUN_FAILED_RERUN_STATES = Object.freeze(['queued', 'in_progress']); +/** The shape of the re-run call — the other `actions/*` path the table reaches; the run id is digits alone. */ +export const RERUN_FAILED_PATH_SHAPE = /\/actions\/runs\/\d+\/rerun-failed-jobs$/; + +/** + * The four fences on the RUN (the header's `run_rerun_failed` section), each + * judged on the run's own answer: `fence` its name, `wants` what the relay + * re-runs, `reads` the field(s) the failure sentence quotes, `holds` the + * predicate. The other two fences — alone in its stroke, the stroke-scoped + * grant — are the validator's. The executor judges these on the read taken + * immediately before the POST; the seat's wrapper judges them earlier, as an + * advisory refusal. + */ +export const RERUN_FAILED_FENCES = Object.freeze([ + Object.freeze({ + fence: 'workflow', + wants: `a run whose path is one of ${RERUN_FAILED_WORKFLOWS.join(', ')} — a closed list, matched whole`, + reads: (run) => run?.path, + holds: (run) => typeof run?.path === 'string' && RERUN_FAILED_WORKFLOWS.includes(run.path), + }), + Object.freeze({ + fence: 'event', + wants: `a ${RERUN_FAILED_EVENT} run — never a push, merge_group, schedule, release or workflow_dispatch one`, + reads: (run) => run?.event, + holds: (run) => run?.event === RERUN_FAILED_EVENT, + }), + Object.freeze({ + fence: 'once', + wants: `run_attempt ${RERUN_FAILED_ATTEMPT} — a run is re-run at most once, and a second failure is real`, + reads: (run) => run?.run_attempt, + holds: (run) => run?.run_attempt === RERUN_FAILED_ATTEMPT, + }), + Object.freeze({ + fence: 'failed', + wants: `status ${RERUN_FAILED_STATUS} with conclusion ${RERUN_FAILED_CONCLUSION}`, + reads: (run) => `status ${run?.status ?? null}, conclusion ${run?.conclusion ?? null}`, + holds: (run) => run?.status === RERUN_FAILED_STATUS && run?.conclusion === RERUN_FAILED_CONCLUSION, + }), +]); + +/** The fences a run's answer fails: `[{ fence, wants, got }]`, in `RERUN_FAILED_FENCES` order; `[]` when all four hold. Pure. */ +export function rerunFailedFenceFailures(run) { + return RERUN_FAILED_FENCES.filter((f) => !f.holds(run)).map((f) => ({ fence: f.fence, wants: f.wants, got: f.reads(run) ?? null })); +} + +/** The sentence a fence refusal prints — every failing fence by name, what the run read and what the relay re-runs. Pure. */ +export function rerunFailedFenceText(failures) { + return failures.map((f) => `fence \`${f.fence}\`: the run reads ${JSON.stringify(f.got)}, and the relay re-runs only ${f.wants}`).join('; '); +} + +/** The names of the jobs that concluded `failure` in one attempt's job list — the jobs the read-back looks for. Pure. */ +export function failedJobNames(jobs) { + const out = []; + for (const j of Array.isArray(jobs) ? jobs : []) { + if (j?.conclusion === RERUN_FAILED_CONCLUSION && typeof j.name === 'string' && !out.includes(j.name)) out.push(j.name); + } + return out; +} + +/** + * What LANDED means for a `run_rerun_failed` (the header's section): `failed` + * the job names attempt 1 failed, `run` the run read again after the POST, + * `jobs` its attempt-2 job list. Answers `{ ok, pending, attempt, rerun, + * notListed, other, why }`: + * + * rerun `[{ name, id, status }]` — each failed job listed in attempt 2 `queued` or `in_progress`; + * notListed the failed names attempt 2 does not list yet — NAMED, never failed: a job is created when + * its needs finish (measured, header), so an aggregate job behind a re-running shard is absent; + * other `[{ name, id, status, conclusion }]` — a failed job listed in any other state; + * ok attempt 2, at least one `rerun`, and no `other`; + * pending not landed YET — attempt 2 not seen, nothing re-running listed, or a job still in a pre-queue + * state (`waiting`, `requested`, `pending`) — the executor re-reads inside its window; + * neither a FAILED action: a failed job listed `completed` in attempt 2 was not re-run as asked. + * + * Pure. + */ +export function rerunFailedReadBack({ failed = [], run = null, jobs = [] } = {}) { + const want = RERUN_FAILED_ATTEMPT + 1; + const attempt = run?.run_attempt ?? null; + if (attempt !== want) { + return { ok: false, pending: true, attempt, rerun: [], notListed: [...failed], other: [], why: `the run reads run_attempt ${JSON.stringify(attempt)}, not ${want} — the re-run has not shown as a new attempt` }; + } + const listed = (Array.isArray(jobs) ? jobs : []).filter((j) => j && typeof j.name === 'string' && (j.run_attempt === undefined || j.run_attempt === want)); + const rerun = []; + const notListed = []; + const other = []; + for (const name of failed) { + const named = listed.filter((j) => j.name === name); + if (named.length === 0) { + notListed.push(name); + continue; + } + const live = named.find((j) => RERUN_FAILED_RERUN_STATES.includes(j.status)); + if (live) rerun.push({ name, id: live.id ?? null, status: live.status }); + else other.push({ name, id: named[0].id ?? null, status: named[0].status ?? null, conclusion: named[0].conclusion ?? null }); + } + const settled = other.filter((j) => j.status === RERUN_FAILED_STATUS); + if (settled.length) { + return { ok: false, pending: false, attempt, rerun, notListed, other, why: `attempt ${want} lists ${settled.map((j) => `\`${j.name}\` (job ${j.id}) ${j.status}${j.conclusion ? ` ${j.conclusion}` : ''}`).join(', ')} — a job attempt ${RERUN_FAILED_ATTEMPT} failed, not ${RERUN_FAILED_RERUN_STATES.join(' or ')}, so it was not re-run as asked` }; + } + if (other.length) return { ok: false, pending: true, attempt, rerun, notListed, other, why: `attempt ${want} lists ${other.map((j) => `\`${j.name}\` (job ${j.id}) ${j.status}`).join(', ')} — not ${RERUN_FAILED_RERUN_STATES.join(' or ')} yet` }; + if (rerun.length === 0) return { ok: false, pending: true, attempt, rerun, notListed, other, why: `attempt ${want} lists none of the ${failed.length} job(s) attempt ${RERUN_FAILED_ATTEMPT} failed yet` }; + return { ok: true, pending: false, attempt, rerun, notListed, other, why: '' }; +} + +/** The read-back line a landing prints: the new attempt and its job ids, then what is not listed yet. Pure. */ +export function rerunFailedLandedText(v) { + const jobs = v.rerun.map((j) => `\`${j.name}\` job ${j.id} ${j.status}`).join(', '); + const later = v.notListed.length ? ` · not listed yet: ${v.notListed.map((n) => `\`${n}\``).join(', ')} (a job is listed when its needs finish)` : ''; + return `attempt ${v.attempt} · re-run: ${jobs}${later}`; +} + /** The seat-side transport selector and its three values. */ export const TRANSPORT_ENV = 'OS_FLEET_TRANSPORT'; export const TRANSPORTS = Object.freeze(['direct', 'dispatch', 'auto']); @@ -398,6 +582,7 @@ export const FIELDS = Object.freeze({ state: Object.freeze({ kind: 'enum', values: Object.freeze(['open', 'closed']) }), state_reason: Object.freeze({ kind: 'enum', values: Object.freeze(['completed', 'not_planned', 'duplicate', 'reopened']) }), target_repo: Object.freeze({ kind: 'enum', values: TRANSFER_TARGETS }), + run_id: Object.freeze({ kind: 'int' }), workflow: Object.freeze({ kind: 'enum', values: WORKFLOW_DISPATCH_ALLOWLIST }), ref: Object.freeze({ kind: 'enum', values: Object.freeze([WORKFLOW_DISPATCH_REF]) }), inputs: Object.freeze({ kind: 'inputs' }), @@ -413,14 +598,15 @@ export const FIELDS = Object.freeze({ * alone (`alsoSpends`: a pull changing `.github/workflows/**` is enqueued only * by a token carrying it), a literal grant on every stroke by the * maintainer's ruling, comment 6104822731 (header). `actions: write` is spent - * by the `workflow_dispatch` row alone, and minted only for a stroke carrying - * it (`STROKE_SCOPED_PERMISSIONS`). + * by the `workflow_dispatch` and `run_rerun_failed` rows alone, and minted + * only for a stroke carrying one of them (`STROKE_SCOPED_PERMISSIONS`). */ export const PERMISSIONS = Object.freeze({ issues: 'write', 'pull-requests': 'write', contents: 'write', workflows: 'write', metadata: 'read', actions: 'write' }); /** * The permissions minted ONLY for a stroke carrying an op that spends them — - * `actions: write`, the `workflow_dispatch` row's alone. `validate.mjs` + * `actions: write`, the `workflow_dispatch` and `run_rerun_failed` rows' + * alone, each of which travels alone in its stroke. `validate.mjs` * computes a stroke's permission set (`tokenPermissionsOf`) and writes each of * these to `$GITHUB_OUTPUT` as `permission_=` or empty; the * workflow's mint step reads that output for exactly these inputs, and @@ -530,7 +716,8 @@ export function transferRemedy(source, target) { * `secondRepo` (optional) the key naming a SECOND repository the token * must reach — `transfer`'s alone (header above); * `alone` (optional) true for a row whose stroke carries exactly that - * one action — `workflow_dispatch`'s alone (header above); + * one action — `workflow_dispatch`'s and `run_rerun_failed`'s, + * the two rows spending `actions: write` (header above); * `closedInputs` (optional) the per-workflow closed inputs — * `WORKFLOW_DISPATCH_INPUTS`, `workflow_dispatch`'s alone: the * action's `inputs` must be exactly the entry its `workflow` @@ -548,14 +735,17 @@ export function transferRemedy(source, target) { * the executor reads the pull's files and sends what * `governedReviewRequests` spells, on a Tier H hit alone; * `requests` the request descriptors the executor issues, in order: - * `{ verb, path, body?, idempotent404?, graphql?, updateBranch?, fromAnswer?, assigned?, labelled? }` + * `{ verb, path, body?, idempotent404?, graphql?, updateBranch?, rerunFailed?, fromAnswer?, assigned?, labelled? }` * — a `graphql` descriptor names the mutation and the pull (or * the issue and target) whose node ids it needs; a pull's * descriptor also carries `landed`, its `PR_LANDED_STATE` * entry, which is how the executor tells a landing from a * bare 200; an `updateBranch` descriptor carries the pull and * the expected head `confirmUpdateBranch` (execute.mjs) judges - * the answer against; a `fromAnswer` descriptor names the + * the answer against; a `rerunFailed` descriptor carries the + * run whose fences the executor reads before the POST and + * whose new attempt it reads back after it + * (`confirmRerunFailed`, execute.mjs); a `fromAnswer` descriptor names the * field of its action's FIRST request's answer that fills the * `answerPlaceholder` in its path (`fillFromAnswer`, * execute.mjs); an `assigned` descriptor lists the logins its @@ -720,6 +910,17 @@ export const OPS = Object.freeze({ return [{ verb: 'POST', path: `/repos/${repo}/actions/workflows/${enc(a.workflow)}/dispatches`, body: { ref: a.ref, ...(inputs ? { inputs } : {}) } }]; }, }), + // The second row whose token carries `actions: write`, minted for a stroke carrying this op alone — see the header. + // `run_id` is its one key. The four fences on the run (`RERUN_FAILED_FENCES`) are judged by the executor on the run's + // own answer, read immediately before this POST, and the POST carries NO body: the platform's optional + // debug-logging flag is never sent. + run_rerun_failed: Object.freeze({ + permission: 'actions', + required: Object.freeze(['run_id']), + optional: Object.freeze([]), + alone: true, + requests: (a, repo) => [{ verb: 'POST', path: `/repos/${repo}/actions/runs/${a.run_id}/rerun-failed-jobs`, rerunFailed: { runId: a.run_id } }], + }), }); export const OP_NAMES = Object.freeze(Object.keys(OPS)); @@ -754,11 +955,13 @@ export const REFUSED_PATH_FAMILIES = Object.freeze([ /\/git\//, /\/branches\//, /\/contents\//, - // Every `actions/*` path but the one dispatch call (`WORKFLOW_DISPATCH_PATH_SHAPE`), then the run, - // job, cache, artifact, secret / variable / runner and workflow enable / disable endpoints - // `actions: write` would otherwise reach, each named — a loosened catch-all still reds on them. - /\/actions\/(?!workflows\/[^/]+\/dispatches$)/, - /\/actions\/runs\//, + // Every `actions/*` path but the dispatch call (`WORKFLOW_DISPATCH_PATH_SHAPE`) and the failed-jobs + // re-run (`RERUN_FAILED_PATH_SHAPE`), then the run, job, cache, artifact, secret / variable / runner + // and workflow enable / disable endpoints `actions: write` would otherwise reach, each named — a + // loosened catch-all still reds on them. The run family refuses every run path but exactly + // `runs/{digits}/rerun-failed-jobs`: the whole-run `rerun`, cancel, delete, logs and attempts stay refused. + /\/actions\/(?!workflows\/[^/]+\/dispatches$|runs\/\d+\/rerun-failed-jobs$)/, + /\/actions\/runs\/(?!\d+\/rerun-failed-jobs$)/, /\/actions\/jobs\//, /\/actions\/caches/, /\/actions\/artifacts/, diff --git a/scripts/pm/fleet-write/validate.mjs b/scripts/pm/fleet-write/validate.mjs index fdde9a9c7aa..fc766a293d1 100644 --- a/scripts/pm/fleet-write/validate.mjs +++ b/scripts/pm/fleet-write/validate.mjs @@ -49,6 +49,14 @@ * and value, or no `inputs` key where the entry is `null`. A missing * input, another value (`refresh_version_pr: false` on `release.yml`, the * publish-repair lane) or any other input (`force`) is refused by name; + * - a stroke carrying a `run_rerun_failed` carries exactly that one action + * too (the row's `alone`), for the same reason: its token carries + * `actions: write`. Its one key is `run_id`, a positive integer; a + * workflow, an attempt, a job, an event or the platform's debug-logging + * flag beside it is refused by name. The four fences on the RUN — the + * workflow, the event, once, failed (`RERUN_FAILED_FENCES` in `ops.mjs`) + * — are facts this file cannot see: the executor judges them on the run + * read immediately before its POST; * - a `pr_create` may carry `assignees` and `labels`, the typed lists * `assign` and `labels_add` take — judged here as every list is — and the * op table turns each into a later request of the SAME action, sent by @@ -63,7 +71,9 @@ * target, the only such row. It also writes one `permission_=` line per * grant in `STROKE_SCOPED_PERMISSIONS` (`tokenPermissionsOf`): the level when * an action in the stroke spends it, empty otherwise — `permission_actions=write` - * for a `workflow_dispatch` stroke, `permission_actions=` for every other. The + * for a `workflow_dispatch` or `run_rerun_failed` stroke, `permission_actions=` + * for every other. So the second row rides the workflow's existing input and + * the workflow names neither op. The * workflow reads those outputs and never the payload, and `--self-test` pins * the list to ONE name for every other op, two for a transfer, the grant to * the one row, and the workflow lines that read both. @@ -103,6 +113,14 @@ import { RELAY_EVENT_TYPE, RELAY_FILES, REQUEST_ID_SHAPE, + RERUN_FAILED_ATTEMPT, + RERUN_FAILED_CONCLUSION, + RERUN_FAILED_EVENT, + RERUN_FAILED_FENCES, + RERUN_FAILED_PATH_SHAPE, + RERUN_FAILED_RERUN_STATES, + RERUN_FAILED_STATUS, + RERUN_FAILED_WORKFLOWS, SESSION_SHAPE, SHA_SHAPE, STROKE_SCOPED_PERMISSIONS, @@ -116,7 +134,12 @@ import { GOVERNED_REVIEW_OPS, answerPlaceholder, dispatchInputsOnTheWire, + failedJobNames, governedReviewRequests, + rerunFailedFenceFailures, + rerunFailedFenceText, + rerunFailedLandedText, + rerunFailedReadBack, } from './ops.mjs'; const SELF_PATH = fileURLToPath(import.meta.url); @@ -244,9 +267,10 @@ export function closedInputsErrors(action, declared, where) { * The rules that span actions — the two rows that widen the token. The one * whose token reaches a second repository (`secondRepo`, `transfer`'s alone): * such a stroke carries that op alone, names ONE second repository, never - * `repo` itself, and each card once. The one whose token carries a grant no - * other row spends (`alone`, `workflow_dispatch`'s): such a stroke carries - * exactly that one action. Runs only over actions that passed + * `repo` itself, and each card once. The ones whose token carries a grant no + * other row spends (`alone`, `workflow_dispatch`'s and `run_rerun_failed`'s, + * the two that spend `actions: write`): such a stroke carries exactly that + * one action. Runs only over actions that passed * `validateAction`. Pure. */ export function strokeErrors(repo, actions) { @@ -309,7 +333,7 @@ export function tokenRepositoriesOf(payload) { * The grants the relay's token must carry for a JUDGED payload: every entry of * `PERMISSIONS` but the stroke-scoped ones, plus each stroke-scoped grant some * action's op spends. `{ …five grants }` for every stroke but a - * `workflow_dispatch` one, which adds `actions: write`. Pure. + * `workflow_dispatch` or `run_rerun_failed` one, which adds `actions: write`. Pure. */ export function tokenPermissionsOf(payload) { const spent = new Set(payload.actions.map((a) => OPS[a.op].permission)); @@ -750,10 +774,12 @@ const SELF_TEST_BATTERIES = Object.freeze({ 'the CLI: a file, the environment, GitHub outputs, and the exit ladder': 10, 'the transfer row: an issue never a pull, a target from the governed roster never the source, a stroke of its own to one target, labels never created': 12, 'the token scope: ONE repository for every op but transfer, source + target for a transfer stroke, and the workflow mints from that list': 8, - 'the workflow_dispatch row: a closed allowlist of files matched whole, each with its closed inputs, ref main alone, a stroke of exactly that one action, actions write minted for that stroke alone, and the dispatch call the ONLY actions path any row reaches': 18, + 'the workflow_dispatch row: a closed allowlist of files matched whole, each with its closed inputs, ref main alone, a stroke of exactly that one action, actions write minted for that stroke alone, and the dispatch call and the failed-jobs re-run the ONLY actions paths any row reaches': 18, "the release.yml entry: refresh_version_pr true and nothing beside it — force, false, absent, a string or any other input refused by name; the wire spells the table's value as a string; every entry's inputs declared on disk; and the file's own guards, evaluated, start the bookkeeping lane alone": 22, + "the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the four fences on the run, each accepted and refused; the read-back's judgment; and every other actions path still refused": 30, }); -const SELF_TEST_BATTERY_FLOOR = 11; +// Raised 11 → 12 with the run_rerun_failed battery: one battery, one row. +const SELF_TEST_BATTERY_FLOOR = 12; const UNATTRIBUTED_BATTERY = '(unattributed)'; const batteryCases = new Map(); @@ -887,6 +913,7 @@ export async function selfTest() { automerge_disable: { pull: 1 }, transfer: { issue: 1, target_repo: `${TARGET_OWNER}/objectui` }, workflow_dispatch: { workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF }, + run_rerun_failed: { run_id: 1 }, }; t('every op in the table has a minimal shape this validator accepts', OP_NAMES.filter((op) => !validatePayload(one({ op, ...minimal[op] })).ok), []); t('…and the minimal-shape ledger names every op, no more and no less', Object.keys(minimal).sort(), [...OP_NAMES].sort()); @@ -911,7 +938,7 @@ export async function selfTest() { { const sample = { issue: 7, pull: 7, comment_id: 7, body: 'b', title: 't', head: 'h', base: 'main', labels: ['a b'], assignees: ['u'], reviewers: ['u'], team_reviewers: ['t'], state: 'closed', state_reason: 'completed', expected_head_sha: SHA, target_repo: `${TARGET_OWNER}/objectui`, - workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF, + workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF, run_id: 7, }; const everyRequest = OP_NAMES.flatMap((op) => { const spec = OPS[op]; @@ -1047,6 +1074,7 @@ export async function selfTest() { assign: { issue: 1, assignees: ['u'] }, unassign: { issue: 1, assignees: ['u'] }, issue_patch: { issue: 1, state: 'closed' }, issue_create: { title: 't', body: 'b' }, pr_create: { title: 't', head: 'h', base: 'main' }, pr_request_reviewers: { pull: 1, reviewers: ['u'] }, pr_update_branch: { pull: 1, expected_head_sha: SHA }, pr_ready: { pull: 1 }, pr_draft: { pull: 1 }, automerge_enable: { pull: 1 }, automerge_disable: { pull: 1 }, workflow_dispatch: { workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF }, + run_rerun_failed: { run_id: 1 }, }; const narrow = OP_NAMES.filter((op) => !OPS[op].secondRepo); t('transfer is the ONLY row that reaches a second repository', OP_NAMES.filter((op) => OPS[op].secondRepo), ['transfer']); @@ -1063,7 +1091,7 @@ export async function selfTest() { } // ── the workflow_dispatch row ───────────────────────────────────────────── - battery('the workflow_dispatch row: a closed allowlist of files matched whole, each with its closed inputs, ref main alone, a stroke of exactly that one action, actions write minted for that stroke alone, and the dispatch call the ONLY actions path any row reaches'); + battery('the workflow_dispatch row: a closed allowlist of files matched whole, each with its closed inputs, ref main alone, a stroke of exactly that one action, actions write minted for that stroke alone, and the dispatch call and the failed-jobs re-run the ONLY actions paths any row reaches'); { const REPO = `${TARGET_OWNER}/objectstack`; const FILE = WORKFLOW_DISPATCH_ALLOWLIST[0]; @@ -1089,24 +1117,24 @@ export async function selfTest() { t('an inputs key on a file whose entry is null is refused BY NAME — the relay sends none there', refuses(one(start({ inputs: { reason: true } })), `actions[0].inputs: ${FILE} declares no inputs`)); t('⛔ a workflow_dispatch beside any other op is refused — the stroke carries that one action alone', refuses(fixturePayload({ actions: [start(), { op: 'comment', issue: 7, body: 'x' }] }), 'actions[1] is `comment` in a stroke that carries a `workflow_dispatch`')); t('⛔ two workflow_dispatch in one stroke are refused too', refuses(fixturePayload({ actions: [start(), start()] }), 'a stroke carries exactly ONE `workflow_dispatch` and nothing else')); - t('the only rows that travel alone or widen the token: workflow_dispatch (alone) and transfer (second repository)', [OP_NAMES.filter((op) => OPS[op].alone), OP_NAMES.filter((op) => OPS[op].secondRepo)], [['workflow_dispatch'], ['transfer']]); - t('…and actions is spent by the workflow_dispatch row and NOTHING else', OP_NAMES.filter((op) => OPS[op].permission === 'actions'), ['workflow_dispatch']); + t('the only rows that travel alone or widen the token: workflow_dispatch and run_rerun_failed (alone) and transfer (second repository)', [OP_NAMES.filter((op) => OPS[op].alone), OP_NAMES.filter((op) => OPS[op].secondRepo)], [['workflow_dispatch', 'run_rerun_failed'], ['transfer']]); + t('…and actions is spent by the workflow_dispatch and run_rerun_failed rows and NOTHING else — each of them alone in its stroke', [OP_NAMES.filter((op) => OPS[op].permission === 'actions'), OP_NAMES.filter((op) => OPS[op].permission === 'actions').every((op) => OPS[op].alone === true)], [['workflow_dispatch', 'run_rerun_failed'], true]); const plain = validatePayload(fixturePayload()).payload; const run = validatePayload(one(start())).payload; t('the token grants: a comment stroke carries the five literal grants and no actions; a workflow_dispatch stroke adds actions write', [tokenPermissionsOf(plain), tokenPermissionsOf(run)], [{ issues: 'write', 'pull-requests': 'write', contents: 'write', workflows: 'write', metadata: 'read' }, { issues: 'write', 'pull-requests': 'write', contents: 'write', workflows: 'write', metadata: 'read', actions: 'write' }]); t('…and the GitHub output spells that as permission_actions=write for the run stroke, EMPTY for every other — the mint action skips an empty input', [githubOutputLines(run).split('\n').at(-2), githubOutputLines(plain).split('\n').at(-2), [...STROKE_SCOPED_PERMISSIONS]], ['permission_actions=write', 'permission_actions=', ['actions']]); // The fence on the grant: every path every row can spell, against the actions families — the dispatch call is the ONE // actions path reached; and a POSITIVE CONTROL that the families would catch what actions: write otherwise reaches. - const filled = { issue: 7, pull: 7, comment_id: 7, body: 'b', title: 't', head: 'h', base: 'main', labels: ['a'], assignees: ['u'], reviewers: ['u'], team_reviewers: ['t'], state: 'closed', state_reason: 'completed', expected_head_sha: SHA, target_repo: `${TARGET_OWNER}/objectui`, workflow: FILE, ref: WORKFLOW_DISPATCH_REF }; + const filled = { issue: 7, pull: 7, comment_id: 7, body: 'b', title: 't', head: 'h', base: 'main', labels: ['a'], assignees: ['u'], reviewers: ['u'], team_reviewers: ['t'], state: 'closed', state_reason: 'completed', expected_head_sha: SHA, target_repo: `${TARGET_OWNER}/objectui`, workflow: FILE, ref: WORKFLOW_DISPATCH_REF, run_id: 7 }; const everyPath = OP_NAMES.flatMap((op) => { const a = { op }; for (const k of [...OPS[op].required, ...OPS[op].optional]) a[k] = filled[k]; return OPS[op].requests(a, REPO).map((r) => r.path); }); const actionsPaths = everyPath.filter((p) => p.includes('/actions/')); - t('⛔ the ONLY actions/* path any row reaches is the dispatch call on an allowlisted file', [actionsPaths, actionsPaths.every((p) => WORKFLOW_DISPATCH_PATH_SHAPE.test(p))], [[`/repos/${REPO}/actions/workflows/${FILE}/dispatches`], true]); - const forbidden = ['runs/1/cancel', 'runs/1/rerun', 'runs/1/rerun-failed-jobs', 'runs/1', 'runs/1/logs', 'runs/1/approve', 'runs/1/pending_deployments', 'jobs/1/rerun', 'caches', 'caches/1', 'artifacts/1', `workflows/${FILE}/enable`, `workflows/${FILE}/disable`, `workflows/${FILE}`, 'secrets/X', 'variables/X', 'permissions', 'runners/1', `workflows/${FILE}/dispatches/x`]; - t('positive control: every cancel / re-run / delete / log / approve / cache / artifact / enable / disable / secret / variable / runner path under actions/* is caught by a refused family', forbidden.filter((tail) => !REFUSED_PATH_FAMILIES.some((re) => re.test(`/repos/${REPO}/actions/${tail}`))), []); + t('⛔ the ONLY actions/* paths any row reaches are the dispatch call on an allowlisted file and the failed-jobs re-run of a run named by its id', [actionsPaths, actionsPaths.every((p) => WORKFLOW_DISPATCH_PATH_SHAPE.test(p) || RERUN_FAILED_PATH_SHAPE.test(p))], [[`/repos/${REPO}/actions/workflows/${FILE}/dispatches`, `/repos/${REPO}/actions/runs/7/rerun-failed-jobs`], true]); + const forbidden = ['runs/1/cancel', 'runs/1/force-cancel', 'runs/1/rerun', 'runs/1/rerun-failed-jobs/x', 'runs/x/rerun-failed-jobs', 'runs/1/attempts/1/rerun-failed-jobs', 'runs/1', 'runs/1/logs', 'runs/1/approve', 'runs/1/pending_deployments', 'jobs/1/rerun', 'caches', 'caches/1', 'artifacts/1', `workflows/${FILE}/enable`, `workflows/${FILE}/disable`, `workflows/${FILE}`, 'secrets/X', 'variables/X', 'permissions', 'runners/1', `workflows/${FILE}/dispatches/x`]; + t('positive control: every cancel / whole-run or job re-run / delete / log / approve / cache / artifact / enable / disable / secret / variable / runner path under actions/* — and a re-run path with anything around its run id — is caught by a refused family', forbidden.filter((tail) => !REFUSED_PATH_FAMILIES.some((re) => re.test(`/repos/${REPO}/actions/${tail}`))), []); } // ── the release.yml entry ──────────────────────────────────────────────── @@ -1166,6 +1194,93 @@ export async function selfTest() { t('⛔ the pin can fail: a release-integrity guard that lost its negation, or a seventh job with no guard, is started by the relay\'s dispatch', [unguarded !== releaseText && started(unguarded, 'workflow_dispatch', REFRESH).runs.includes('release-integrity'), started(added, 'workflow_dispatch', REFRESH).runs.includes('extra-lane')], [true, true]); } + // ── the run_rerun_failed row ────────────────────────────────────────────── + battery("the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the four fences on the run, each accepted and refused; the read-back's judgment; and every other actions path still refused"); + { + const REPO = `${TARGET_OWNER}/objectstack`; + const RUN = 39713789453; + const spec = OPS.run_rerun_failed; + const rerun = (extra = {}) => ({ op: 'run_rerun_failed', run_id: RUN, ...extra }); + const one = (action) => fixturePayload({ actions: [action] }); + // The row and its one request. + t('the row: run_id required and nothing optional, an integer field, spending actions, travelling alone, reaching no second repository, with no closed inputs', [spec.required, spec.optional, FIELDS.run_id, spec.permission, spec.alone, spec.secondRepo ?? null, spec.closedInputs ?? null], [['run_id'], [], { kind: 'int' }, 'actions', true, null, null]); + const req = spec.requests(rerun(), REPO); + t("its one request is POST /repos/{repo}/actions/runs/{run_id}/rerun-failed-jobs with NO body — the platform's debug-logging flag is never sent — carrying the run the executor fences and reads back", [req.length, req[0].verb, req[0].path, 'body' in req[0], req[0].rerunFailed], [1, 'POST', `/repos/${REPO}/actions/runs/${RUN}/rerun-failed-jobs`, false, { runId: RUN }]); + t('…a path the re-run shape names and no refused family reaches', [RERUN_FAILED_PATH_SHAPE.test(req[0].path), REFUSED_PATH_FAMILIES.filter((re) => re.test(req[0].path)).map(String)], [true, []]); + // Accepted, and refused by value and by key — the validator's half of "nothing widens it". + const ok = validatePayload(one(rerun())); + t('accepted: the minimal stroke, whose normalised action is op and run_id alone', [ok.ok, ok.payload?.actions[0]], [true, { op: 'run_rerun_failed', run_id: RUN }], ok.errors.join(' | ')); + t('⛔ refused: run_id as a numeric string, zero, a negative, a float or an object — the integer kind, judged here', [String(RUN), 0, -1, 1.5, { id: 1 }].map((run_id) => refuses(one(rerun({ run_id })), 'actions[0].run_id must be a positive integer')), [true, true, true, true, true]); + t('⛔ refused: a missing run_id — the run is named on every request, never defaulted', refuses(one({ op: 'run_rerun_failed' }), 'actions[0].run_id is required by `run_rerun_failed`')); + t('⛔ refused BY NAME: a workflow, an attempt, a job, an event, a ref or the debug-logging flag beside run_id — no request-time key widens the fences', ['workflow', 'run_attempt', 'job_id', 'event', 'ref', 'enable_debug_logging'].map((key) => refuses(one(rerun({ [key]: key === 'workflow' ? 'ci.yml' : 1 })), `actions[0].${key} is not a key \`run_rerun_failed\` takes (it takes run_id)`)), [true, true, true, true, true, true]); + // Fence: alone in its stroke. + t('⛔ fence alone, refused: beside a comment, beside a workflow_dispatch, and two of them — each stroke refused naming the action that does not belong', [ + refuses(fixturePayload({ actions: [rerun(), { op: 'comment', issue: 7, body: 'x' }] }), 'actions[1] is `comment` in a stroke that carries a `run_rerun_failed`'), + refuses(fixturePayload({ actions: [rerun(), { op: 'workflow_dispatch', workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF }] }), 'actions[1] is `workflow_dispatch` in a stroke that carries a `run_rerun_failed`'), + refuses(fixturePayload({ actions: [rerun(), rerun({ run_id: 2 })] }), 'a stroke carries exactly ONE `run_rerun_failed` and nothing else'), + ], [true, true, true]); + t('fence alone, accepted: the stroke of that one action passes the stroke rules', strokeErrors(REPO, [rerun()]), []); + // Fence: the stroke-scoped grant — ruling 3, carried by the existing input with no workflow edit. + const stroke = ok.payload; + t('fence grant, accepted side: its stroke mints the five literal grants plus actions write, for ONE repository', stroke ? [tokenPermissionsOf(stroke), tokenRepositoriesOf(stroke)] : 'refused', [{ issues: 'write', 'pull-requests': 'write', contents: 'write', workflows: 'write', metadata: 'read', actions: 'write' }, [REPO]]); + t('⭐ …and the GitHub output spells permission_actions=write for its stroke, exactly as for a workflow_dispatch stroke', stroke ? [githubOutputLines(stroke).split('\n').at(-2), githubOutputLines(validatePayload(one({ op: 'workflow_dispatch', workflow: WORKFLOW_DISPATCH_ALLOWLIST[0], ref: WORKFLOW_DISPATCH_REF })).payload).split('\n').at(-2)] : 'refused', ['permission_actions=write', 'permission_actions=write']); + const minimalOf = { + comment: { issue: 1, body: 'b' }, comment_edit: { comment_id: 1, body: 'b' }, labels_add: { issue: 1, labels: ['a'] }, labels_remove: { issue: 1, labels: ['a'] }, + assign: { issue: 1, assignees: ['u'] }, unassign: { issue: 1, assignees: ['u'] }, issue_patch: { issue: 1, state: 'closed' }, issue_create: { title: 't', body: 'b' }, + pr_create: { title: 't', head: 'h', base: 'main' }, pr_request_reviewers: { pull: 1, reviewers: ['u'] }, pr_update_branch: { pull: 1, expected_head_sha: SHA }, pr_ready: { pull: 1 }, pr_draft: { pull: 1 }, + automerge_enable: { pull: 1 }, automerge_disable: { pull: 1 }, transfer: { issue: 1, target_repo: `${TARGET_OWNER}/objectui` }, + }; + const spendsNoActions = OP_NAMES.filter((op) => OPS[op].permission !== 'actions'); + t('…the ledger of ops spending no actions grant is every op but the two that do, no more and no less', [Object.keys(minimalOf).sort(), [...spendsNoActions].sort()], [[...spendsNoActions].sort(), [...spendsNoActions].sort()]); + const scopedLine = (actions) => { + const v = validatePayload(fixturePayload({ actions })); + return v.ok ? githubOutputLines(v.payload).split('\n').at(-2) : `refused: ${v.errors.join(' | ')}`; + }; + t('⛔ fence grant, refused side: every op spending no actions grant writes an EMPTY permission_actions — each alone in its stroke, and the stroke of them all that shares one token', [spendsNoActions.filter((op) => scopedLine([{ op, ...minimalOf[op] }]) !== 'permission_actions='), scopedLine(spendsNoActions.filter((op) => !OPS[op].secondRepo).map((op) => ({ op, ...minimalOf[op] })))], [[], 'permission_actions=']); + const repoRoot = resolve(SELF_PATH, '../../../..'); + const workflowText = readFileSync(resolve(repoRoot, '.github/workflows/fleet-write.yml'), 'utf8'); + const mintsFromOutput = [...workflowText.matchAll(/^\s+permission-actions:\s*\$\{\{\s*steps\.validate\.outputs\.permission_actions\s*\}\}\s*$/gm)].length; + t('⭐ ruling 3, no workflow edit: fleet-write.yml mints permission-actions from the validator\'s permission_actions output — the one input this row rides — and names neither this op nor its path', [mintsFromOutput, workflowText.includes('run_rerun_failed'), workflowText.includes('rerun-failed-jobs')], [1, false, false]); + // The four fences on the run — the closed values, then each accepted and refused on the run's own answer. + t('the fences on the run: the closed workflow list is exactly ci.yml, frozen; the event pull_request; the attempt 1; completed with failure; and the read-back accepts queued and in_progress alone', [[...RERUN_FAILED_WORKFLOWS], Object.isFrozen(RERUN_FAILED_WORKFLOWS), RERUN_FAILED_EVENT, RERUN_FAILED_ATTEMPT, RERUN_FAILED_STATUS, RERUN_FAILED_CONCLUSION, [...RERUN_FAILED_RERUN_STATES], RERUN_FAILED_FENCES.map((f) => f.fence)], [['.github/workflows/ci.yml'], true, 'pull_request', 1, 'completed', 'failure', ['queued', 'in_progress'], ['workflow', 'event', 'once', 'failed']]); + t('every listed workflow is a file on disk that runs on pull_request — so the list names a run the fences can accept', RERUN_FAILED_WORKFLOWS.map((p) => [p, existsSync(resolve(repoRoot, p)) && /^ {2}pull_request:/m.test(readFileSync(resolve(repoRoot, p), 'utf8'))]), RERUN_FAILED_WORKFLOWS.map((p) => [p, true])); + // The measured run as its attempt 1 read before the maintainer re-ran it (run 39713789453; fields as the run endpoint answers them). + const MEASURED = { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure' }; + const fenced = (patch) => rerunFailedFenceFailures({ ...MEASURED, ...patch }).map((f) => f.fence); + t('accepted: the measured run passes all four fences', rerunFailedFenceFailures(MEASURED), []); + t('⛔ fence workflow, refused: another workflow, the bare file name, a ref-suffixed path, a missing path — each naming the workflow fence alone', [{ path: '.github/workflows/lint.yml' }, { path: 'ci.yml' }, { path: '.github/workflows/ci.yml@refs/heads/main' }, { path: undefined }].map(fenced), [['workflow'], ['workflow'], ['workflow'], ['workflow']]); + t('⛔ fence event, refused: push, merge_group, schedule, release, workflow_dispatch and pull_request_target runs — each naming the event fence alone', ['push', 'merge_group', 'schedule', 'release', 'workflow_dispatch', 'pull_request_target'].map((event) => fenced({ event })), [['event'], ['event'], ['event'], ['event'], ['event'], ['event']]); + t('⛔ fence once, refused: attempt 2, attempt 3, the string "1" and a missing attempt — each naming the once fence alone', [{ run_attempt: 2 }, { run_attempt: 3 }, { run_attempt: '1' }, { run_attempt: undefined }].map(fenced), [['once'], ['once'], ['once'], ['once']]); + t('⛔ fence failed, refused: still in progress, completed with success, completed cancelled, completed with no conclusion — each naming the failed fence alone', [{ status: 'in_progress', conclusion: null }, { conclusion: 'success' }, { conclusion: 'cancelled' }, { conclusion: null }].map(fenced), [['failed'], ['failed'], ['failed'], ['failed']]); + const all = rerunFailedFenceFailures({ path: '.github/workflows/release.yml', event: 'push', run_attempt: 2, status: 'completed', conclusion: 'success' }); + t('every failing fence is named in ONE pass, and the sentence quotes what the run read beside what the relay re-runs', [all.map((f) => f.fence), rerunFailedFenceText(all).includes('fence `once`: the run reads 2'), rerunFailedFenceText(all).includes('"push"'), rerunFailedFenceText(all).split('; ').length], [['workflow', 'event', 'once', 'failed'], true, true, 4]); + // The read-back's judgment, on the measured attempts (jobs by name and state, as the attempts endpoint lists them). + const attempt1 = [ + { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + { id: 119611140839, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }, + { id: 119611142366, name: 'Console Pin Gate', run_attempt: 1, status: 'completed', conclusion: 'skipped' }, + { id: 119617410962, name: 'Test Core', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + ]; + const failed = failedJobNames(attempt1); + t('the baseline: the jobs attempt 1 failed, by name — the shard and the aggregate that needs it; a pass or a skip is not one', failed, ['Test Core (3/6)', 'Test Core']); + const rerunMoment = [ + { id: 119642047246, name: 'Test Core (3/6)', run_attempt: 2, status: 'queued', conclusion: null }, + { id: 119642048282, name: 'Test Core (1/6)', run_attempt: 2, status: 'completed', conclusion: 'success' }, + ]; + const landed = rerunFailedReadBack({ failed, run: { run_attempt: 2 }, jobs: rerunMoment }); + t('⭐ lands: attempt 2, the failed shard queued, the aggregate not listed yet — NAMED, never failed; the line names the new attempt and its job id', [landed.ok, landed.pending, landed.notListed, rerunFailedLandedText(landed)], [true, false, ['Test Core'], 'attempt 2 · re-run: `Test Core (3/6)` job 119642047246 queued · not listed yet: `Test Core` (a job is listed when its needs finish)']); + t('…and in_progress lands as queued does', rerunFailedReadBack({ failed, run: { run_attempt: 2 }, jobs: [{ ...rerunMoment[0], status: 'in_progress' }] }).ok, true); + const notYet = rerunFailedReadBack({ failed, run: { run_attempt: 1 }, jobs: rerunMoment }); + t('⛔ a read that does not see attempt 2 is not a landing — pending inside the window, and its reason names the attempt read', [notYet.ok, notYet.pending, notYet.why.includes('run_attempt 1, not 2')], [false, true, true]); + const carried = rerunFailedReadBack({ failed, run: { run_attempt: 2 }, jobs: [{ ...rerunMoment[0], status: 'completed', conclusion: 'failure' }] }); + t('⛔ a failed job listed completed in attempt 2 was not re-run as asked — not a landing and not pending: a FAILED action naming the job', [carried.ok, carried.pending, carried.why.includes('`Test Core (3/6)` (job 119642047246) completed failure')], [false, false, true]); + t('…a failed job still waiting, or none of them listed yet, is pending — re-read, never landed early', [rerunFailedReadBack({ failed, run: { run_attempt: 2 }, jobs: [{ ...rerunMoment[0], status: 'waiting' }] }), rerunFailedReadBack({ failed, run: { run_attempt: 2 }, jobs: [rerunMoment[1]] })].map((v) => [v.ok, v.pending]), [[false, true], [false, true]]); + // Every other actions path stays refused, by no row reachable, and a payload naming one as an op writes nothing. + const others = ['runs/7/rerun', 'jobs/7/rerun', 'runs/7/cancel', 'runs/7/force-cancel', 'runs/7', 'runs/7/logs', 'runs/7/attempts/1/logs', 'runs/7/rerun-failed-jobs/x', 'runs/x/rerun-failed-jobs', 'caches', 'caches/7', 'artifacts', 'artifacts/7', 'secrets/X', 'secrets/public-key', 'variables', 'variables/X']; + t('⛔ still refused: the whole-run re-run, a job re-run, cancel, force-cancel, delete, logs, caches, artifacts, secrets and variables — each caught by a refused family', others.filter((tail) => !REFUSED_PATH_FAMILIES.some((re) => re.test(`/repos/${REPO}/actions/${tail}`))), []); + t('⛔ …with zero writes: a payload naming any of them as an op is refused as no op the relay executes', ['run_rerun', 'job_rerun', 'run_cancel', 'run_delete', 'cache_delete', 'artifact_delete', 'secret_set', 'variable_set'].map((op) => refuses(one({ op, run_id: 7 }), 'is not an op the relay executes')), [true, true, true, true, true, true, true, true]); + } + // ── the floor, BEFORE the verdict ───────────────────────────────────────── const declared = Object.keys(SELF_TEST_BATTERIES); const floor = []; @@ -1198,7 +1313,8 @@ export async function selfTest() { `✓ fleet-write/validate self-test: ${cases.length} cases pass across ${declared.length} batteries — a closed envelope, one organization, ` + 'closed keys per op with typed values, the platform\'s own client_payload ceilings pinned with their source, no row that can reach a refused family, ' + 'a token list of ONE repository for every op but transfer, whose second is one governed target, and actions write reaching one dispatch call on one allowlisted file ' + - "with that file's closed inputs, minted for that stroke alone — release.yml's bookkeeping lane and no other, read from its own guards.", + "with that file's closed inputs — release.yml's bookkeeping lane and no other, read from its own guards — or one failed-jobs re-run of a run named by its id alone, " + + 'each minted for its stroke alone, every other actions path still refused.', ); selfTestReachedVerdict = true; return 0; From fc7272873cfa0c980288cd1cabe7a4dfa75498d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 07:00:48 +0000 Subject: [PATCH 2/6] feat(pm/fleet-write): the executor fences run_rerun_failed on the run and reads it back The four fences on the run are judged in the relay, on the run read immediately before the one bodiless POST; a fence that fails, an unread run or an unread or empty baseline sends nothing. The 201 lands only when the run reads back as attempt 2 with the jobs attempt 1 failed queued or in progress, and the row names their new job ids. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- scripts/pm/fleet-write/execute.mjs | 224 +++++++++++++++++++++++++++-- 1 file changed, 216 insertions(+), 8 deletions(-) diff --git a/scripts/pm/fleet-write/execute.mjs b/scripts/pm/fleet-write/execute.mjs index c660b170c9e..0b788f75486 100644 --- a/scripts/pm/fleet-write/execute.mjs +++ b/scripts/pm/fleet-write/execute.mjs @@ -56,8 +56,10 @@ * `refresh_version_pr: "true"` alone for `release.yml`, its bookkeeping lane) * — its row reads `accepted`; the run it started is read back by the SEAT * (`dispatch.mjs`), never here, because the answer carries no run id, and the - * token that request spends `actions: write` on reaches that one path and no - * other (`ops.mjs`, `REFUSED_PATH_FAMILIES`). + * token that request spends `actions: write` on reaches that path and the + * failed-jobs re-run below, and no other (`ops.mjs`, `REFUSED_PATH_FAMILIES`). + * A `run_rerun_failed` is fenced on the run and read back HERE (the section + * of that name below). * The FIRST failure stops the run — later actions are NOT attempted, and the * summary says which — because a seat that dispatched five related writes * must be able to read exactly where the board was left. A 404 on a directed @@ -121,6 +123,37 @@ * Any other status is FAILED as every REST 4xx is. The PUT is paced like * every other write verb. * + * ## The failed-jobs re-run — `run_rerun_failed`, fenced on the RUN and read back + * + * The four fences on the run (`RERUN_FAILED_FENCES` in `ops.mjs`: the + * workflow, the event, once, failed) are facts about the run, so this file is + * where they are AUTHORITATIVE (`rerunFailedPreflight`). It first reads the + * job list of the run's attempt 1, which is the read-back's baseline: the + * names of the jobs that concluded `failure`. It then reads + * `GET /repos/{repo}/actions/runs/{run_id}` — the last request before the + * POST — and judges the four fences on that answer. A run that could not be + * read, a fence that fails (every failing one named, with what the run read), + * a baseline that could not be read whole, or a baseline with no failed job + * stops the action with the POST NOT SENT: its row says `not sent`, zero + * writes, exit 5. A run re-run or changed by anyone between the seat's + * request and this read is refused here. ⛔ The seat wrapper's earlier check + * is advisory and is never trusted in place of this one. + * + * The POST answers 201 with an empty body, which is an acceptance, not a + * landing. `confirmRerunFailed` re-reads the run every `RERUN_FAILED_POLL_MS` + * for up to `RERUN_FAILED_WINDOW_MS` until it shows `run_attempt` 2, then + * reads attempt 2's job list and lands only on `rerunFailedReadBack` + * (`ops.mjs`): every baseline job listed there is `queued` or + * `in_progress`, at least one is, and the row names the new attempt and + * those job ids. A baseline job not listed yet is named on the row, because + * a job is listed when its needs finish. A failed job listed `completed` in + * attempt 2 is a FAILED action. A window that never shows attempt 2, or that + * never settles, is a FAILED action whose sentence says UNCONFIRMED: the + * re-run may still start, so read the run and ⛔ never re-send. A re-send + * would meet attempt 2 at the `once` fence anyway, and be refused there with + * zero writes. Every read is an unpaced GET, and the one write is paced like + * every write verb. + * * ## The assignees and labels legs — `pr_create` with `assignees` / `labels`, the requests that read an earlier answer * * The op table spells each leg's path with a placeholder and `fromAnswer: @@ -212,7 +245,8 @@ * 3 PREREQUISITE NOT MET — no token, no sender, the permission read did * not answer, or the platform was unreachable; zero writes. * 4 the sender is not authorized on the target repo; zero writes. - * 5 an action FAILED; the summary names it and which earlier ones landed. + * 5 an action FAILED; the summary names it and which earlier ones landed — + * a `run_rerun_failed` refused at a fence among them, its POST not sent. * 10 the write throttle refused (its prescription was printed), or a stroke's * rule ⑤ batch could not be recorded; zero writes in that second case. */ @@ -229,7 +263,23 @@ import { scrub } from '../fleet-token.mjs'; import { classifyHttp } from '../label-write.mjs'; import { announceBatch, EXIT_WRITE_PACE_REFUSED, fmtDuration, isWriteMethod, noteResponse, paceFilePath, paceWrite, releaseWriteLease, unrecordableText } from '../write-pace.mjs'; import { ANNOTATED_OPS, parseRelayAnnotation, RELAY_ANNOTATION_PREFIX, relayAnnotationMessage, strokeBudgetRefusal, strokePacing } from './dispatch.mjs'; -import { GOVERNED_REVIEW_FILES_PAGE_CAP, GOVERNED_REVIEW_FILES_PAGE_SIZE, GOVERNED_REVIEW_OPS, OPS, PERMISSIONS, PR_LANDED_STATE, answerPlaceholder, governedReviewRequests, transferRemedy } from './ops.mjs'; +import { + GOVERNED_REVIEW_FILES_PAGE_CAP, + GOVERNED_REVIEW_FILES_PAGE_SIZE, + GOVERNED_REVIEW_OPS, + OPS, + PERMISSIONS, + PR_LANDED_STATE, + RERUN_FAILED_ATTEMPT, + answerPlaceholder, + failedJobNames, + governedReviewRequests, + rerunFailedFenceFailures, + rerunFailedFenceText, + rerunFailedLandedText, + rerunFailedReadBack, + transferRemedy, +} from './ops.mjs'; import { PAYLOAD_ENV, refusalText, tokenRepositoriesOf, validatePayload } from './validate.mjs'; const SELF_PATH = fileURLToPath(import.meta.url); @@ -479,6 +529,82 @@ export async function confirmUpdateBranch(req, answer, payload, api, t, { sleep, return { ok: false, why: `HTTP ${UPDATE_BRANCH_ACCEPTED} accepted, but the head ${head === null ? 'could not be read' : `is still ${shortSha(head)}`} ${windowMs} ms later — UNCONFIRMED: the merge may still land; read the pull before anything else, ⛔ do not re-send` }; } +/** The failed-jobs re-run's read-back window and poll interval (header): the run is re-read until it shows the new attempt. */ +export const RERUN_FAILED_WINDOW_MS = 60_000; +export const RERUN_FAILED_POLL_MS = 3_000; +/** An attempt's job list is walked at the endpoint's page maximum, to a cap — a walk past it is a truncated SUBSET. */ +export const RERUN_FAILED_JOBS_PAGE_SIZE = 100; +export const RERUN_FAILED_JOBS_PAGE_CAP = 10; + +/** + * One attempt's job list, every page: `{ jobs }`, or `{ fail: { call, status, why } }` for a page that did not + * answer 200 with a `jobs` array, or a walk past `RERUN_FAILED_JOBS_PAGE_CAP` pages. Never throws. + */ +async function readAttemptJobs(repo, runId, attempt, api, t) { + const jobs = []; + const call = `GET /repos/${repo}/actions/runs/${runId}/attempts/${attempt}/jobs`; + for (let page = 1; page <= RERUN_FAILED_JOBS_PAGE_CAP; page++) { + const r = await rest(api, `/repos/${repo}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=${RERUN_FAILED_JOBS_PAGE_SIZE}&page=${page}`, {}, t); + if (r.status !== 200 || !Array.isArray(r.json?.jobs)) return { fail: { call, status: r.status, why: `${call} -> HTTP ${r.status}${r.detail ? ` (${r.detail})` : ''}: attempt ${attempt}'s job list could not be read` } }; + jobs.push(...r.json.jobs); + const total = Number.isInteger(r.json.total_count) ? r.json.total_count : null; + if (r.json.jobs.length < RERUN_FAILED_JOBS_PAGE_SIZE || (total !== null && jobs.length >= total)) return { jobs }; + } + return { fail: { call, status: 200, why: `attempt ${attempt}'s job list hit its ${RERUN_FAILED_JOBS_PAGE_CAP}-page cap with ${jobs.length} job(s) collected and more pending — a truncated list is a SUBSET` } }; +} + +/** + * The fences, AUTHORITATIVE (header: "The failed-jobs re-run"): attempt 1's job list (the read-back's baseline), + * then the run — the last read before the POST — judged against `RERUN_FAILED_FENCES`. Returns `{ run, failed }` + * to send, or `{ fail: why }` — the sentence the row prints, the POST NOT SENT. Never throws on a status. + */ +export async function rerunFailedPreflight(req, payload, api, t) { + const { runId } = req.rerunFailed; + const baseline = await readAttemptJobs(payload.repo, runId, RERUN_FAILED_ATTEMPT, api, t); + const run = await rest(api, `/repos/${payload.repo}/actions/runs/${runId}`, {}, t); + if (run.status !== 200 || !run.json || typeof run.json !== 'object') { + return { fail: `${run.call} -> HTTP ${run.status}${run.detail ? ` (${run.detail})` : ''}: the run could not be read, so its fences were not judged — nothing sent` }; + } + const failures = rerunFailedFenceFailures(run.json); + if (failures.length) return { fail: `REFUSED at the relay's fence on run ${runId}, nothing sent — ${rerunFailedFenceText(failures)}` }; + if (baseline.fail) return { fail: `${baseline.fail.why}, so the read-back has no baseline — nothing sent` }; + const failed = failedJobNames(baseline.jobs); + if (!failed.length) return { fail: `attempt ${RERUN_FAILED_ATTEMPT} of run ${runId} lists ${baseline.jobs.length} job(s) and none concluded failure, so a re-run's read-back would have nothing to confirm — nothing sent` }; + return { run: run.json, failed }; +} + +/** + * Did the re-run LAND? Judged by reading the run back, never by the 201 (header): the run re-read until it shows + * the new attempt, then that attempt's job list judged by `rerunFailedReadBack`. Returns `{ ok, why, result }` — + * `result` the row's text on a landing, naming the new attempt's job ids. Never throws on a status. + */ +export async function confirmRerunFailed(req, answer, pre, payload, api, t, { sleep, now, windowMs = RERUN_FAILED_WINDOW_MS, pollMs = RERUN_FAILED_POLL_MS } = {}) { + const platform = `HTTP ${answer.status}${answer.detail ? ` — ${answer.detail}` : ''}`; + if (!(answer.status >= 200 && answer.status < 300)) return { ok: false, why: platform }; + const { runId } = req.rerunFailed; + const want = RERUN_FAILED_ATTEMPT + 1; + const started = now(); + let last = 'nothing was read back'; + for (;;) { + const run = await rest(api, `/repos/${payload.repo}/actions/runs/${runId}`, {}, t); + if (run.status !== 200 || !run.json || typeof run.json !== 'object') last = `${run.call} -> HTTP ${run.status}${run.detail ? ` (${run.detail})` : ''}: the run could not be re-read`; + else if (run.json.run_attempt !== want) last = rerunFailedReadBack({ failed: pre.failed, run: run.json }).why; + else { + const listed = await readAttemptJobs(payload.repo, runId, want, api, t); + if (listed.fail) last = listed.fail.why; + else { + const v = rerunFailedReadBack({ failed: pre.failed, run: run.json, jobs: listed.jobs }); + if (v.ok) return { ok: true, why: '', result: rerunFailedLandedText(v) }; + if (!v.pending) return { ok: false, why: `${platform} accepted, but ${v.why} — read the run before anything else, ⛔ do not re-send` }; + last = v.why; + } + } + if (now() - started >= windowMs) break; + await sleep(pollMs); + } + return { ok: false, why: `${platform} accepted, but ${last} ${windowMs} ms later — UNCONFIRMED: the re-run may still start; read the run before anything else, ⛔ do not re-send (a second send meets attempt ${want} at the once fence and is refused)` }; +} + /** The one thing a seat needs from an answer: the id / number / url of what was written — or that a bodiless 204 accepted it. */ export function resultOf(req, json, status = null) { if (status === 204 && (json === null || json === undefined)) return 'accepted (HTTP 204, no body)'; @@ -730,6 +856,7 @@ export async function executeFleetWrite({ payload: raw, sender, token, api = DEF }; const t = { fetch: deps.fetch, token, pace: deps.pace }; const ub = { sleep: deps.sleep ?? ((ms) => new Promise((r) => setTimeout(r, ms))), now: deps.now ?? (() => Date.now()), ...(deps.updateBranch ?? {}) }; + const rr = { sleep: ub.sleep, now: ub.now, ...(deps.rerunFailed ?? {}) }; const judged = validatePayload(raw); if (!judged.ok) { @@ -804,6 +931,15 @@ export async function executeFleetWrite({ payload: raw, sender, token, api = DEF break; } const req = filled.req; + // The failed-jobs re-run (header): its fences on the run, judged HERE on the read before the POST — refused, NOT SENT. + const pre = req.rerunFailed ? await rerunFailedPreflight(req, payload, api, t) : null; + if (pre?.fail) { + const why = scrub(pre.fail, [token]); + rows.push({ action: i + 1, op: action.op, call: `${req.verb} ${req.path}`, status: 'not sent', result: `FAILED: ${why}` }); + stoppedAt = { action: i + 1, op: action.op, why }; + log(` ✗ action ${i + 1} ${action.op}: ${why}`); + break; + } let body = req.body ?? null; if (req.graphql) { const resolved = await graphqlVariables(req, payload, api, t); @@ -820,7 +956,8 @@ export async function executeFleetWrite({ payload: raw, sender, token, api = DEF const r = req.graphql ? await rest(api, '/graphql', { method: 'POST', body }, t) : await rest(api, req.path, { method: req.verb, body }, t); if (opener === null) opener = r; // The base sync is confirmed by MEASURE (header): the head polled off the expected sha, or a 422 judged by the compare. - const landed = req.updateBranch ? await confirmUpdateBranch(req, r, payload, api, t, ub) : requestLanded(req, r); + // The failed-jobs re-run is confirmed by reading the run back (header): attempt 2, its failed jobs queued or in progress. + const landed = req.updateBranch ? await confirmUpdateBranch(req, r, payload, api, t, ub) : req.rerunFailed ? await confirmRerunFailed(req, r, pre, payload, api, t, rr) : requestLanded(req, r); const call = req.graphql ? `POST /graphql ${req.graphql.mutation}` : r.call; rows.push({ action: i + 1, op: action.op, call, status: r.status, result: landed.ok ? (landed.idempotent ? 'already absent (idempotent)' : scrub(landed.result ?? resultOf(req, r.json, r.status), [token])) : `FAILED: ${scrub(landed.why, [token])}` }); if (!landed.ok) { @@ -889,9 +1026,10 @@ const SELF_TEST_BATTERIES = Object.freeze({ 'the release.yml refresh: ONE POST to its dispatch endpoint carrying { ref: main, inputs: { refresh_version_pr: "true" } } under the App token, landing as accepted; force, false, absent, a string or any other input refused with ZERO requests, exit 2': 9, "the batch gap on the runner: a stroke carrying DEFAULT_BATCH_THRESHOLD or more content-creating actions is announced as ONE rule ⑤ batch before its first write — on an injected clock every write through the last content-creating one leaves a full DEFAULT_BATCH_GAP_MS after the one before, and the pacing is exactly what the seat budgets; below the threshold, after the batch and on a landing stroke rule ①'s gap; a stroke the job cannot hold refused with ZERO requests; an announcement the log cannot record refused before the first write": 10, "the governed-review follow-up: after pr_create (at the number the create answered) and after pr_ready, the pull's file list is read in full; a Tier H hit requests review from every GOVERNED_APPROVERS account not already requested, assigned or reviewed — the pull's own author assigned instead, never requested; a Tier S or ungoverned diff sends nothing and says so; an unreadable or truncated list, an unreadable pull or review list, or a refused write is a FAILED action": 29, + "the run_rerun_failed: the four fences judged HERE on the run read immediately before ONE bodiless POST — each refused with the POST not sent and accepted on the measured run; a run, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation": 17, }); -// Raised 17 → 18 with the governed-review follow-up battery: one battery, one row. -const SELF_TEST_BATTERY_FLOOR = 18; +// Raised 17 → 18 with the governed-review follow-up battery, 18 → 19 with the run_rerun_failed battery: one battery, one row. +const SELF_TEST_BATTERY_FLOOR = 19; const UNATTRIBUTED_BATTERY = '(unattributed)'; const batteryCases = new Map(); @@ -1356,6 +1494,75 @@ export async function selfTest() { t('⛔ the release entry\'s inputs on the shard-timings refresh — a file whose entry is null — are refused with zero requests, exit 2', [shard.exit, shard.seen.length], [EXIT_REFUSED, 0]); } + // ── the failed-jobs re-run ────────────────────────────────────────────── + battery("the run_rerun_failed: the four fences judged HERE on the run read immediately before ONE bodiless POST — each refused with the POST not sent and accepted on the measured run; a run, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation"); + { + const RUN = 39713789453; + const RUN_GET = `GET /repos/${REPO}/actions/runs/${RUN}`; + const A1 = `GET /repos/${REPO}/actions/runs/${RUN}/attempts/1/jobs`; + const A2 = `GET /repos/${REPO}/actions/runs/${RUN}/attempts/2/jobs`; + const RERUN = `POST /repos/${REPO}/actions/runs/${RUN}/rerun-failed-jobs`; + // The measured run (39713789453): attempt 1 failed in one shard and the aggregate that needs it; attempt 2 listed the + // shard queued at once, the passed shards completed (carried over), and the aggregate only when its need finished. + const runAt = (patch = {}) => ({ status: 200, json: { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', ...patch } }); + const attempt2Run = runAt({ run_attempt: 2, status: 'queued', conclusion: null }); + const attempt1 = { status: 200, json: { total_count: 3, jobs: [ + { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + { id: 119611140839, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }, + { id: 119617410962, name: 'Test Core', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + ] } }; + const attempt2 = (status = 'queued') => ({ status: 200, json: { total_count: 2, jobs: [ + { id: 119642047246, name: 'Test Core (3/6)', run_attempt: 2, status, conclusion: status === 'completed' ? 'failure' : null }, + { id: 119642048282, name: 'Test Core (1/6)', run_attempt: 2, status: 'completed', conclusion: 'success' }, + ] } }); + let clock = 0; + const timing = { sleep: async (ms) => { clock += ms; }, now: () => clock }; + const stroke = base([{ op: 'run_rerun_failed', run_id: RUN }]); + const happy = { ...allowed, [A1]: attempt1, [RUN_GET]: [runAt(), attempt2Run], [RERUN]: { status: 201, json: {} }, [A2]: attempt2() }; + const LANDED = 'attempt 2 · re-run: `Test Core (3/6)` job 119642047246 queued · not listed yet: `Test Core` (a job is listed when its needs finish)'; + + clock = 0; + // Its own pace log, never the shared one: the shared log's later cases spend write-pace's hourly budget for the + // fixture token exactly (measured: one more write there refuses the annotations battery's third write, exit 10). + const ok = await run(stroke, happy, timing); + t('⭐ the measured run passes every fence: the sender gate, attempt 1\'s jobs (the baseline), the run — the read immediately before the POST — then ONE POST, then the run and attempt 2\'s jobs read back; exit 0', [ok.exit, ok.seen.map((s) => s.call)], [EXIT_OK, [PERM, A1, RUN_GET, RERUN, RUN_GET, A2]], ok.logs.join(' | ')); + t('…the one write is that POST with NO body, under the App token like every request', [writes(ok.seen).map((s) => [s.call, s.body]), ok.seen.every((s) => s.auth === `Bearer ${TOKEN}`)], [[[RERUN, null]], true]); + t('…the 201 is confirmed by the read-back: the row names attempt 2, the failed shard\'s new job id queued, and the aggregate not listed yet', [ok.rows[0]?.status, ok.rows[0]?.result, ok.summary.includes(`| 1 | \`run_rerun_failed\` | \`${RERUN}\` | 201 | ${LANDED} |`)], [201, LANDED, true], ok.summary); + t('…no other actions/* write leaves, and no annotation is emitted', [ok.seen.filter((s) => s.call.includes('/actions/') && !s.call.startsWith('GET ') && s.call !== RERUN), ok.annotations, ok.logs.filter((l) => l.startsWith('::'))], [[], [], []]); + // Each fence, refused at execute time: the run changed between the seat's request and this run — the POST is NOT SENT. + const fenced = async (patch) => { + const r = await run(stroke, { ...happy, [RUN_GET]: runAt(patch) }); + return [r.exit, writes(r.seen).length, r.seen.at(-1)?.call, r.rows[0]?.call, r.rows[0]?.status, /REFUSED at the relay's fence/.test(r.rows[0]?.result ?? '')]; + }; + const refusedRow = [EXIT_ACTION_FAILED, 0, RUN_GET, RERUN, 'not sent', true]; + t('⛔ fence workflow, refused HERE: a run of another workflow — zero writes, the POST row not sent, the run read the last request', [await fenced({ path: '.github/workflows/lint.yml' }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ path: '.github/workflows/lint.yml' }) })).rows[0]?.result.includes('fence `workflow`: the run reads ".github/workflows/lint.yml"')], [refusedRow, true]); + t('⛔ fence event, refused HERE: a push run — zero writes, the POST not sent, the event fence named', [await fenced({ event: 'push' }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ event: 'push' }) })).rows[0]?.result.includes('fence `event`: the run reads "push"')], [refusedRow, true]); + t('⛔ fence once, refused HERE: a run a person re-ran meanwhile (attempt 2) — zero writes, the POST not sent, the once fence named', [await fenced({ run_attempt: 2 }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ run_attempt: 2 }) })).rows[0]?.result.includes('fence `once`: the run reads 2')], [refusedRow, true]); + t('⛔ fence failed, refused HERE: a run that concluded success, and one still in progress — zero writes, the POST not sent, the failed fence named', [await fenced({ conclusion: 'success' }), await fenced({ status: 'in_progress', conclusion: null })], [refusedRow, refusedRow]); + const unread = await run(stroke, { ...happy, [RUN_GET]: { status: 404, json: { message: 'Not Found' } } }); + t('⛔ a run that cannot be read is never judged: zero writes, the POST not sent, the read named', [unread.exit, writes(unread.seen).length, unread.rows[0]?.status, unread.rows[0]?.result.includes('the run could not be read')], [EXIT_ACTION_FAILED, 0, 'not sent', true]); + const noBase = await run(stroke, { ...happy, [A1]: { status: 403, json: { message: 'Resource not accessible by integration' } } }); + t('⛔ a baseline that cannot be read: zero writes — a POST whose read-back has nothing to confirm is never sent', [noBase.exit, writes(noBase.seen).length, noBase.rows[0]?.result.includes("attempt 1's job list could not be read")], [EXIT_ACTION_FAILED, 0, true]); + const allPassed = await run(stroke, { ...happy, [A1]: { status: 200, json: { total_count: 1, jobs: [{ id: 1, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }] } } }); + t('⛔ a baseline with no failed job: zero writes, the row saying the read-back would have nothing to confirm', [allPassed.exit, writes(allPassed.seen).length, allPassed.rows[0]?.result.includes('none concluded failure')], [EXIT_ACTION_FAILED, 0, true]); + const denied = await run(stroke, { ...happy, [RERUN]: { status: 403, json: { message: 'Resource not accessible by integration' } } }); + t('a 403 on the POST (the token minted without actions write) is a FAILED action with the platform\'s sentence, and nothing is read back after it', [denied.exit, denied.rows[0]?.status, denied.rows[0]?.result.includes('Resource not accessible by integration'), denied.seen.some((s) => s.call === A2)], [EXIT_ACTION_FAILED, 403, true, false]); + clock = 0; + const never = await run(stroke, { ...happy, [RUN_GET]: [runAt(), runAt()] }, timing); + t('⛔ a read-back that never sees attempt 2 is a FAILED action — UNCONFIRMED, naming the attempt it read and never to re-send — after the window, polled at the interval', [never.exit, never.rows[0]?.status, ['UNCONFIRMED', 'run_attempt 1, not 2', 'do not re-send', 'once fence'].every((s) => never.rows[0]?.result.includes(s)), clock, never.seen.filter((s) => s.call === RUN_GET).length], [EXIT_ACTION_FAILED, 201, true, RERUN_FAILED_WINDOW_MS, 1 + RERUN_FAILED_WINDOW_MS / RERUN_FAILED_POLL_MS + 1], never.rows[0]?.result); + clock = 0; + const late = await run(stroke, { ...happy, [RUN_GET]: [runAt(), runAt(), runAt(), attempt2Run] }, timing); + t('a new attempt the platform shows on a later read lands, read again at the interval until it does', [late.exit, late.rows[0]?.result, clock], [EXIT_OK, LANDED, 2 * RERUN_FAILED_POLL_MS]); + const completed = await run(stroke, { ...happy, [A2]: attempt2('completed') }, timing); + t('⛔ a failed job listed completed in attempt 2 was not re-run as asked: a FAILED action at once, naming the job', [completed.exit, completed.rows[0]?.result.includes('`Test Core (3/6)` (job 119642047246) completed failure'), completed.seen.filter((s) => s.call === A2).length], [EXIT_ACTION_FAILED, true, 1]); + clock = 0; + const blind = await run(stroke, { ...happy, [A2]: { status: 502, json: { message: 'Bad Gateway' } } }, timing); + t('⛔ an attempt-2 job list that never answers is a FAILED action, UNCONFIRMED, naming the read', [blind.exit, blind.rows[0]?.result.includes("attempt 2's job list could not be read"), blind.rows[0]?.result.includes('UNCONFIRMED')], [EXIT_ACTION_FAILED, true, true]); + const beside = await run(base([{ op: 'run_rerun_failed', run_id: RUN }, { op: 'comment', issue: 1, body: 'x' }]), happy); + const stringId = await run(base([{ op: 'run_rerun_failed', run_id: String(RUN) }]), happy); + t('⛔ the validator is the executor\'s gate too: beside another action, or with run_id a string, refused with ZERO requests, exit 2', [[beside.exit, beside.seen.length], [stringId.exit, stringId.seen.length]], [[EXIT_REFUSED, 0], [EXIT_REFUSED, 0]]); + } + // ── the summary ───────────────────────────────────────────────────────── battery('the summary: request, sender and role, session, target, one row per request'); { @@ -1635,7 +1842,8 @@ export async function selfTest() { `✓ fleet-write/execute self-test: ${cases.length} cases pass across ${declared.length} batteries — the sender gate from the target repo's answer, ` + 'every op as the request the table declares, stop at the first failure with later actions untouched, the idempotent label DELETE, the GraphQL ' + 'ops behind a node-id read, a pull mutation landing only when its answer shows the state it asked for (armed or queued, off, ready, draft), a base sync confirmed by the head leaving its expected sha or measured as a no-op, an assignees leg sent to the number the create answered and landing only when its answer lists every login sent, ' + - 'a governed-review follow-up after pr_create and pr_ready that reads the pull\'s whole file list and, on a Tier H hit alone, requests every GOVERNED_APPROVERS account not already covered (the author assigned instead), zero writes when covered and a FAILED action when the list cannot be read whole, a transfer gated on both repositories and landing only on its target, one summary row per request, ' + + 'a governed-review follow-up after pr_create and pr_ready that reads the pull\'s whole file list and, on a Tier H hit alone, requests every GOVERNED_APPROVERS account not already covered (the author assigned instead), zero writes when covered and a FAILED action when the list cannot be read whole, a transfer gated on both repositories and landing only on its target, ' + + 'a failed-jobs re-run fenced on the run read just before its one POST (zero writes on any fence) and landing only when attempt 2 reads back with its failed jobs queued or in progress, one summary row per request, ' + "one annotation per created or moved card carrying the number the platform answered, read back by the reader's own parser, a stroke at or " + 'above the batch threshold announced as ONE rule ⑤ batch and paced at the batch gap (one the job cannot hold refused with zero requests), and a known ' + 'token that came back out of NO summary, log or error.', From 847734aa95fe799f2b8e6f0859c67fd7781538a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 07:08:27 +0000 Subject: [PATCH 3/6] feat(pm): rerun-failed.mjs, the relay-only seat wrapper for run_rerun_failed The wrapper reads the run and judges the four fences as an advisory early refusal, names the jobs attempt 1 failed, sends ONE run_rerun_failed stroke through sendFleetWrite, and reads attempt 2's job ids back as a reading. It has no direct path: a direct route is a prerequisite refusal with zero writes. dispatch.mjs's no-run conformance learns a named relay-only sender, whose control refuses under direct instead of writing, and gains the seat battery for the stroke. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- scripts/pm/fleet-write/dispatch.mjs | 64 ++- scripts/pm/rerun-failed.mjs | 596 ++++++++++++++++++++++++++++ 2 files changed, 655 insertions(+), 5 deletions(-) create mode 100644 scripts/pm/rerun-failed.mjs diff --git a/scripts/pm/fleet-write/dispatch.mjs b/scripts/pm/fleet-write/dispatch.mjs index 9d543f0e55b..60d5e1cd255 100644 --- a/scripts/pm/fleet-write/dispatch.mjs +++ b/scripts/pm/fleet-write/dispatch.mjs @@ -241,6 +241,15 @@ * 422 as a no-op, before it calls the action landed — so a `success` run * IS that confirmation; the seat reads the new head with one * `GET /repos/{repo}/pulls/{n}` after, and this file reads nothing back; + * - `run_rerun_failed` carries no body, names no workflow and is + * confirmed INSIDE the run too: the executor judges the four fences on + * the run before its POST and lands the action only when the run reads + * back as attempt 2 with its failed jobs queued or in progress + * (`execute.mjs`, `confirmRerunFailed`). So a `success` run IS that + * confirmation and this file reads nothing back; the seat's wrapper + * (`scripts/pm/rerun-failed.mjs`) reads the new attempt's job ids after, + * as a reading for its note. A run that failed (a fence refused it, or + * the re-run did not read back) is exit 5 like any failed run; * - a `pr_create` carrying `assignees` is read back at its body's locator * as before, and its row ALSO carries the pull's `assignees` as the * platform lists them (`assigneesReading`; `--json` carries them too) — @@ -301,7 +310,13 @@ * (any request but a GET), printing the shared `unconfirmedText` sentence; * - the control drives each under `direct`: at least one direct write and no * dispatch — so the empty list above is a measurement, never a recorder - * that sees nothing; + * that sees nothing. A RELAY-ONLY sender — one exporting + * `relayOnly = true` because its op has no direct path at all + * (`rerun-failed.mjs`: the relay's executor is where its fences are + * authoritative) — answers the control the other way round: under + * `direct` it refuses with a PREREQUISITE exit, ZERO direct writes and + * no dispatch, and the recorder holds the reads it made first, so its + * empty write list is a measurement too; * - a sender without the probe fails the pin until it carries one and gives * the same answer. ⛔ There is no third, quieter answer to an accepted * dispatch with no run. This file's own CLI (`with-fleet.sh --actions` @@ -412,6 +427,8 @@ export const RELAY_JOB_TIMEOUT_MS = 10 * 60 * 1000; /** The export every sender carries for the no-run conformance (the header's section of that name). */ const RELAY_MISS_PROBE = 'relayMissProbe'; +/** The export a RELAY-ONLY sender sets to `true`: its op has no direct path, so its control refuses instead of writing (header). */ +const RELAY_ONLY = 'relayOnly'; // --------------------------------------------------------------------------- // Pure core @@ -1443,8 +1460,10 @@ const SELF_TEST_BATTERIES = Object.freeze({ "the started run: an action naming a workflow is read back as the newest workflow_dispatch run of that file created since the dispatch — the target repo's run list, read after the relay run completed and re-read within a bounded window; found is success naming id, url and actor; none in the window or an unreadable list is UNCONFIRMED (6) after ONE dispatch; a stroke naming no workflow reads none": 11, "the release.yml refresh at the seat: the stroke packs with inputs refresh_version_pr true and travels in ONE dispatch; its started run is read back from release.yml's own run list; force, false, absent or any other input is refused before anything is sent, exit 2": 6, "the batch gap at the seat: content creation read off the op table; a stroke at or above write-pace's threshold is batched through its last content-creating write and waited for exactly that much longer; a stroke whose pacing the relay job cannot hold is refused before anything is sent, naming the per-op doors; the job's timeout mirrors the workflow's": 13, + "the run_rerun_failed stroke at the seat: it packs as ONE action carrying run_id alone and travels in ONE dispatch; beside any other action, or with a key that widens it, it is refused before anything is sent; it creates no content, carries no body and names no workflow, so nothing is read back here — a success run is the executor's confirmation; a run that failed at the relay's fence is exit 5 after ONE dispatch, never re-sent": 6, }); -const SELF_TEST_BATTERY_FLOOR = 21; +// Raised 21 → 22 with the run_rerun_failed seat battery: one battery, one row. +const SELF_TEST_BATTERY_FLOOR = 22; const UNATTRIBUTED_BATTERY = '(unattributed)'; const batteryCases = new Map(); @@ -2201,6 +2220,30 @@ export async function selfTest() { t('--dry-run prints the relay request carrying the refresh with its inputs and sends nothing, exit 0', [dry.code, dry.seen.length, dry.json?.event_type, dry.json?.client_payload?.actions], [EXIT_OK, 0, RELAY_EVENT_TYPE, [REFRESH]]); } + // ── the failed-jobs re-run at the seat ────────────────────────────────── + battery("the run_rerun_failed stroke at the seat: it packs as ONE action carrying run_id alone and travels in ONE dispatch; beside any other action, or with a key that widens it, it is refused before anything is sent; it creates no content, carries no body and names no workflow, so nothing is read back here — a success run is the executor's confirmation; a run that failed at the relay's fence is exit 5 after ONE dispatch, never re-sent"); + { + const REPO = 'objectstack-ai/objectstack'; + const RERUN = { op: 'run_rerun_failed', run_id: 39713789453 }; + const pack = (actions) => packRequest({ repo: REPO, session: SESSION, actions, requestId: 'fw-test-1' }); + const packed = pack([RERUN]); + t('the stroke packs: one action, op and run_id alone — no workflow, event or attempt travels, those are the run\'s to answer', [packed.ok, packed.payload?.actions, Object.keys(packed.payload?.actions?.[0] ?? {})], [true, [RERUN], ['op', 'run_id']], packed.errors.join(' | ')); + const refusedFor = (actions, needle) => { + const p = pack(actions); + return !p.ok && p.payload === null && p.errors.some((e) => e.includes(needle)); + }; + t('⛔ alone, at the packer: beside a comment or a workflow_dispatch it is refused before anything is sent, naming the action that does not belong', [refusedFor([RERUN, { op: 'comment', issue: 1, body: 'x' }], 'actions[1] is `comment` in a stroke that carries a `run_rerun_failed`'), refusedFor([RERUN, { op: 'workflow_dispatch', workflow: 'shard-timings-refresh.yml', ref: 'main' }], 'actions[1] is `workflow_dispatch` in a stroke that carries a `run_rerun_failed`')], [true, true]); + t('⛔ nothing widens it, at the packer: a workflow or an attempt beside run_id is refused by name before anything is sent', [refusedFor([{ ...RERUN, workflow: '.github/workflows/ci.yml' }], 'actions[0].workflow is not a key `run_rerun_failed` takes'), refusedFor([{ ...RERUN, run_attempt: 1 }], 'actions[0].run_attempt is not a key `run_rerun_failed` takes')], [true, true]); + const pacing = strokePacing(packed.payload); + t('derived from the table: it is no body op, no run op, no content-creating op and no annotated op, and its stroke is ONE write, never batched', [BODY_OPS.includes(RERUN.op), RUN_OPS.includes(RERUN.op), CONTENT_CREATING_OPS.includes(RERUN.op), Object.hasOwn(ANNOTATED_OPS, RERUN.op), pacing.writes, pacing.batched], [false, false, false, false, 1, false]); + const landed = await drive({ runs: () => [RUN('completed', 'success')] }, { stroke: packed.payload }); + const sent = landed.seen.filter((s) => s.call === DISPATCH); + const beyondTheRelayRun = landed.seen.filter((s) => s.call !== DISPATCH && s.call !== RUNS && s.call !== `GET /repos/${RELAY_REPO}/actions/runs/42`); + t('a success run: ONE dispatch carrying the action as judged, and no read beyond the relay run\'s own — nothing read back at the target, the executor\'s read-back landed it — exit 0', [sent.length, sent[0]?.body?.client_payload?.actions, landed.state, landed.readBack?.state, landed.runs?.state, beyondTheRelayRun.map((s) => s.call), exitForResult(landed)], [1, [RERUN], 'success', 'none', 'none', [], EXIT_OK], landed.logs.join(' | ')); + const fenced = await drive({ runs: () => [RUN('completed', 'failure')] }, { stroke: packed.payload }); + t('⛔ a run that failed — the relay refused at a fence, or the re-run did not read back — is exit 5 after ONE dispatch, never re-sent and never a direct write', [fenced.state, exitForResult(fenced), fenced.seen.filter((s) => s.call === DISPATCH).length, fenced.seen.filter((s) => s.call !== DISPATCH && !s.call.startsWith('GET ')).length], ['failure', EXIT_PLATFORM_REFUSAL, 1, 0]); + } + // ── the issue_create re-list ──────────────────────────────────────────── battery('the issue_create re-list: a list that lags the create reads back IDENTICAL on a bounded re-list; a real miss is still unfound (exit 6) after exactly the declared window; an unreadable re-list is unread; the create is never re-sent'); { @@ -2410,8 +2453,13 @@ export async function selfTest() { .filter((f) => /\bsendFleetWrite\b/u.test(readReal(join(REPO_ROOT, f), 'utf8'))) .sort(); // The discovery's floor, BY NAME: a scan gone blind — a moved root, a renamed import — finds fewer and reds here. - const KNOWN = ['scripts/pm/close-cards.mjs', 'scripts/pm/issue-create.mjs', 'scripts/pm/issue-transfer.mjs', 'scripts/pm/label-write.mjs', 'scripts/pm/post-stamped.mjs']; + const KNOWN = ['scripts/pm/close-cards.mjs', 'scripts/pm/issue-create.mjs', 'scripts/pm/issue-transfer.mjs', 'scripts/pm/label-write.mjs', 'scripts/pm/post-stamped.mjs', 'scripts/pm/rerun-failed.mjs']; t('the discovery finds every sender known today — the scan is not blind', KNOWN.filter((k) => !senders.includes(k)), []); + // Declaring `relayOnly` swaps a sender's control from "it writes directly" to "it refuses": named here, so no sender takes that swap unreviewed. + const RELAY_ONLY_KNOWN = ['scripts/pm/rerun-failed.mjs']; + const declaredRelayOnly = []; + for (const sender of senders) if ((await import(pathToFileURL(join(REPO_ROOT, sender)).href))[RELAY_ONLY] === true) declaredRelayOnly.push(sender); + t('the relay-only senders are exactly the ones named here — a sender declaring itself relay-only to skip the write control reds until it is named', declaredRelayOnly, RELAY_ONLY_KNOWN); const relayRoute = (requested) => ({ requested, transport: 'dispatch', reason: `conformance: ${requested} → dispatch`, error: null, session: SESSION }); const DIRECT_ROUTE = { requested: 'direct', transport: 'direct', reason: 'conformance: direct', error: null, session: null }; @@ -2448,7 +2496,8 @@ export async function selfTest() { } }; for (const sender of senders) { - const probe = (await import(pathToFileURL(join(REPO_ROOT, sender)).href))[RELAY_MISS_PROBE]; + const mod = await import(pathToFileURL(join(REPO_ROOT, sender)).href); + const probe = mod[RELAY_MISS_PROBE]; t(`${sender} sends a relay dispatch and exports ${RELAY_MISS_PROBE} — a sender without it fails here until it answers like the rest`, typeof probe, 'function'); if (typeof probe !== 'function') continue; for (const [requested, state] of [['auto', 'no-run'], ['dispatch', 'no-run'], ['auto', 'timeout']]) { @@ -2463,6 +2512,7 @@ export async function selfTest() { ); } // The control: the same probe under direct WRITES, and the recorder sees it — so the empty lists above are a measurement. + // A relay-only sender (`relayOnly`, header) has no direct write to see: under direct it REFUSES, and the recorder holds its reads. const control = { dispatches: 0 }; const c = await drive(probe, { route: DIRECT_ROUTE, @@ -2472,7 +2522,11 @@ export async function selfTest() { }, pace: freshPace(), }); - t(`…the control: ${sender} under direct issues its write directly — the recorder sees it — and never dispatches`, [control.dispatches, c.calls.filter(isDirectWrite).length > 0], [0, true], `${c.exit} · ${c.calls.join(' | ')}`); + if (mod[RELAY_ONLY] === true) { + t(`…the control: ${sender} is relay-only — under direct it refuses (exit ${EXIT_PREREQUISITE}) with ZERO direct writes and no dispatch, and the recorder holds the reads it made first`, [control.dispatches, c.exit, c.calls.filter(isDirectWrite), c.calls.length > 0], [0, EXIT_PREREQUISITE, [], true], `${c.exit} · ${c.calls.join(' | ')}`); + } else { + t(`…the control: ${sender} under direct issues its write directly — the recorder sees it — and never dispatches`, [control.dispatches, c.calls.filter(isDirectWrite).length > 0], [0, true], `${c.exit} · ${c.calls.join(' | ')}`); + } } // This file's own CLI, the route `with-fleet.sh --actions` execs: it has no direct path, and it answers the same. diff --git a/scripts/pm/rerun-failed.mjs b/scripts/pm/rerun-failed.mjs new file mode 100644 index 00000000000..201d4db8fc6 --- /dev/null +++ b/scripts/pm/rerun-failed.mjs @@ -0,0 +1,596 @@ +#!/usr/bin/env node +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * rerun-failed — re-run the FAILED jobs of one `ci.yml` pull-request run, ONCE, as + * `objectstack-fleet[bot]`, through the fleet-write relay's `run_rerun_failed` op. + * + * node scripts/pm/rerun-failed.mjs --run 39713789453 # a run on the board repo + * node scripts/pm/rerun-failed.mjs --run ID --repo objectstack-ai/ # a run on another fleet repository + * node scripts/pm/rerun-failed.mjs --run ID --dry-run # read, judge and pack; send nothing + * node scripts/pm/rerun-failed.mjs --run ID --json # the outcome as one JSON line on stdout + * node scripts/pm/rerun-failed.mjs --self-test # offline, no network + * + * ## When a seat may run it + * + * pm-dispatch `references/platform-readings.md` carries the rule. A seat may + * re-run a PR's failed `ci.yml` jobs ONCE through this tool. It may do so only + * after it has written down, on the PR or the card, why it judges the failure a + * flake, and a second failure is real. ⛔ A dev never re-runs. + * + * ## What it does, in order + * + * 1. It reads the run (`GET /repos/{repo}/actions/runs/{id}`) and judges the + * four fences on the run with the predicates the relay's executor uses + * (`RERUN_FAILED_FENCES` in `fleet-write/ops.mjs`): the workflow, the + * event, once, failed. A run that fails one is REFUSED here (exit 2) with + * every failing fence named and nothing sent. It then reads attempt 1's + * job list and names the jobs that failed there, which are what the + * relay's read-back looks for. A list with no failed job is refused the + * same way. ⛔ This check is ADVISORY: an early refusal with its reason. + * The executor judges the four fences again on the run it reads + * immediately before its POST, and that check is the authoritative one; + * a pass here never stands in for it. + * 2. It packs ONE action, `{ op: 'run_rerun_failed', run_id }`, which the + * shared validator judges (alone in its stroke, `run_id` alone), and + * sends it as ONE `repository_dispatch` (`fleet-write/dispatch.mjs`'s + * `sendFleetWrite`), which waits for the relay's run. + * 3. After a `success` run it reads the run back at the seat, its attempt + * and attempt 2's re-run jobs, and prints their ids. That is a READING + * for the seat's note, never a verdict: the executor's own read-back + * landed the action. After a FAILED run it re-reads the run and prints + * what the fences say now, so the seat learns why without the job log it + * cannot read. + * + * ## RELAY-ONLY + * + * The re-run lands as the fleet through the relay or it does not land. The + * relay's token is minted with `actions: write` for this stroke alone, and its + * executor is where the fences are authoritative, so ⛔ there is no direct + * path. A route that resolves to `direct` (a developer's machine, + * `OS_FLEET_TRANSPORT=direct`, a relay the board has switched off) is a + * PREREQUISITE refusal, exit 3, with zero writes. An accepted dispatch with no + * run, or none completed, is UNCONFIRMED, exit 6, and never a direct write + * after it. `fleet-write/dispatch.mjs`'s no-run conformance drives this file's + * `relayMissProbe`, and its relay-only control pins the direct refusal + * (`relayOnly` below). + * + * ## Exit codes — capture them BEFORE any pipe + * + * 0 the relay's run succeeded, so attempt 2 started with its failed jobs + * queued or in progress; with `--dry-run`, the fences hold and the + * payload is printed, nothing sent. + * 2 usage, a fence the run fails or a run with no failed job (advisory, + * nothing sent), or the payload refused by the validator. + * 3 PREREQUISITE NOT MET — no token, the run or its jobs unreadable, no + * session, or a route that is not the relay. Nothing sent. + * 5 the dispatch was refused, or the relay's run FAILED: its executor + * refused at a fence, or the re-run did not read back. Go READ the run; + * ⛔ do not re-send blind (a re-send after a landing meets the once fence). + * 6 UNCONFIRMED — no run, or none completed within the ceiling. Go READ + * the run; ⛔ do not re-send blind. + * 10 the write throttle refused the dispatch. + */ + +import process from 'node:process'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; + +import { isEntrypoint } from '../invoked-as.mjs'; +import { EXIT_PREREQUISITE_NOT_MET, PROXY_FLAG, proxyRearmPlan } from './check-half-states.mjs'; +import { EXIT_UNCONFIRMED, exitForResult, packRequest, resolveRoute, sendFleetWrite, sessionFrom, unconfirmedText } from './fleet-write/dispatch.mjs'; +import { + RELAY_EVENT_TYPE, + RELAY_REPO, + RERUN_FAILED_ATTEMPT, + RERUN_FAILED_RERUN_STATES, + TARGET_REPO_SHAPE, + failedJobNames, + rerunFailedFenceFailures, + rerunFailedFenceText, +} from './fleet-write/ops.mjs'; +import { refusalText } from './fleet-write/validate.mjs'; + +const SELF_PATH = fileURLToPath(import.meta.url); +const API = 'https://api.github.com'; +const TOOL = 'scripts/pm/rerun-failed.mjs'; +const PROXY_REARM_GUARD = 'OS_RERUN_FAILED_PROXY_REARMED'; + +export const EXIT_OK = 0; +export const EXIT_USAGE = 2; +export const EXIT_PREREQUISITE = EXIT_PREREQUISITE_NOT_MET; +export const EXIT_PLATFORM_REFUSAL = 5; +export { EXIT_UNCONFIRMED }; + +/** + * This sender has no direct path: `fleet-write/dispatch.mjs`'s no-run conformance reads this export and holds + * the control to a refusal under `direct` — zero writes, no dispatch — instead of a direct write. + */ +export const relayOnly = true; + +/** The op this tool sends — the relay table's row, spelled once here. */ +export const OP = 'run_rerun_failed'; + +/** The page size and cap of the job-list reads — the endpoint's maximum, and a walk past the cap is refused. */ +export const JOBS_PAGE_SIZE = 100; +export const JOBS_PAGE_CAP = 10; + +// --------------------------------------------------------------------------- +// The pure core +// --------------------------------------------------------------------------- + +const isObject = (v) => v !== null && typeof v === 'object' && !Array.isArray(v); +const shortSha = (sha) => (typeof sha === 'string' ? sha.slice(0, 7) : '?'); + +/** The command line, judged. Returns `{ ok, options }` or `{ ok: false, error }`. Pure. */ +export function parseOptions(argv) { + const options = { run: null, repo: RELAY_REPO, dryRun: false, json: false }; + const args = [...argv]; + while (args.length) { + const a = args.shift(); + const value = () => { + const v = args.shift(); + return v === undefined || v.startsWith('--') ? null : v; + }; + if (a === '--run') { + const v = value(); + if (v === null || !/^[1-9][0-9]*$/.test(v) || !Number.isSafeInteger(Number(v))) return { ok: false, error: `--run needs a workflow run id, a positive integer — got ${JSON.stringify(v)}` }; + options.run = Number(v); + } else if (a === '--repo') { + const v = value(); + if (v === null || !TARGET_REPO_SHAPE.test(v) || v.includes('..')) return { ok: false, error: `--repo needs objectstack-ai/ — got ${JSON.stringify(v)}` }; + options.repo = v; + } else if (a === '--dry-run') options.dryRun = true; + else if (a === '--json') options.json = true; + else return { ok: false, error: `unrecognised argument ${JSON.stringify(a)}` }; + } + if (options.run === null) return { ok: false, error: '--run is required — the run whose failed jobs to re-run' }; + return { ok: true, options }; +} + +/** One line about the run, for the transcript: what it is, where it stands, and which pull request head it ran on. Pure. */ +export function runLine(repo, run) { + const pulls = Array.isArray(run?.pull_requests) && run.pull_requests.length ? ` · pull ${run.pull_requests.map((p) => `#${p?.number ?? '?'}`).join(', ')}` : ''; + return `run ${run?.id ?? '?'} on ${repo} — ${run?.name ?? '?'} (${run?.path ?? '?'}) · event ${run?.event ?? '?'} · attempt ${run?.run_attempt ?? '?'} · ${run?.status ?? '?'}/${run?.conclusion ?? 'none'} · head ${shortSha(run?.head_sha)}${pulls}`; +} + +/** The jobs of an attempt listed `queued` or `in_progress` — the re-run's own jobs, for the seat's reading. Pure. */ +export function rerunJobs(jobs) { + return (Array.isArray(jobs) ? jobs : []).filter((j) => RERUN_FAILED_RERUN_STATES.includes(j?.status)).map((j) => ({ name: j.name, id: j.id, status: j.status })); +} + +// --------------------------------------------------------------------------- +// Transport — reads only. ⛔ This file issues no write of its own: the one write is the dispatch, sent and paced by +// `sendFleetWrite`. Nothing here throws on an HTTP status. +// --------------------------------------------------------------------------- + +function liveCall(token) { + return async (path) => { + let res; + try { + res = await fetch(`${API}${path}`, { headers: { accept: 'application/vnd.github+json', 'x-github-api-version': '2022-11-28', ...(token ? { authorization: `Bearer ${token}` } : {}) } }); + } catch (e) { + return { status: 0, json: null, detail: e?.message ?? 'fetch threw', call: `GET ${path}` }; + } + let json = null; + try { + json = await res.json(); + } catch { + json = null; + } + return { status: res.status, json, detail: typeof json?.message === 'string' ? json.message : '', call: `GET ${path}` }; + }; +} + +/** One attempt's job list, every page: `{ jobs }` or `{ fail }` — the sentence. Never throws. */ +async function readJobs(call, repo, runId, attempt) { + const jobs = []; + for (let page = 1; page <= JOBS_PAGE_CAP; page++) { + const r = await call(`/repos/${repo}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=${JOBS_PAGE_SIZE}&page=${page}`); + if (r.status !== 200 || !Array.isArray(r.json?.jobs)) return { fail: `${r.call} -> HTTP ${r.status}${r.detail ? ` (${r.detail})` : ''}: attempt ${attempt}'s job list could not be read` }; + jobs.push(...r.json.jobs); + const total = Number.isInteger(r.json.total_count) ? r.json.total_count : null; + if (r.json.jobs.length < JOBS_PAGE_SIZE || (total !== null && jobs.length >= total)) return { jobs }; + } + return { fail: `attempt ${attempt}'s job list hit its ${JOBS_PAGE_CAP}-page cap with more pending — a truncated list is a SUBSET` }; +} + +/** The run, read: `{ run }` or `{ fail }`. Never throws. */ +async function readRun(call, repo, runId) { + const r = await call(`/repos/${repo}/actions/runs/${runId}`); + if (r.status !== 200 || !isObject(r.json)) return { fail: `${r.call} -> HTTP ${r.status}${r.detail ? ` (${r.detail})` : ''}: the run could not be read` }; + return { run: r.json }; +} + +// --------------------------------------------------------------------------- +// The run +// --------------------------------------------------------------------------- + +/** + * The whole tool, every dependency injectable: `call(path)` answers a GET, `route` the transport (resolved when + * absent), `send(payload, deps)` the relay (`sendFleetWrite` when absent), `env` the session source for a dry run. + * Returns `{ exit, ... }` — never throws on an HTTP status. + */ +export async function runRerunFailed(options, deps = {}) { + const log = deps.log ?? ((l) => console.error(l)); + const env = deps.env ?? process.env; + const token = deps.token ?? env.GITHUB_TOKEN ?? env.GH_TOKEN ?? ''; + const call = deps.call ?? liveCall(token); + const { run: runId, repo } = options; + const out = { repo, run: runId, exit: null, fences: [], failed: [], transport: null, relay: null, reading: null }; + + // ── 1. the fences, advisory ──────────────────────────────────────────────── + const read = await readRun(call, repo, runId); + if (read.fail) { + log(`rerun-failed: PREREQUISITE NOT MET — ${read.fail}. ⛔ NOT MEASURED: no fence was judged and nothing was sent.`); + return { ...out, exit: EXIT_PREREQUISITE }; + } + log(`rerun-failed: ${runLine(repo, read.run)}`); + const failures = rerunFailedFenceFailures(read.run); + if (failures.length) { + log(`rerun-failed: REFUSED before the dispatch — ${rerunFailedFenceText(failures)}. The relay judges the same fences on the run again and would refuse it too. ⛔ Nothing sent.`); + return { ...out, fences: failures, exit: EXIT_USAGE }; + } + const baseline = await readJobs(call, repo, runId, RERUN_FAILED_ATTEMPT); + if (baseline.fail) { + log(`rerun-failed: PREREQUISITE NOT MET — ${baseline.fail}, so the jobs the re-run would confirm are unknown. ⛔ Nothing sent.`); + return { ...out, exit: EXIT_PREREQUISITE }; + } + const failed = failedJobNames(baseline.jobs); + if (!failed.length) { + log(`rerun-failed: REFUSED before the dispatch — attempt ${RERUN_FAILED_ATTEMPT} lists ${baseline.jobs.length} job(s) and none concluded failure, so the relay's read-back would have nothing to confirm and the relay refuses it. ⛔ Nothing sent.`); + return { ...out, exit: EXIT_USAGE }; + } + log(`rerun-failed: the four fences hold (advisory — the relay judges them again on the run it reads before its POST); attempt ${RERUN_FAILED_ATTEMPT} failed ${failed.map((n) => `\`${n}\``).join(', ')}.`); + const action = { op: OP, run_id: runId }; + + // ── 2. the stroke — relay-only ───────────────────────────────────────────── + if (options.dryRun) { + const session = sessionFrom(env); + if (!session) { + log('rerun-failed: PREREQUISITE NOT MET — no session id for the envelope (OS_FLEET_SESSION, or the cloud container\'s own). ⛔ Nothing sent.'); + return { ...out, failed, exit: EXIT_PREREQUISITE }; + } + const packed = packRequest({ repo, session, actions: [action] }); + if (!packed.ok) { + log(refusalText(packed.errors)); + return { ...out, failed, exit: EXIT_USAGE }; + } + log('rerun-failed: DRY RUN — nothing sent. This is the relay request the dispatch would carry:'); + return { ...out, failed, exit: EXIT_OK, dryRun: { event_type: RELAY_EVENT_TYPE, client_payload: packed.payload } }; + } + const route = deps.route ?? (await resolveRoute(env)); + out.transport = route.transport ?? null; + if (route.error) { + log(`rerun-failed: PREREQUISITE NOT MET — ${route.error} ⛔ Nothing sent.`); + return { ...out, failed, exit: EXIT_PREREQUISITE }; + } + if (route.transport !== 'dispatch') { + log( + `rerun-failed: PREREQUISITE NOT MET — the route is ${route.transport ?? 'unresolved'} (${route.reason}), and this op is RELAY-ONLY: the relay's token alone carries actions write for it, ` + + 'and its executor alone is where the fences are authoritative. ⛔ Not falling back to a direct write — that would be a re-run booked against the personal account. Nothing sent.', + ); + return { ...out, failed, exit: EXIT_PREREQUISITE }; + } + log(`rerun-failed: transport dispatch — ${route.reason}`); + const packed = packRequest({ repo, session: route.session, actions: [action] }); + if (!packed.ok) { + log(refusalText(packed.errors)); + return { ...out, failed, exit: EXIT_USAGE }; + } + const send = deps.send ?? sendFleetWrite; + const result = await send(packed.payload, { token, log, ...(deps.sendDeps ?? {}) }); + out.relay = { state: result.state, run: result.run ?? null, requestId: result.requestId ?? packed.payload.request_id }; + + // ── 3. the outcome, read at the seat ─────────────────────────────────────── + if (!result.ok) { + if (result.state === 'no-run' || result.state === 'timeout') { + log(unconfirmedText(result, TOOL)); + return { ...out, failed, exit: EXIT_UNCONFIRMED }; + } + if (result.state === 'failure') { + const now = await readRun(call, repo, runId); + const why = now.fail ? `${now.fail} either` : `${runLine(repo, now.run)}; ${rerunFailedFenceFailures(now.run).length ? `the fences now read: ${rerunFailedFenceText(rerunFailedFenceFailures(now.run))}` : 'the four fences still hold, so the relay refused or the re-run did not read back for another reason — read the relay run'}`; + log(`rerun-failed: ✗ the relay's run FAILED (${result.detail ?? 'conclusion failure'})${result.run?.url ? ` ${result.run.url}` : ''} — its executor refused at a fence, or the re-run did not read back as attempt ${RERUN_FAILED_ATTEMPT + 1}. Read now: ${why}. ⛔ Do not re-send blind.`); + return { ...out, failed, exit: exitForResult(result) }; + } + log(`rerun-failed: ✗ the relay answered ${result.state}${result.detail ? ` (${result.detail})` : ''}. ⛔ Do not re-send blind.`); + return { ...out, failed, exit: exitForResult(result) }; + } + const after = await readRun(call, repo, runId); + const jobs = after.run && after.run.run_attempt === RERUN_FAILED_ATTEMPT + 1 ? await readJobs(call, repo, runId, RERUN_FAILED_ATTEMPT + 1) : { fail: after.fail ?? `the run reads attempt ${after.run?.run_attempt ?? '?'}` }; + const reading = jobs.jobs ? rerunJobs(jobs.jobs) : null; + out.reading = { attempt: after.run?.run_attempt ?? null, jobs: reading, why: jobs.fail ?? null }; + log( + `rerun-failed: ✓ the relay re-ran run ${runId}'s failed jobs (its executor read attempt ${RERUN_FAILED_ATTEMPT + 1} back before it called the action landed)${result.run?.url ? ` — relay run ${result.run.url}` : ''}. ` + + (reading ? `Reading now: attempt ${after.run.run_attempt} · ${reading.length ? reading.map((j) => `\`${j.name}\` job ${j.id} ${j.status}`).join(', ') : 'no job queued or in progress any more'}.` : `Reading now: ${jobs.fail} — a reading, not a verdict; the relay's row holds the job ids.`), + ); + return { ...out, failed, exit: EXIT_OK }; +} + +// --------------------------------------------------------------------------- +// The no-run conformance probe (`fleet-write/dispatch.mjs`, its header's section of that name) +// --------------------------------------------------------------------------- + +const PROBE_RUN = 39713789453; + +/** An offline board for the probe and the self-test: a run and its two attempts' job lists, every read recorded. */ +function fakeBoard({ run = {}, attempt1 = null, attempt2 = null, after = null } = {}) { + const calls = []; + const REPO = RELAY_REPO; + // `run: null` is a run the platform does not answer for; an object patches the measured run. + const base = run === null ? null : { id: PROBE_RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', head_sha: '25b3a6c4bff905d462d4ff44030e5465c342254a', pull_requests: [{ number: 25155 }], ...run }; + const jobs1 = attempt1 ?? [ + { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + { id: 119611140839, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }, + { id: 119617410962, name: 'Test Core', run_attempt: 1, status: 'completed', conclusion: 'failure' }, + ]; + const jobs2 = attempt2 ?? [ + { id: 119642047246, name: 'Test Core (3/6)', run_attempt: 2, status: 'queued', conclusion: null }, + { id: 119642048282, name: 'Test Core (1/6)', run_attempt: 2, status: 'completed', conclusion: 'success' }, + ]; + let runReads = 0; + const call = async (path) => { + calls.push(`GET ${path.split('?')[0]}`); + const p = path.split('?')[0]; + if (p === `/repos/${REPO}/actions/runs/${PROBE_RUN}`) { + runReads += 1; + const json = runReads > 1 && after && base ? { ...base, ...after } : base; + return json === null ? { status: 404, json: { message: 'Not Found' }, detail: 'Not Found', call: `GET ${p}` } : { status: 200, json, detail: '', call: `GET ${p}` }; + } + if (p === `/repos/${REPO}/actions/runs/${PROBE_RUN}/attempts/1/jobs`) return jobs1 === 'unread' ? { status: 403, json: { message: 'Forbidden' }, detail: 'Forbidden', call: `GET ${p}` } : { status: 200, json: { total_count: jobs1.length, jobs: jobs1 }, detail: '', call: `GET ${p}` }; + if (p === `/repos/${REPO}/actions/runs/${PROBE_RUN}/attempts/2/jobs`) return { status: 200, json: { total_count: jobs2.length, jobs: jobs2 }, detail: '', call: `GET ${p}` }; + return { status: 404, json: { message: 'Not Found' }, detail: 'Not Found', call: `GET ${p}` }; + }; + return { call, calls }; +} + +/** + * The no-run conformance probe: ONE re-run through this tool's real path against the offline board, on the route and + * the sender the conformance hands in. Answers the exit, every request that left this process directly + * (`METHOD /path` — reads only, by construction), and what the tool printed. + */ +export async function relayMissProbe({ route, send }) { + const board = fakeBoard(); + const lines = []; + const r = await runRerunFailed({ run: PROBE_RUN, repo: RELAY_REPO, dryRun: false, json: false }, { call: board.call, route, send, log: (l) => lines.push(l), env: {}, token: 'ghs_FixtureTokenNotRealAtAll0000000000000' }); + return { exit: r.exit, calls: [...board.calls], text: lines.join('\n') }; +} + +// --------------------------------------------------------------------------- +// Self-test — offline. The battery roster is the floor (AGENTS.md, "Writing a --self-test"). +// --------------------------------------------------------------------------- + +const SELF_TEST_BATTERIES = Object.freeze({ + 'the command line: --run a positive integer, --repo in the one organization, the board repo by default, anything else refused': 6, + 'the fences at the seat, advisory: the measured run passes all four; each failing fence refused by name with nothing sent; a run or job list that cannot be read, or one with no failed job, sends nothing': 8, + 'relay-only: a direct route, or a route that errs, refuses with zero writes and no dispatch; the dispatch route packs ONE action, run_id alone; --dry-run prints it and sends nothing': 5, + "the outcomes: a success run is read back at the seat naming attempt 2's re-run job ids; a failed run re-reads the run and names what the fences say now, exit 5; no run or none completed is UNCONFIRMED, exit 6; never a second dispatch": 6, + 'the wiring: this file issues no write of its own, imports the fences rather than restating them, and declares itself relay-only': 3, +}); +const SELF_TEST_BATTERY_FLOOR = 5; +const UNATTRIBUTED_BATTERY = '(unattributed)'; + +const batteryCases = new Map(); +let openBattery = null; +const battery = (name) => { + openBattery = name; +}; +let selfTestReachedVerdict = false; + +export async function selfTest() { + const cases = []; + const t = (name, actual, expected = true, detail) => { + const bucket = openBattery ?? UNATTRIBUTED_BATTERY; + batteryCases.set(bucket, (batteryCases.get(bucket) ?? 0) + 1); + const ok = JSON.stringify(actual) === JSON.stringify(expected); + cases.push({ name, ok, detail: ok ? '' : `${detail ? `${detail} — ` : ''}got ${JSON.stringify(actual)}, want ${JSON.stringify(expected)}` }); + }; + const SESSION = 'session_01ABCDEFGHJKMNPQRSTVWXYZ'; + const DISPATCH_ROUTE = { requested: 'auto', transport: 'dispatch', reason: 'self-test: dispatch', error: null, session: SESSION }; + const DIRECT_ROUTE = { requested: 'direct', transport: 'direct', reason: 'self-test: direct', error: null, session: null }; + const RELAY_RUN = { id: 42, url: 'https://github.test/run/42', status: 'completed' }; + /** A fake relay: counts dispatches and answers the shape `sendFleetWrite` returns for the outcome named. */ + const relay = (outcome, sent) => async (payload) => { + sent.push(payload); + const base = { requestId: payload.request_id, startMs: 90_000, ceilingMs: 300_000, status: 204, verdict: 'ok', dispatchedAt: 0 }; + if (outcome === 'success') return { ...base, state: 'success', ok: true, run: { ...RELAY_RUN, conclusion: 'success' }, detail: 'conclusion success' }; + if (outcome === 'failure') return { ...base, state: 'failure', ok: false, run: { ...RELAY_RUN, conclusion: 'failure' }, detail: 'conclusion failure' }; + if (outcome === 'timeout') return { ...base, state: 'timeout', ok: false, run: { ...RELAY_RUN, status: 'in_progress', conclusion: null }, detail: 'run 42 in_progress' }; + return { ...base, state: 'no-run', ok: false, run: null, detail: 'no run appeared' }; + }; + const drive = async (options, { board = {}, route = DISPATCH_ROUTE, outcome = 'success', env = {} } = {}) => { + const b = fakeBoard(board); + const sent = []; + const lines = []; + const r = await runRerunFailed({ run: PROBE_RUN, repo: RELAY_REPO, dryRun: false, json: false, ...options }, { call: b.call, route, send: relay(outcome, sent), log: (l) => lines.push(l), env, token: 'ghs_FixtureTokenNotRealAtAll0000000000000' }); + return { ...r, calls: b.calls, sent, text: lines.join('\n') }; + }; + const RUN_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}`; + const A1_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}/attempts/1/jobs`; + const A2_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}/attempts/2/jobs`; + + // ── the command line ────────────────────────────────────────────────────── + battery('the command line: --run a positive integer, --repo in the one organization, the board repo by default, anything else refused'); + { + t('--run alone: the board repo, no dry run', parseOptions(['--run', '39713789453']), { ok: true, options: { run: 39713789453, repo: RELAY_REPO, dryRun: false, json: false } }); + t('--repo names another fleet repository, --dry-run and --json are read', parseOptions(['--run', '7', '--repo', 'objectstack-ai/objectui', '--dry-run', '--json']).options, { run: 7, repo: 'objectstack-ai/objectui', dryRun: true, json: true }); + t('⛔ --run that is not a positive integer is refused — zero, a sign, a float, a word, nothing', ['0', '-1', '1.5', 'abc', '01'].map((v) => parseOptions(['--run', v]).ok).concat(parseOptions(['--run']).ok), [false, false, false, false, false, false]); + t('⛔ no --run is refused, naming the flag', [parseOptions([]).ok, parseOptions([]).error?.includes('--run')], [false, true]); + t('⛔ a repository outside the organization, or with a path, is refused', [parseOptions(['--run', '7', '--repo', 'someone/objectstack']).ok, parseOptions(['--run', '7', '--repo', 'objectstack-ai/a/b']).ok], [false, false]); + t('⛔ an unknown flag is refused, never ignored — a workflow or an attempt is not this tool\'s to name', [parseOptions(['--run', '7', '--workflow', 'ci.yml']).ok, parseOptions(['--run', '7', '--attempt', '1']).ok], [false, false]); + } + + // ── the fences at the seat ──────────────────────────────────────────────── + battery('the fences at the seat, advisory: the measured run passes all four; each failing fence refused by name with nothing sent; a run or job list that cannot be read, or one with no failed job, sends nothing'); + { + const ok = await drive({}); + t('accepted: the measured run passes the four fences — the run read, then attempt 1\'s jobs — and the line names the failed jobs and the pull request head', [ok.exit, ok.calls.slice(0, 2), ok.text.includes('attempt 1 failed `Test Core (3/6)`, `Test Core`'), ok.text.includes('head 25b3a6c · pull #25155')], [EXIT_OK, [RUN_PATH, A1_PATH], true, true], ok.text); + const fenced = async (run, fence) => { + const r = await drive({}, { board: { run } }); + return [r.exit, r.sent.length, r.calls.length, r.text.includes(`fence \`${fence}\``) && r.text.includes('Nothing sent')]; + }; + t('⛔ fence workflow, refused at the seat: a run of lint.yml — exit 2, no dispatch, no further read', await fenced({ path: '.github/workflows/lint.yml' }, 'workflow'), [EXIT_USAGE, 0, 1, true]); + t('⛔ fence event, refused at the seat: a merge_group run', await fenced({ event: 'merge_group' }, 'event'), [EXIT_USAGE, 0, 1, true]); + t('⛔ fence once, refused at the seat: a run already at attempt 2 — a second failure is real', await fenced({ run_attempt: 2 }, 'once'), [EXIT_USAGE, 0, 1, true]); + t('⛔ fence failed, refused at the seat: a run that succeeded', await fenced({ conclusion: 'success' }, 'failed'), [EXIT_USAGE, 0, 1, true]); + const unread = await drive({}, { board: { run: null } }); + t('a run that cannot be read is NOT MEASURED: exit 3, no dispatch', [unread.exit, unread.sent.length, unread.text.includes('NOT MEASURED')], [EXIT_PREREQUISITE, 0, true]); + const noJobs = await drive({}, { board: { attempt1: 'unread' } }); + t('a job list that cannot be read: exit 3, no dispatch', [noJobs.exit, noJobs.sent.length], [EXIT_PREREQUISITE, 0]); + const passed = await drive({}, { board: { attempt1: [{ id: 1, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }] } }); + t('⛔ a job list with no failed job: exit 2, no dispatch — the relay would refuse a re-run with nothing to confirm', [passed.exit, passed.sent.length, passed.text.includes('none concluded failure')], [EXIT_USAGE, 0, true]); + } + + // ── relay-only ──────────────────────────────────────────────────────────── + battery('relay-only: a direct route, or a route that errs, refuses with zero writes and no dispatch; the dispatch route packs ONE action, run_id alone; --dry-run prints it and sends nothing'); + { + const direct = await drive({}, { route: DIRECT_ROUTE }); + t('⛔ a direct route is a PREREQUISITE refusal: exit 3, no dispatch, the reads only, the line saying RELAY-ONLY and never a direct write', [direct.exit, direct.sent.length, direct.calls, direct.text.includes('RELAY-ONLY') && direct.text.includes('Not falling back')], [EXIT_PREREQUISITE, 0, [RUN_PATH, A1_PATH], true]); + const erring = await drive({}, { route: { requested: 'dispatch', transport: 'dispatch', reason: 'x', error: 'OS_FLEET_TRANSPORT=dispatch but the relay is not live.', session: SESSION } }); + t('⛔ a route that errs is a PREREQUISITE refusal, its error printed, no dispatch', [erring.exit, erring.sent.length, erring.text.includes('the relay is not live')], [EXIT_PREREQUISITE, 0, true]); + const sent = await drive({}); + t('the dispatch route sends ONE stroke of ONE action: op run_rerun_failed and run_id alone, the session on the envelope', [sent.sent.length, sent.sent[0]?.actions, sent.sent[0]?.session, sent.sent[0]?.repo], [1, [{ op: OP, run_id: PROBE_RUN }], SESSION, RELAY_REPO]); + const dry = await drive({ dryRun: true }, { env: { OS_FLEET_SESSION: SESSION } }); + t('--dry-run: the fences read and judged, the relay request printed, nothing sent, exit 0', [dry.exit, dry.sent.length, dry.dryRun?.event_type, dry.dryRun?.client_payload?.actions], [EXIT_OK, 0, RELAY_EVENT_TYPE, [{ op: OP, run_id: PROBE_RUN }]]); + const drySessionless = await drive({ dryRun: true }, { env: {} }); + t('⛔ --dry-run with no session id is exit 3 — the envelope carries the seat\'s identity and it is never invented', [drySessionless.exit, drySessionless.sent.length], [EXIT_PREREQUISITE, 0]); + } + + // ── the outcomes ────────────────────────────────────────────────────────── + battery("the outcomes: a success run is read back at the seat naming attempt 2's re-run job ids; a failed run re-reads the run and names what the fences say now, exit 5; no run or none completed is UNCONFIRMED, exit 6; never a second dispatch"); + { + const landed = await drive({}, { board: { after: { run_attempt: 2, status: 'queued', conclusion: null } } }); + t('⭐ success: exit 0 after ONE dispatch, the run and attempt 2\'s jobs read back, the line naming the re-run job id', [landed.exit, landed.sent.length, landed.calls.slice(2), landed.text.includes('attempt 2 · `Test Core (3/6)` job 119642047246 queued'), landed.reading?.jobs], [EXIT_OK, 1, [RUN_PATH, A2_PATH], true, [{ name: 'Test Core (3/6)', id: 119642047246, status: 'queued' }]], landed.text); + const unreadAfter = await drive({}, { board: { after: { run_attempt: 1 } } }); + t('…a reading that does not see attempt 2 at the seat is SAID, and the exit stays 0 — the executor\'s read-back is the verdict, this is a reading', [unreadAfter.exit, unreadAfter.text.includes('a reading, not a verdict')], [EXIT_OK, true]); + const failed = await drive({}, { outcome: 'failure', board: { after: { run_attempt: 2, status: 'in_progress', conclusion: null } } }); + t('⛔ a failed relay run: exit 5, ONE dispatch, the run re-read and the fence that fails NOW named — here a person re-ran it first', [failed.exit, failed.sent.length, failed.text.includes('FAILED') && failed.text.includes('fence `once`: the run reads 2')], [EXIT_PLATFORM_REFUSAL, 1, true], failed.text); + const failedHolding = await drive({}, { outcome: 'failure' }); + t('…and when the fences still hold, the line says to read the relay run, never to re-send', [failedHolding.exit, failedHolding.text.includes('read the relay run') && failedHolding.text.includes('Do not re-send blind')], [EXIT_PLATFORM_REFUSAL, true]); + const noRun = await drive({}, { outcome: 'no-run' }); + const stuck = await drive({}, { outcome: 'timeout' }); + t('⛔ no run, or none completed: UNCONFIRMED exit 6 after ONE dispatch, the shared sentence printed', [[noRun.exit, noRun.sent.length, noRun.text.includes('UNCONFIRMED') && noRun.text.includes('do not re-run')], [stuck.exit, stuck.sent.length, stuck.text.includes('UNCONFIRMED')]], [[EXIT_UNCONFIRMED, 1, true], [EXIT_UNCONFIRMED, 1, true]]); + const probe = await relayMissProbe({ route: DISPATCH_ROUTE, send: relay('no-run', []) }); + t('the conformance probe answers what the tool does: exit 6, reads only, the UNCONFIRMED sentence', [probe.exit, probe.calls.every((c) => c.startsWith('GET ')), probe.text.includes('UNCONFIRMED')], [EXIT_UNCONFIRMED, true, true]); + } + + // ── the wiring ──────────────────────────────────────────────────────────── + battery('the wiring: this file issues no write of its own, imports the fences rather than restating them, and declares itself relay-only'); + { + const { readFileSync } = await import('node:fs'); + const own = readFileSync(SELF_PATH, 'utf8'); + const code = own.replace(/\/\*[\s\S]*?\*\//g, '').replace(/^\s*\/\/.*$/gm, ''); + t('⛔ no write verb in this file\'s code — its one write is the dispatch, which sendFleetWrite sends and paces', /\b(?:method|verb)\s*:\s*'(?:POST|PATCH|PUT|DELETE)'/.test(code), false); + t('the fences are the relay table\'s, imported — no fence predicate or allowlist is spelled here', [/\brerunFailedFenceFailures\b/.test(own), /RERUN_FAILED_WORKFLOWS\s*=/.test(code), /'\.github\/workflows\/ci\.yml'\]/.test(code)], [true, false, false]); + t('the relay-only declaration the conformance reads is set', relayOnly, true); + } + + // ── the floor, BEFORE the verdict ───────────────────────────────────────── + const declared = Object.keys(SELF_TEST_BATTERIES); + const floor = []; + if (declared.length < SELF_TEST_BATTERY_FLOOR) { + floor.push(`the battery ledger declares ${declared.length} batteries, below its floor of ${SELF_TEST_BATTERY_FLOOR} — a section that stopped being declared is a section nothing floors.`); + } + for (const [name, count] of batteryCases) { + if (name in SELF_TEST_BATTERIES) continue; + floor.push(`self-test battery "${name}" registered ${count} case(s) but is not declared in SELF_TEST_BATTERIES — an assertion attributed to no declared battery is one nothing floors.`); + } + for (const name of declared) { + const count = batteryCases.get(name) ?? 0; + if (count >= SELF_TEST_BATTERIES[name]) continue; + floor.push( + count === 0 + ? `self-test battery "${name}" DID NOT RUN — 0 cases registered, ${SELF_TEST_BATTERIES[name]} pinned. The verdict below would have claimed those cases hold.` + : `self-test battery "${name}" registered ${count} case(s), below its pinned floor of ${SELF_TEST_BATTERIES[name]} — cases that used to run no longer do.`, + ); + } + for (const message of floor) cases.push({ name: message, ok: false, detail: '' }); + + const failedCases = cases.filter((c) => !c.ok); + for (const c of failedCases) console.error(` ✗ ${c.name}${c.detail ? ` — ${c.detail}` : ''}`); + if (failedCases.length) { + console.error(`✗ rerun-failed self-test: ${failedCases.length} of ${cases.length} case(s) failed.`); + selfTestReachedVerdict = true; + return 1; + } + console.log( + `✓ rerun-failed self-test: ${cases.length} cases pass across ${declared.length} batteries — the four fences judged at the seat as an advisory refusal ` + + 'naming each failing one, nothing sent; relay-only, a direct route refused with zero writes; ONE dispatch of ONE action carrying run_id alone; ' + + "a success read back naming attempt 2's job ids, a failure re-read and explained, and no run UNCONFIRMED — never a second dispatch.", + ); + selfTestReachedVerdict = true; + return 0; +} + +// --------------------------------------------------------------------------- +// Dispatch +// --------------------------------------------------------------------------- + +const USAGE = [ + 'usage:', + ' node scripts/pm/rerun-failed.mjs --run [--repo objectstack-ai/] [--dry-run] [--json]', + ' node scripts/pm/rerun-failed.mjs --self-test', + '', + ' Re-runs the FAILED jobs of one ci.yml pull-request run, once, through the fleet-write relay (relay-only).', + ' Only after the seat has written on the PR or the card why it judges the failure a flake; a second failure is real.', + ` Exits: ${EXIT_OK} re-run started (or dry run) · ${EXIT_USAGE} usage / a fence refused / payload refused · ${EXIT_PREREQUISITE} prerequisite ·`, + ` ${EXIT_PLATFORM_REFUSAL} dispatch refused or the relay run failed · ${EXIT_UNCONFIRMED} UNCONFIRMED · 10 throttle refused`, +].join('\n'); + +/** Route node's `fetch` through the session proxy before the first request — a re-exec, as every fleet tool does. */ +function rearmThroughProxy(args) { + const plan = proxyRearmPlan({ env: process.env, execArgv: process.execArgv, flagSupported: process.allowedNodeEnvironmentFlags.has(PROXY_FLAG), guard: PROXY_REARM_GUARD }); + if (plan.hint) { + console.error(`ℹ️ ${plan.reason}. A refusal below may be about the route, not this container.`); + return null; + } + if (!plan.rearm) return null; + console.error(`ℹ️ re-exec with ${plan.flag}: ${plan.reason}.`); + const quiet = process.allowedNodeEnvironmentFlags.has('--disable-warning') ? ['--disable-warning=UNDICI-EHPA'] : []; + const child = spawnSync(process.execPath, [plan.flag, ...quiet, SELF_PATH, ...args], { stdio: 'inherit', env: { ...process.env, [PROXY_REARM_GUARD]: '1' } }); + if (typeof child.status === 'number') return child.status; + console.error(`⚠️ could not re-exec with ${plan.flag} (${child.error?.message ?? 'no exit status'}); continuing in-process — every request will bypass the proxy.`); + return null; +} + +export async function main(argv) { + if (argv.includes('--help') || argv.includes('-h')) { + console.log(USAGE); + return EXIT_OK; + } + if (argv.includes('--self-test')) return selfTest(); + const parsed = parseOptions(argv); + if (!parsed.ok) { + console.error(`rerun-failed: ${parsed.error}\n`); + console.error(USAGE); + return EXIT_USAGE; + } + if (!(process.env.GITHUB_TOKEN ?? process.env.GH_TOKEN)) { + console.error('rerun-failed: PREREQUISITE NOT MET — no GITHUB_TOKEN / GH_TOKEN in the environment. ⛔ NOT MEASURED: nothing was read and nothing was sent.'); + return EXIT_PREREQUISITE; + } + const relayed = rearmThroughProxy(argv); + if (relayed !== null) return relayed; + const result = await runRerunFailed(parsed.options, {}); + if (result.dryRun) console.log(JSON.stringify(result.dryRun, null, 2)); + if (parsed.options.json) { + console.log(JSON.stringify({ repo: result.repo, run: result.run, exit: result.exit, fences: result.fences, failed_jobs: result.failed, transport: result.transport, relay: result.relay, reading: result.reading })); + } + return result.exit; +} + +// ⛔ No top-level await on main: this file imports `fleet-write/dispatch.mjs`, whose read-back imports post-stamped, +// which imports dispatch back — the entry settles first, as dispatch.mjs's own entry does. +if (isEntrypoint(import.meta.url)) { + main(process.argv.slice(2)).then((code) => { + if (process.argv.includes('--self-test') && !selfTestReachedVerdict) { + console.error( + '\n✗ rerun-failed self-test: selfTest() returned without reaching its verdict, so no success line was\n' + + 'printed. Exiting 0 here would report a self-test that never finished as one that passed.\n', + ); + process.exit(1); + } + process.exit(code); + }); +} From 3a8aecdd61acce2e0de811288cf7efb04337c89e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 07:11:22 +0000 Subject: [PATCH 4/6] docs(pm-dispatch): the seat's once-only re-run line, in the re-run readings The line replaces the dated reading that a re-run was doable through MCP: a seat re-runs a PR's failed ci.yml jobs once through rerun-failed.mjs, only after writing down why it judges the failure a flake; a second failure is real. The REST 403 / MCP 201 reading stands at its earlier line. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- .claude/skills/pm-dispatch/references/platform-readings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.claude/skills/pm-dispatch/references/platform-readings.md b/.claude/skills/pm-dispatch/references/platform-readings.md index 9ce3f3a076f..93106aea40f 100644 --- a/.claude/skills/pm-dispatch/references/platform-readings.md +++ b/.claude/skills/pm-dispatch/references/platform-readings.md @@ -289,7 +289,7 @@ - 判定跨天翻面 ⇒ ⛔ 一次拒绝不是能力边界;`permissions.allow` 条目才是仓库侧确定性通道。 - `rerun_failed_jobs` 复用原 run 的提交与合并 ref,不拿新 main 重算。 - ⇒ 基上缺已合修复时重跑无效,只能 `git merge origin/main` 推提交;判别看修复合并时间。 -- 读数 2026-09-13:席位 REST `/jobs/{id}/rerun` 403、MCP `rerun_failed_jobs` 201 ⇒ 重跑可做,锁 1 未禁。 +- 经 `rerun-failed.mjs` 重跑 PR `ci.yml` 失败 job 仅一次,先在 PR 或卡写明 flake 理由;二败即真红。 - 同一 head 上轻量兄弟 workflow `success` 加重量级载体 `cancelled` 是普通取代的预期签名。 - 同名 `failure` 也会被带另一诊断的后一次 `failure` 取代 ⇒ 任何判定前先按名取最新一次。 - cancel-in-progress 窗口只罩得住慢载体 ⇒ 先比对 run `head_sha` 与 PR 当前 head,不开调查。 From 120ca3ca5808157edb1088f671406713601aef93 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 08:23:40 +0000 Subject: [PATCH 5/6] =?UTF-8?q?feat(pm/fleet-write):=20run=5Frerun=5Ffaile?= =?UTF-8?q?d's=20seventh=20fence=20=E2=80=94=20the=20run's=20head=20is=20i?= =?UTF-8?q?ts=20pull's=20current=20head?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run_rerun_failed is now refused unless the run's head_sha equals the current head of the pull it names. The pull is the run's pull_requests[0].number, and its head is read with the pull's own GET /repos/{repo}/pulls/{n}, never from the copy the run embeds. A run naming no pull (a fork pull request's run, or the field empty) is refused by name. ci.yml's concurrency group is keyed by the pull number with cancel-in-progress true, so a stale head's re-run would cancel the current head's run. The executor judges it authoritatively: the run and then the pull are the last two reads before the POST, and a stale head, a run naming no pull or an unreadable pull is zero writes with the POST row not sent. The wrapper judges it earlier as an advisory refusal naming the fence. RERUN_FAILED_FENCES gains the head fence, every predicate now takes (run, pull), and the self-test batteries that judge it gain cases on both sides, with their pins raised: validate 30 to 33, execute 17 to 21, rerun-failed's seat fences 8 to 11. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- scripts/pm/fleet-write/dispatch.mjs | 5 +- scripts/pm/fleet-write/execute.mjs | 92 +++++++++++++++------- scripts/pm/fleet-write/ops.mjs | 82 +++++++++++++++----- scripts/pm/fleet-write/validate.mjs | 35 ++++++--- scripts/pm/rerun-failed.mjs | 115 ++++++++++++++++++---------- 5 files changed, 225 insertions(+), 104 deletions(-) diff --git a/scripts/pm/fleet-write/dispatch.mjs b/scripts/pm/fleet-write/dispatch.mjs index 60d5e1cd255..815c5bf8958 100644 --- a/scripts/pm/fleet-write/dispatch.mjs +++ b/scripts/pm/fleet-write/dispatch.mjs @@ -242,8 +242,9 @@ * IS that confirmation; the seat reads the new head with one * `GET /repos/{repo}/pulls/{n}` after, and this file reads nothing back; * - `run_rerun_failed` carries no body, names no workflow and is - * confirmed INSIDE the run too: the executor judges the four fences on - * the run before its POST and lands the action only when the run reads + * confirmed INSIDE the run too: the executor judges the five fences on + * the run and its pull's current head before its POST, and lands the + * action only when the run reads * back as attempt 2 with its failed jobs queued or in progress * (`execute.mjs`, `confirmRerunFailed`). So a `success` run IS that * confirmation and this file reads nothing back; the seat's wrapper diff --git a/scripts/pm/fleet-write/execute.mjs b/scripts/pm/fleet-write/execute.mjs index 0b788f75486..e5dc1de5a76 100644 --- a/scripts/pm/fleet-write/execute.mjs +++ b/scripts/pm/fleet-write/execute.mjs @@ -125,19 +125,24 @@ * * ## The failed-jobs re-run — `run_rerun_failed`, fenced on the RUN and read back * - * The four fences on the run (`RERUN_FAILED_FENCES` in `ops.mjs`: the - * workflow, the event, once, failed) are facts about the run, so this file is - * where they are AUTHORITATIVE (`rerunFailedPreflight`). It first reads the - * job list of the run's attempt 1, which is the read-back's baseline: the - * names of the jobs that concluded `failure`. It then reads - * `GET /repos/{repo}/actions/runs/{run_id}` — the last request before the - * POST — and judges the four fences on that answer. A run that could not be - * read, a fence that fails (every failing one named, with what the run read), - * a baseline that could not be read whole, or a baseline with no failed job - * stops the action with the POST NOT SENT: its row says `not sent`, zero - * writes, exit 5. A run re-run or changed by anyone between the seat's - * request and this read is refused here. ⛔ The seat wrapper's earlier check - * is advisory and is never trusted in place of this one. + * The five fences on the run and its pull (`RERUN_FAILED_FENCES` in + * `ops.mjs`: the workflow, the event, once, failed, the head) are facts about + * the run, so this file is where they are AUTHORITATIVE + * (`rerunFailedPreflight`). It first reads the job list of the run's attempt + * 1, which is the read-back's baseline: the names of the jobs that concluded + * `failure`. It then reads `GET /repos/{repo}/actions/runs/{run_id}` and, for + * the pull the run names (`pull_requests[0].number`), that pull's own + * `GET /repos/{repo}/pulls/{n}` — the last two requests before the POST — and + * judges the five fences on those answers. The head fence compares the run's + * `head_sha` with the pull's CURRENT head from that GET, never with the copy + * the run embeds, and a run naming no pull is refused by name without a pull + * read. A run or pull that could not be read, a fence that fails (every + * failing one named, with what was read), a baseline that could not be read + * whole, or a baseline with no failed job stops the action with the POST NOT + * SENT: its row says `not sent`, zero writes, exit 5. A run re-run, or a pull + * pushed to, by anyone between the seat's request and these reads is refused + * here. ⛔ The seat wrapper's earlier check is advisory and is never trusted + * in place of this one. * * The POST answers 201 with an empty body, which is an acceptance, not a * landing. `confirmRerunFailed` re-reads the run every `RERUN_FAILED_POLL_MS` @@ -277,6 +282,7 @@ import { rerunFailedFenceFailures, rerunFailedFenceText, rerunFailedLandedText, + rerunFailedPullNumber, rerunFailedReadBack, transferRemedy, } from './ops.mjs'; @@ -555,8 +561,10 @@ async function readAttemptJobs(repo, runId, attempt, api, t) { /** * The fences, AUTHORITATIVE (header: "The failed-jobs re-run"): attempt 1's job list (the read-back's baseline), - * then the run — the last read before the POST — judged against `RERUN_FAILED_FENCES`. Returns `{ run, failed }` - * to send, or `{ fail: why }` — the sentence the row prints, the POST NOT SENT. Never throws on a status. + * then the run, then the pull it names — the last two reads before the POST — judged against + * `RERUN_FAILED_FENCES`. The pull's head is the pull's OWN answer, never the run's embedded copy. Returns + * `{ run, pull, failed }` to send, or `{ fail: why }` — the sentence the row prints, the POST NOT SENT. Never throws + * on a status. */ export async function rerunFailedPreflight(req, payload, api, t) { const { runId } = req.rerunFailed; @@ -565,12 +573,22 @@ export async function rerunFailedPreflight(req, payload, api, t) { if (run.status !== 200 || !run.json || typeof run.json !== 'object') { return { fail: `${run.call} -> HTTP ${run.status}${run.detail ? ` (${run.detail})` : ''}: the run could not be read, so its fences were not judged — nothing sent` }; } - const failures = rerunFailedFenceFailures(run.json); + // The head fence's read: the pull the run names, by its own GET. A run naming none is judged without it and refused by name. + const number = rerunFailedPullNumber(run.json); + let pull = null; + if (number !== null) { + const p = await rest(api, `/repos/${payload.repo}/pulls/${number}`, {}, t); + if (p.status !== 200 || !p.json || typeof p.json !== 'object') { + return { fail: `${p.call} -> HTTP ${p.status}${p.detail ? ` (${p.detail})` : ''}: pull #${number}, the run's pull request, could not be read, so the head fence was not judged — nothing sent` }; + } + pull = p.json; + } + const failures = rerunFailedFenceFailures(run.json, pull); if (failures.length) return { fail: `REFUSED at the relay's fence on run ${runId}, nothing sent — ${rerunFailedFenceText(failures)}` }; if (baseline.fail) return { fail: `${baseline.fail.why}, so the read-back has no baseline — nothing sent` }; const failed = failedJobNames(baseline.jobs); if (!failed.length) return { fail: `attempt ${RERUN_FAILED_ATTEMPT} of run ${runId} lists ${baseline.jobs.length} job(s) and none concluded failure, so a re-run's read-back would have nothing to confirm — nothing sent` }; - return { run: run.json, failed }; + return { run: run.json, pull, failed }; } /** @@ -931,7 +949,7 @@ export async function executeFleetWrite({ payload: raw, sender, token, api = DEF break; } const req = filled.req; - // The failed-jobs re-run (header): its fences on the run, judged HERE on the read before the POST — refused, NOT SENT. + // The failed-jobs re-run (header): its fences on the run and its pull, judged HERE on the reads before the POST — refused, NOT SENT. const pre = req.rerunFailed ? await rerunFailedPreflight(req, payload, api, t) : null; if (pre?.fail) { const why = scrub(pre.fail, [token]); @@ -1026,7 +1044,8 @@ const SELF_TEST_BATTERIES = Object.freeze({ 'the release.yml refresh: ONE POST to its dispatch endpoint carrying { ref: main, inputs: { refresh_version_pr: "true" } } under the App token, landing as accepted; force, false, absent, a string or any other input refused with ZERO requests, exit 2': 9, "the batch gap on the runner: a stroke carrying DEFAULT_BATCH_THRESHOLD or more content-creating actions is announced as ONE rule ⑤ batch before its first write — on an injected clock every write through the last content-creating one leaves a full DEFAULT_BATCH_GAP_MS after the one before, and the pacing is exactly what the seat budgets; below the threshold, after the batch and on a landing stroke rule ①'s gap; a stroke the job cannot hold refused with ZERO requests; an announcement the log cannot record refused before the first write": 10, "the governed-review follow-up: after pr_create (at the number the create answered) and after pr_ready, the pull's file list is read in full; a Tier H hit requests review from every GOVERNED_APPROVERS account not already requested, assigned or reviewed — the pull's own author assigned instead, never requested; a Tier S or ungoverned diff sends nothing and says so; an unreadable or truncated list, an unreadable pull or review list, or a refused write is a FAILED action": 29, - "the run_rerun_failed: the four fences judged HERE on the run read immediately before ONE bodiless POST — each refused with the POST not sent and accepted on the measured run; a run, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation": 17, + // 17 → 21 with the head fence: a stale head, the embedded copy never trusted, a run naming no pull, a pull unread. + "the run_rerun_failed: the five fences judged HERE on the run and its pull, read immediately before ONE bodiless POST — the pull's current head by its own GET, never the run's embedded copy, and a run naming no pull refused by name; each refused with the POST not sent and accepted on the measured run; a run, pull, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation": 21, }); // Raised 17 → 18 with the governed-review follow-up battery, 18 → 19 with the run_rerun_failed battery: one battery, one row. const SELF_TEST_BATTERY_FLOOR = 19; @@ -1495,16 +1514,22 @@ export async function selfTest() { } // ── the failed-jobs re-run ────────────────────────────────────────────── - battery("the run_rerun_failed: the four fences judged HERE on the run read immediately before ONE bodiless POST — each refused with the POST not sent and accepted on the measured run; a run, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation"); + battery("the run_rerun_failed: the five fences judged HERE on the run and its pull, read immediately before ONE bodiless POST — the pull's current head by its own GET, never the run's embedded copy, and a run naming no pull refused by name; each refused with the POST not sent and accepted on the measured run; a run, pull, baseline or fence that fails is zero writes; the 201 confirmed only by reading attempt 2 back with its failed jobs queued or in progress, naming their job ids; attempt 2 never seen, a failed job completed, or a job list unread is FAILED; no annotation"); { const RUN = 39713789453; + const PULL = 25155; + const HEAD = '25b3a6c4bff905d462d4ff44030e5465c342254a'; + const MOVED = '9f0e1d2c3b4a59687766554433221100ffeeddcc'; const RUN_GET = `GET /repos/${REPO}/actions/runs/${RUN}`; + const PULL_GET = `GET /repos/${REPO}/pulls/${PULL}`; const A1 = `GET /repos/${REPO}/actions/runs/${RUN}/attempts/1/jobs`; const A2 = `GET /repos/${REPO}/actions/runs/${RUN}/attempts/2/jobs`; const RERUN = `POST /repos/${REPO}/actions/runs/${RUN}/rerun-failed-jobs`; // The measured run (39713789453): attempt 1 failed in one shard and the aggregate that needs it; attempt 2 listed the // shard queued at once, the passed shards completed (carried over), and the aggregate only when its need finished. - const runAt = (patch = {}) => ({ status: 200, json: { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', ...patch } }); + // It ran on pull #25155's head, and the run EMBEDS a copy of that pull — which the head fence never reads. + const runAt = (patch = {}) => ({ status: 200, json: { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', head_sha: HEAD, pull_requests: [{ number: PULL, head: { sha: HEAD } }], ...patch } }); + const pullAt = (sha = HEAD) => ({ status: 200, json: { number: PULL, state: 'open', head: { sha } } }); const attempt2Run = runAt({ run_attempt: 2, status: 'queued', conclusion: null }); const attempt1 = { status: 200, json: { total_count: 3, jobs: [ { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, @@ -1518,27 +1543,36 @@ export async function selfTest() { let clock = 0; const timing = { sleep: async (ms) => { clock += ms; }, now: () => clock }; const stroke = base([{ op: 'run_rerun_failed', run_id: RUN }]); - const happy = { ...allowed, [A1]: attempt1, [RUN_GET]: [runAt(), attempt2Run], [RERUN]: { status: 201, json: {} }, [A2]: attempt2() }; + const happy = { ...allowed, [A1]: attempt1, [RUN_GET]: [runAt(), attempt2Run], [PULL_GET]: pullAt(), [RERUN]: { status: 201, json: {} }, [A2]: attempt2() }; const LANDED = 'attempt 2 · re-run: `Test Core (3/6)` job 119642047246 queued · not listed yet: `Test Core` (a job is listed when its needs finish)'; clock = 0; // Its own pace log, never the shared one: the shared log's later cases spend write-pace's hourly budget for the // fixture token exactly (measured: one more write there refuses the annotations battery's third write, exit 10). const ok = await run(stroke, happy, timing); - t('⭐ the measured run passes every fence: the sender gate, attempt 1\'s jobs (the baseline), the run — the read immediately before the POST — then ONE POST, then the run and attempt 2\'s jobs read back; exit 0', [ok.exit, ok.seen.map((s) => s.call)], [EXIT_OK, [PERM, A1, RUN_GET, RERUN, RUN_GET, A2]], ok.logs.join(' | ')); + t('⭐ the measured run passes every fence: the sender gate, attempt 1\'s jobs (the baseline), the run and then its pull — by the pull\'s own GET, the two reads immediately before the POST — then ONE POST, then the run and attempt 2\'s jobs read back; exit 0', [ok.exit, ok.seen.map((s) => s.call)], [EXIT_OK, [PERM, A1, RUN_GET, PULL_GET, RERUN, RUN_GET, A2]], ok.logs.join(' | ')); t('…the one write is that POST with NO body, under the App token like every request', [writes(ok.seen).map((s) => [s.call, s.body]), ok.seen.every((s) => s.auth === `Bearer ${TOKEN}`)], [[[RERUN, null]], true]); t('…the 201 is confirmed by the read-back: the row names attempt 2, the failed shard\'s new job id queued, and the aggregate not listed yet', [ok.rows[0]?.status, ok.rows[0]?.result, ok.summary.includes(`| 1 | \`run_rerun_failed\` | \`${RERUN}\` | 201 | ${LANDED} |`)], [201, LANDED, true], ok.summary); t('…no other actions/* write leaves, and no annotation is emitted', [ok.seen.filter((s) => s.call.includes('/actions/') && !s.call.startsWith('GET ') && s.call !== RERUN), ok.annotations, ok.logs.filter((l) => l.startsWith('::'))], [[], [], []]); - // Each fence, refused at execute time: the run changed between the seat's request and this run — the POST is NOT SENT. - const fenced = async (patch) => { - const r = await run(stroke, { ...happy, [RUN_GET]: runAt(patch) }); + // Each fence, refused at execute time: the run (or its pull) changed between the seat's request and this run — the POST is NOT SENT. + const fenced = async (patch, answers = {}) => { + const r = await run(stroke, { ...happy, [RUN_GET]: runAt(patch), ...answers }); return [r.exit, writes(r.seen).length, r.seen.at(-1)?.call, r.rows[0]?.call, r.rows[0]?.status, /REFUSED at the relay's fence/.test(r.rows[0]?.result ?? '')]; }; - const refusedRow = [EXIT_ACTION_FAILED, 0, RUN_GET, RERUN, 'not sent', true]; - t('⛔ fence workflow, refused HERE: a run of another workflow — zero writes, the POST row not sent, the run read the last request', [await fenced({ path: '.github/workflows/lint.yml' }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ path: '.github/workflows/lint.yml' }) })).rows[0]?.result.includes('fence `workflow`: the run reads ".github/workflows/lint.yml"')], [refusedRow, true]); + const refusedRow = [EXIT_ACTION_FAILED, 0, PULL_GET, RERUN, 'not sent', true]; + t('⛔ fence workflow, refused HERE: a run of another workflow — zero writes, the POST row not sent, the pull read the last request', [await fenced({ path: '.github/workflows/lint.yml' }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ path: '.github/workflows/lint.yml' }) })).rows[0]?.result.includes('fence `workflow`: the run reads ".github/workflows/lint.yml"')], [refusedRow, true]); t('⛔ fence event, refused HERE: a push run — zero writes, the POST not sent, the event fence named', [await fenced({ event: 'push' }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ event: 'push' }) })).rows[0]?.result.includes('fence `event`: the run reads "push"')], [refusedRow, true]); t('⛔ fence once, refused HERE: a run a person re-ran meanwhile (attempt 2) — zero writes, the POST not sent, the once fence named', [await fenced({ run_attempt: 2 }), (await run(stroke, { ...happy, [RUN_GET]: runAt({ run_attempt: 2 }) })).rows[0]?.result.includes('fence `once`: the run reads 2')], [refusedRow, true]); t('⛔ fence failed, refused HERE: a run that concluded success, and one still in progress — zero writes, the POST not sent, the failed fence named', [await fenced({ conclusion: 'success' }), await fenced({ status: 'in_progress', conclusion: null })], [refusedRow, refusedRow]); + // The head fence: the pull's CURRENT head, by the pull's own GET — accepted on the happy path above (head_sha equal to it). + const stale = await run(stroke, { ...happy, [PULL_GET]: pullAt(MOVED) }); + t('⛔ fence head, refused HERE: the pull was pushed to since the run (a stale head) — zero writes, the POST not sent, the pull read the last request, the head fence naming both heads', [await fenced({}, { [PULL_GET]: pullAt(MOVED) }), stale.rows[0]?.result.includes('fence `head`: the run reads "head_sha 25b3a6c4bf, and pull #25155\'s own GET answers #25155 at head 9f0e1d2c3b"')], [refusedRow, true], stale.rows[0]?.result); + t("⛔ …the run's embedded copy of the pull is never trusted: it agrees with the run's head_sha, and the pull's own answer still refuses", [stale.seen.filter((s) => s.call === PULL_GET).length, runAt().json.pull_requests[0].head.sha === HEAD, writes(stale.seen).length], [1, true, 0]); + const fork = []; + for (const r of [runAt({ pull_requests: [] }), runAt({ pull_requests: undefined })]) fork.push(await run(stroke, { ...happy, [RUN_GET]: r })); + t("⛔ fence head, refused HERE: a run naming no pull request — pull_requests empty (a fork pull request's run) or absent — zero writes, NO pull read, the POST not sent, the head fence named", fork.map((r) => [r.exit, writes(r.seen).length, r.seen.some((s) => s.call.startsWith(`GET /repos/${REPO}/pulls/`)), r.rows[0]?.status, r.rows[0]?.result.includes('fence `head`: the run reads "no pull_requests entry')]), [[EXIT_ACTION_FAILED, 0, false, 'not sent', true], [EXIT_ACTION_FAILED, 0, false, 'not sent', true]]); + const pullUnread = await run(stroke, { ...happy, [PULL_GET]: { status: 404, json: { message: 'Not Found' } } }); + t('⛔ a pull that cannot be read leaves the head fence unjudged: zero writes, the POST not sent, the read named', [pullUnread.exit, writes(pullUnread.seen).length, pullUnread.rows[0]?.status, pullUnread.rows[0]?.result.includes('pull #25155, the run\'s pull request, could not be read, so the head fence was not judged')], [EXIT_ACTION_FAILED, 0, 'not sent', true], pullUnread.rows[0]?.result); const unread = await run(stroke, { ...happy, [RUN_GET]: { status: 404, json: { message: 'Not Found' } } }); t('⛔ a run that cannot be read is never judged: zero writes, the POST not sent, the read named', [unread.exit, writes(unread.seen).length, unread.rows[0]?.status, unread.rows[0]?.result.includes('the run could not be read')], [EXIT_ACTION_FAILED, 0, 'not sent', true]); const noBase = await run(stroke, { ...happy, [A1]: { status: 403, json: { message: 'Resource not accessible by integration' } } }); @@ -1843,7 +1877,7 @@ export async function selfTest() { 'every op as the request the table declares, stop at the first failure with later actions untouched, the idempotent label DELETE, the GraphQL ' + 'ops behind a node-id read, a pull mutation landing only when its answer shows the state it asked for (armed or queued, off, ready, draft), a base sync confirmed by the head leaving its expected sha or measured as a no-op, an assignees leg sent to the number the create answered and landing only when its answer lists every login sent, ' + 'a governed-review follow-up after pr_create and pr_ready that reads the pull\'s whole file list and, on a Tier H hit alone, requests every GOVERNED_APPROVERS account not already covered (the author assigned instead), zero writes when covered and a FAILED action when the list cannot be read whole, a transfer gated on both repositories and landing only on its target, ' + - 'a failed-jobs re-run fenced on the run read just before its one POST (zero writes on any fence) and landing only when attempt 2 reads back with its failed jobs queued or in progress, one summary row per request, ' + + "a failed-jobs re-run fenced on the run and its pull's own current head, read just before its one POST (zero writes on any fence), and landing only when attempt 2 reads back with its failed jobs queued or in progress, one summary row per request, " + "one annotation per created or moved card carrying the number the platform answered, read back by the reader's own parser, a stroke at or " + 'above the batch threshold announced as ONE rule ⑤ batch and paced at the batch gap (one the job cannot hold refused with zero requests), and a known ' + 'token that came back out of NO summary, log or error.', diff --git a/scripts/pm/fleet-write/ops.mjs b/scripts/pm/fleet-write/ops.mjs index 0bf473d1f8b..6b3e31bc88f 100644 --- a/scripts/pm/fleet-write/ops.mjs +++ b/scripts/pm/fleet-write/ops.mjs @@ -196,7 +196,7 @@ * shard was listed at once and started two seconds later; and the aggregate * job that needs every shard was listed only when its need finished, 16 * minutes later. It spends the `actions: write` `workflow_dispatch` spends, so - * it is fenced the same way. These are the six fences: + * it is fenced the same way. These are the seven fences: * * - the WORKFLOW: the run's `path` is one of `RERUN_FAILED_WORKFLOWS`, a * closed list matched whole and joined only by a PR that adds the path @@ -207,6 +207,13 @@ * - ONCE: the run's `run_attempt` is 1. A second failure is real, and a * re-send after a landing meets attempt 2 here and is refused; * - FAILED: the run is `completed` with `conclusion` `failure`; + * - the HEAD: the run's `head_sha` is its pull request's CURRENT head. The + * pull is the run's `pull_requests[0].number`, and its head is read with + * the pull's own `GET /repos/{repo}/pulls/{n}`, never from the copy the + * run embeds. `ci.yml`'s concurrency group is keyed by the pull number + * with `cancel-in-progress: true`, so a stale head's re-run would cancel + * the current head's run. A run with no `pull_requests` entry (a fork + * pull request's run, or the field empty) is refused by name; * - ALONE: a stroke carrying it carries exactly that one action (`alone`), * so the wider token serves one request; * - the GRANT: `actions: write` is minted for that stroke alone, through the @@ -216,14 +223,15 @@ * * The payload is `run_id` and nothing else: no workflow, attempt, job or * debug-logging key exists for it, and the POST carries no body. The first - * four fences are facts about the RUN, not the request, so the EXECUTOR judges - * them (`RERUN_FAILED_FENCES`) on `GET /repos/{repo}/actions/runs/{run_id}`, - * read immediately before the POST. A run that changed between the seat's - * request and the relay's run, for example re-run by a person meanwhile, is - * refused THERE with zero writes. The seat's wrapper - * (`scripts/pm/rerun-failed.mjs`) reads the same four before it dispatches. - * That is an early refusal naming the fence that fails, and it never replaces - * the executor's. + * five fences are facts about the RUN and its pull, not the request, so the + * EXECUTOR judges them (`RERUN_FAILED_FENCES`) on + * `GET /repos/{repo}/actions/runs/{run_id}` and then the pull's + * `GET /repos/{repo}/pulls/{n}`, the last two reads before the POST. A run + * that changed between the seat's request and the relay's run, for example + * re-run by a person meanwhile, or a pull that a push moved on, is refused + * THERE with zero writes. The seat's wrapper (`scripts/pm/rerun-failed.mjs`) + * reads the same five before it dispatches. That is an early refusal naming + * the fence that fails, and it never replaces the executor's. * * The 201 the POST answers is an acceptance, not a landing. The executor reads * the run again until it shows `run_attempt` 2, then that attempt's job list, @@ -394,13 +402,27 @@ export const RERUN_FAILED_RERUN_STATES = Object.freeze(['queued', 'in_progress'] export const RERUN_FAILED_PATH_SHAPE = /\/actions\/runs\/\d+\/rerun-failed-jobs$/; /** - * The four fences on the RUN (the header's `run_rerun_failed` section), each - * judged on the run's own answer: `fence` its name, `wants` what the relay - * re-runs, `reads` the field(s) the failure sentence quotes, `holds` the - * predicate. The other two fences — alone in its stroke, the stroke-scoped - * grant — are the validator's. The executor judges these on the read taken - * immediately before the POST; the seat's wrapper judges them earlier, as an - * advisory refusal. + * The pull request a run's `head` fence reads: the run's `pull_requests[0].number`, or `null` when the run names + * none (a fork pull request's run, whose `pull_requests` the platform leaves empty, or a missing field). Pure. + */ +export function rerunFailedPullNumber(run) { + const n = Array.isArray(run?.pull_requests) ? run.pull_requests[0]?.number : undefined; + return Number.isSafeInteger(n) && n > 0 ? n : null; +} + +const shortHead = (sha) => (typeof sha === 'string' ? sha.slice(0, 10) : null); + +/** + * The five fences on the RUN and its pull (the header's `run_rerun_failed` + * section): `fence` its name, `wants` what the relay re-runs, `reads` the + * field(s) the failure sentence quotes, `holds` the predicate. Each takes + * `(run, pull)`: `run` the run's own answer, `pull` the answer of + * `GET /repos/{repo}/pulls/{n}` for the pull `rerunFailedPullNumber` names, + * or `null` when the run names none. ⛔ The `head` fence never reads the copy + * of the pull the run embeds. The other two fences — alone in its stroke, the + * stroke-scoped grant — are the validator's. The executor judges these on the + * two reads taken immediately before the POST; the seat's wrapper judges them + * earlier, as an advisory refusal. */ export const RERUN_FAILED_FENCES = Object.freeze([ Object.freeze({ @@ -427,11 +449,29 @@ export const RERUN_FAILED_FENCES = Object.freeze([ reads: (run) => `status ${run?.status ?? null}, conclusion ${run?.conclusion ?? null}`, holds: (run) => run?.status === RERUN_FAILED_STATUS && run?.conclusion === RERUN_FAILED_CONCLUSION, }), + Object.freeze({ + fence: 'head', + wants: + "a run whose head_sha is its pull request's CURRENT head (the pull named by pull_requests[0], read with its own GET, " + + "never the run's embedded copy), since ci.yml's concurrency group is keyed by the pull number and a stale head's re-run cancels the current head's run", + reads: (run, pull) => { + const n = rerunFailedPullNumber(run); + if (n === null) return "no pull_requests entry — a fork pull request's run, or none"; + return `head_sha ${shortHead(run?.head_sha)}, and pull #${n}'s own GET answers #${pull?.number ?? null} at head ${shortHead(pull?.head?.sha)}`; + }, + holds: (run, pull) => { + const n = rerunFailedPullNumber(run); + return n !== null && typeof run?.head_sha === 'string' && run.head_sha !== '' && pull?.number === n && pull?.head?.sha === run.head_sha; + }, + }), ]); -/** The fences a run's answer fails: `[{ fence, wants, got }]`, in `RERUN_FAILED_FENCES` order; `[]` when all four hold. Pure. */ -export function rerunFailedFenceFailures(run) { - return RERUN_FAILED_FENCES.filter((f) => !f.holds(run)).map((f) => ({ fence: f.fence, wants: f.wants, got: f.reads(run) ?? null })); +/** + * The fences a run's answer and its pull's fail: `[{ fence, wants, got }]`, in `RERUN_FAILED_FENCES` order; `[]` + * when all five hold. `pull` is the pull's own answer, or `null` when the run names no pull. Pure. + */ +export function rerunFailedFenceFailures(run, pull = null) { + return RERUN_FAILED_FENCES.filter((f) => !f.holds(run, pull)).map((f) => ({ fence: f.fence, wants: f.wants, got: f.reads(run, pull) ?? null })); } /** The sentence a fence refusal prints — every failing fence by name, what the run read and what the relay re-runs. Pure. */ @@ -911,8 +951,8 @@ export const OPS = Object.freeze({ }, }), // The second row whose token carries `actions: write`, minted for a stroke carrying this op alone — see the header. - // `run_id` is its one key. The four fences on the run (`RERUN_FAILED_FENCES`) are judged by the executor on the run's - // own answer, read immediately before this POST, and the POST carries NO body: the platform's optional + // `run_id` is its one key. The five fences on the run and its pull (`RERUN_FAILED_FENCES`) are judged by the executor + // on their own answers, read immediately before this POST, and the POST carries NO body: the platform's optional // debug-logging flag is never sent. run_rerun_failed: Object.freeze({ permission: 'actions', diff --git a/scripts/pm/fleet-write/validate.mjs b/scripts/pm/fleet-write/validate.mjs index fc766a293d1..9293ea97f5f 100644 --- a/scripts/pm/fleet-write/validate.mjs +++ b/scripts/pm/fleet-write/validate.mjs @@ -53,10 +53,11 @@ * too (the row's `alone`), for the same reason: its token carries * `actions: write`. Its one key is `run_id`, a positive integer; a * workflow, an attempt, a job, an event or the platform's debug-logging - * flag beside it is refused by name. The four fences on the RUN — the - * workflow, the event, once, failed (`RERUN_FAILED_FENCES` in `ops.mjs`) - * — are facts this file cannot see: the executor judges them on the run - * read immediately before its POST; + * flag beside it is refused by name. The five fences on the RUN and its + * pull — the workflow, the event, once, failed, the head + * (`RERUN_FAILED_FENCES` in `ops.mjs`) — are facts this file cannot see: + * the executor judges them on the run and the pull's own answer, read + * immediately before its POST; * - a `pr_create` may carry `assignees` and `labels`, the typed lists * `assign` and `labels_add` take — judged here as every list is — and the * op table turns each into a later request of the SAME action, sent by @@ -139,6 +140,7 @@ import { rerunFailedFenceFailures, rerunFailedFenceText, rerunFailedLandedText, + rerunFailedPullNumber, rerunFailedReadBack, } from './ops.mjs'; @@ -776,7 +778,7 @@ const SELF_TEST_BATTERIES = Object.freeze({ 'the token scope: ONE repository for every op but transfer, source + target for a transfer stroke, and the workflow mints from that list': 8, 'the workflow_dispatch row: a closed allowlist of files matched whole, each with its closed inputs, ref main alone, a stroke of exactly that one action, actions write minted for that stroke alone, and the dispatch call and the failed-jobs re-run the ONLY actions paths any row reaches': 18, "the release.yml entry: refresh_version_pr true and nothing beside it — force, false, absent, a string or any other input refused by name; the wire spells the table's value as a string; every entry's inputs declared on disk; and the file's own guards, evaluated, start the bookkeeping lane alone": 22, - "the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the four fences on the run, each accepted and refused; the read-back's judgment; and every other actions path still refused": 30, + "the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the five fences on the run and its pull, each accepted and refused — the head by the pull's own answer, never the run's embedded copy, and a run naming no pull refused by name; the read-back's judgment; and every other actions path still refused": 33, }); // Raised 11 → 12 with the run_rerun_failed battery: one battery, one row. const SELF_TEST_BATTERY_FLOOR = 12; @@ -1195,7 +1197,7 @@ export async function selfTest() { } // ── the run_rerun_failed row ────────────────────────────────────────────── - battery("the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the four fences on the run, each accepted and refused; the read-back's judgment; and every other actions path still refused"); + battery("the run_rerun_failed row: run_id alone, a positive integer, nothing that widens it; one bodiless POST to the run's rerun-failed-jobs; a stroke of exactly that one action; actions write minted for that stroke and EMPTY for every other, through the workflow's existing input; the five fences on the run and its pull, each accepted and refused — the head by the pull's own answer, never the run's embedded copy, and a run naming no pull refused by name; the read-back's judgment; and every other actions path still refused"); { const REPO = `${TARGET_OWNER}/objectstack`; const RUN = 39713789453; @@ -1241,19 +1243,28 @@ export async function selfTest() { const workflowText = readFileSync(resolve(repoRoot, '.github/workflows/fleet-write.yml'), 'utf8'); const mintsFromOutput = [...workflowText.matchAll(/^\s+permission-actions:\s*\$\{\{\s*steps\.validate\.outputs\.permission_actions\s*\}\}\s*$/gm)].length; t('⭐ ruling 3, no workflow edit: fleet-write.yml mints permission-actions from the validator\'s permission_actions output — the one input this row rides — and names neither this op nor its path', [mintsFromOutput, workflowText.includes('run_rerun_failed'), workflowText.includes('rerun-failed-jobs')], [1, false, false]); - // The four fences on the run — the closed values, then each accepted and refused on the run's own answer. - t('the fences on the run: the closed workflow list is exactly ci.yml, frozen; the event pull_request; the attempt 1; completed with failure; and the read-back accepts queued and in_progress alone', [[...RERUN_FAILED_WORKFLOWS], Object.isFrozen(RERUN_FAILED_WORKFLOWS), RERUN_FAILED_EVENT, RERUN_FAILED_ATTEMPT, RERUN_FAILED_STATUS, RERUN_FAILED_CONCLUSION, [...RERUN_FAILED_RERUN_STATES], RERUN_FAILED_FENCES.map((f) => f.fence)], [['.github/workflows/ci.yml'], true, 'pull_request', 1, 'completed', 'failure', ['queued', 'in_progress'], ['workflow', 'event', 'once', 'failed']]); + // The five fences on the run and its pull — the closed values, then each accepted and refused on the run's and the pull's own answers. + t('the fences on the run: the closed workflow list is exactly ci.yml, frozen; the event pull_request; the attempt 1; completed with failure; and the read-back accepts queued and in_progress alone', [[...RERUN_FAILED_WORKFLOWS], Object.isFrozen(RERUN_FAILED_WORKFLOWS), RERUN_FAILED_EVENT, RERUN_FAILED_ATTEMPT, RERUN_FAILED_STATUS, RERUN_FAILED_CONCLUSION, [...RERUN_FAILED_RERUN_STATES], RERUN_FAILED_FENCES.map((f) => f.fence)], [['.github/workflows/ci.yml'], true, 'pull_request', 1, 'completed', 'failure', ['queued', 'in_progress'], ['workflow', 'event', 'once', 'failed', 'head']]); t('every listed workflow is a file on disk that runs on pull_request — so the list names a run the fences can accept', RERUN_FAILED_WORKFLOWS.map((p) => [p, existsSync(resolve(repoRoot, p)) && /^ {2}pull_request:/m.test(readFileSync(resolve(repoRoot, p), 'utf8'))]), RERUN_FAILED_WORKFLOWS.map((p) => [p, true])); // The measured run as its attempt 1 read before the maintainer re-ran it (run 39713789453; fields as the run endpoint answers them). - const MEASURED = { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure' }; - const fenced = (patch) => rerunFailedFenceFailures({ ...MEASURED, ...patch }).map((f) => f.fence); - t('accepted: the measured run passes all four fences', rerunFailedFenceFailures(MEASURED), []); + // It ran on pull #25155's head; PULL is that pull's OWN answer (GET /pulls/25155), which the head fence reads — never the copy the run embeds. + const HEAD = '25b3a6c4bff905d462d4ff44030e5465c342254a'; + const MOVED = '9f0e1d2c3b4a59687766554433221100ffeeddcc'; + const MEASURED = { id: RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', head_sha: HEAD, pull_requests: [{ number: 25155, head: { sha: HEAD } }] }; + const PULL = { number: 25155, state: 'open', head: { sha: HEAD } }; + // `fenced` takes ONE argument — it is passed to Array.map, whose index would otherwise land in the pull's place. + const fencedWith = (patch, pull) => rerunFailedFenceFailures({ ...MEASURED, ...patch }, pull).map((f) => f.fence); + const fenced = (patch) => fencedWith(patch, PULL); + t('accepted: the measured run passes all five fences — its head_sha is the current head of the pull it names, by that pull\'s own answer', [rerunFailedPullNumber(MEASURED), rerunFailedFenceFailures(MEASURED, PULL)], [25155, []]); + t('⛔ fence head, refused: a stale head (the pull was pushed to since), an answer for another pull, a run with no head_sha, and no pull answer at all — each naming the head fence alone', [fencedWith({}, { ...PULL, head: { sha: MOVED } }), fencedWith({}, { ...PULL, number: 25156 }), fenced({ head_sha: undefined }), fencedWith({}, null)], [['head'], ['head'], ['head'], ['head']]); + t("⛔ fence head, refused: the run's EMBEDDED copy of the pull is never read — it agrees with head_sha, and the pull's own answer that moved on still refuses", [MEASURED.pull_requests[0].head.sha === MEASURED.head_sha, fencedWith({}, { ...PULL, head: { sha: MOVED } })], [true, ['head']]); + t("⛔ fence head, refused by name: a run naming no pull request — pull_requests empty (a fork pull request's run), absent, or an entry with no number — and the sentence says so", [[], undefined, [{ head: { sha: HEAD } }]].map((pull_requests) => [rerunFailedPullNumber({ ...MEASURED, pull_requests }), fencedWith({ pull_requests }, null)]).concat([rerunFailedFenceText(rerunFailedFenceFailures({ ...MEASURED, pull_requests: [] }, null)).includes('fence `head`: the run reads "no pull_requests entry')]), [[null, ['head']], [null, ['head']], [null, ['head']], true]); t('⛔ fence workflow, refused: another workflow, the bare file name, a ref-suffixed path, a missing path — each naming the workflow fence alone', [{ path: '.github/workflows/lint.yml' }, { path: 'ci.yml' }, { path: '.github/workflows/ci.yml@refs/heads/main' }, { path: undefined }].map(fenced), [['workflow'], ['workflow'], ['workflow'], ['workflow']]); t('⛔ fence event, refused: push, merge_group, schedule, release, workflow_dispatch and pull_request_target runs — each naming the event fence alone', ['push', 'merge_group', 'schedule', 'release', 'workflow_dispatch', 'pull_request_target'].map((event) => fenced({ event })), [['event'], ['event'], ['event'], ['event'], ['event'], ['event']]); t('⛔ fence once, refused: attempt 2, attempt 3, the string "1" and a missing attempt — each naming the once fence alone', [{ run_attempt: 2 }, { run_attempt: 3 }, { run_attempt: '1' }, { run_attempt: undefined }].map(fenced), [['once'], ['once'], ['once'], ['once']]); t('⛔ fence failed, refused: still in progress, completed with success, completed cancelled, completed with no conclusion — each naming the failed fence alone', [{ status: 'in_progress', conclusion: null }, { conclusion: 'success' }, { conclusion: 'cancelled' }, { conclusion: null }].map(fenced), [['failed'], ['failed'], ['failed'], ['failed']]); const all = rerunFailedFenceFailures({ path: '.github/workflows/release.yml', event: 'push', run_attempt: 2, status: 'completed', conclusion: 'success' }); - t('every failing fence is named in ONE pass, and the sentence quotes what the run read beside what the relay re-runs', [all.map((f) => f.fence), rerunFailedFenceText(all).includes('fence `once`: the run reads 2'), rerunFailedFenceText(all).includes('"push"'), rerunFailedFenceText(all).split('; ').length], [['workflow', 'event', 'once', 'failed'], true, true, 4]); + t('every failing fence is named in ONE pass, and the sentence quotes what the run read beside what the relay re-runs', [all.map((f) => f.fence), rerunFailedFenceText(all).includes('fence `once`: the run reads 2'), rerunFailedFenceText(all).includes('"push"'), rerunFailedFenceText(all).split('; ').length], [['workflow', 'event', 'once', 'failed', 'head'], true, true, 5]); // The read-back's judgment, on the measured attempts (jobs by name and state, as the attempts endpoint lists them). const attempt1 = [ { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, diff --git a/scripts/pm/rerun-failed.mjs b/scripts/pm/rerun-failed.mjs index 201d4db8fc6..fa5511bfb7f 100644 --- a/scripts/pm/rerun-failed.mjs +++ b/scripts/pm/rerun-failed.mjs @@ -20,17 +20,20 @@ * * ## What it does, in order * - * 1. It reads the run (`GET /repos/{repo}/actions/runs/{id}`) and judges the - * four fences on the run with the predicates the relay's executor uses - * (`RERUN_FAILED_FENCES` in `fleet-write/ops.mjs`): the workflow, the - * event, once, failed. A run that fails one is REFUSED here (exit 2) with - * every failing fence named and nothing sent. It then reads attempt 1's - * job list and names the jobs that failed there, which are what the - * relay's read-back looks for. A list with no failed job is refused the - * same way. ⛔ This check is ADVISORY: an early refusal with its reason. - * The executor judges the four fences again on the run it reads - * immediately before its POST, and that check is the authoritative one; - * a pass here never stands in for it. + * 1. It reads the run (`GET /repos/{repo}/actions/runs/{id}`) and the pull + * it names (`pull_requests[0].number`, by the pull's own + * `GET /repos/{repo}/pulls/{n}`), and judges the five fences on them with + * the predicates the relay's executor uses (`RERUN_FAILED_FENCES` in + * `fleet-write/ops.mjs`): the workflow, the event, once, failed, the head + * (the run's `head_sha` is the pull's CURRENT head, never the copy the + * run embeds; a run naming no pull is refused by name). A run that fails + * one is REFUSED here (exit 2) with every failing fence named and nothing + * sent. It then reads attempt 1's job list and names the jobs that failed + * there, which are what the relay's read-back looks for. A list with no + * failed job is refused the same way. ⛔ This check is ADVISORY: an early + * refusal with its reason. The executor judges the five fences again on + * the run and pull it reads immediately before its POST, and that check + * is the authoritative one; a pass here never stands in for it. * 2. It packs ONE action, `{ op: 'run_rerun_failed', run_id }`, which the * shared validator judges (alone in its stroke, `run_id` alone), and * sends it as ONE `repository_dispatch` (`fleet-write/dispatch.mjs`'s @@ -38,9 +41,9 @@ * 3. After a `success` run it reads the run back at the seat, its attempt * and attempt 2's re-run jobs, and prints their ids. That is a READING * for the seat's note, never a verdict: the executor's own read-back - * landed the action. After a FAILED run it re-reads the run and prints - * what the fences say now, so the seat learns why without the job log it - * cannot read. + * landed the action. After a FAILED run it re-reads the run and its pull + * and prints what the fences say now, so the seat learns why without the + * job log it cannot read. * * ## RELAY-ONLY * @@ -62,8 +65,8 @@ * payload is printed, nothing sent. * 2 usage, a fence the run fails or a run with no failed job (advisory, * nothing sent), or the payload refused by the validator. - * 3 PREREQUISITE NOT MET — no token, the run or its jobs unreadable, no - * session, or a route that is not the relay. Nothing sent. + * 3 PREREQUISITE NOT MET — no token, the run, its pull or its jobs + * unreadable, no session, or a route that is not the relay. Nothing sent. * 5 the dispatch was refused, or the relay's run FAILED: its executor * refused at a fence, or the re-run did not read back. Go READ the run; * ⛔ do not re-send blind (a re-send after a landing meets the once fence). @@ -88,6 +91,7 @@ import { failedJobNames, rerunFailedFenceFailures, rerunFailedFenceText, + rerunFailedPullNumber, } from './fleet-write/ops.mjs'; import { refusalText } from './fleet-write/validate.mjs'; @@ -202,6 +206,24 @@ async function readRun(call, repo, runId) { return { run: r.json }; } +/** + * The fences' two reads, the executor's order: the run, then the pull it names by the pull's OWN GET (never the + * copy the run embeds) — `{ run, pull, failures }`, `pull` null when the run names none (the head fence then + * fails by name), or `{ fail }` for a read that did not answer. Never throws. + */ +async function readFences(call, repo, runId) { + const read = await readRun(call, repo, runId); + if (read.fail) return { fail: read.fail }; + const number = rerunFailedPullNumber(read.run); + let pull = null; + if (number !== null) { + const p = await call(`/repos/${repo}/pulls/${number}`); + if (p.status !== 200 || !isObject(p.json)) return { run: read.run, fail: `${p.call} -> HTTP ${p.status}${p.detail ? ` (${p.detail})` : ''}: pull #${number}, the run's pull request, could not be read, so the head fence was not judged` }; + pull = p.json; + } + return { run: read.run, pull, failures: rerunFailedFenceFailures(read.run, pull) }; +} + // --------------------------------------------------------------------------- // The run // --------------------------------------------------------------------------- @@ -220,13 +242,13 @@ export async function runRerunFailed(options, deps = {}) { const out = { repo, run: runId, exit: null, fences: [], failed: [], transport: null, relay: null, reading: null }; // ── 1. the fences, advisory ──────────────────────────────────────────────── - const read = await readRun(call, repo, runId); + const read = await readFences(call, repo, runId); + if (read.run) log(`rerun-failed: ${runLine(repo, read.run)}`); if (read.fail) { - log(`rerun-failed: PREREQUISITE NOT MET — ${read.fail}. ⛔ NOT MEASURED: no fence was judged and nothing was sent.`); + log(`rerun-failed: PREREQUISITE NOT MET — ${read.fail}. ⛔ NOT MEASURED: ${read.run ? 'the fences were not all judged' : 'no fence was judged'} and nothing was sent.`); return { ...out, exit: EXIT_PREREQUISITE }; } - log(`rerun-failed: ${runLine(repo, read.run)}`); - const failures = rerunFailedFenceFailures(read.run); + const { failures } = read; if (failures.length) { log(`rerun-failed: REFUSED before the dispatch — ${rerunFailedFenceText(failures)}. The relay judges the same fences on the run again and would refuse it too. ⛔ Nothing sent.`); return { ...out, fences: failures, exit: EXIT_USAGE }; @@ -241,7 +263,7 @@ export async function runRerunFailed(options, deps = {}) { log(`rerun-failed: REFUSED before the dispatch — attempt ${RERUN_FAILED_ATTEMPT} lists ${baseline.jobs.length} job(s) and none concluded failure, so the relay's read-back would have nothing to confirm and the relay refuses it. ⛔ Nothing sent.`); return { ...out, exit: EXIT_USAGE }; } - log(`rerun-failed: the four fences hold (advisory — the relay judges them again on the run it reads before its POST); attempt ${RERUN_FAILED_ATTEMPT} failed ${failed.map((n) => `\`${n}\``).join(', ')}.`); + log(`rerun-failed: the five fences hold — the run's head is pull #${read.pull?.number}'s current head (advisory — the relay judges them again on the run and pull it reads before its POST); attempt ${RERUN_FAILED_ATTEMPT} failed ${failed.map((n) => `\`${n}\``).join(', ')}.`); const action = { op: OP, run_id: runId }; // ── 2. the stroke — relay-only ───────────────────────────────────────────── @@ -289,8 +311,8 @@ export async function runRerunFailed(options, deps = {}) { return { ...out, failed, exit: EXIT_UNCONFIRMED }; } if (result.state === 'failure') { - const now = await readRun(call, repo, runId); - const why = now.fail ? `${now.fail} either` : `${runLine(repo, now.run)}; ${rerunFailedFenceFailures(now.run).length ? `the fences now read: ${rerunFailedFenceText(rerunFailedFenceFailures(now.run))}` : 'the four fences still hold, so the relay refused or the re-run did not read back for another reason — read the relay run'}`; + const now = await readFences(call, repo, runId); + const why = now.fail ? `${now.fail} either` : `${runLine(repo, now.run)}; ${now.failures.length ? `the fences now read: ${rerunFailedFenceText(now.failures)}` : 'the five fences still hold, so the relay refused or the re-run did not read back for another reason — read the relay run'}`; log(`rerun-failed: ✗ the relay's run FAILED (${result.detail ?? 'conclusion failure'})${result.run?.url ? ` ${result.run.url}` : ''} — its executor refused at a fence, or the re-run did not read back as attempt ${RERUN_FAILED_ATTEMPT + 1}. Read now: ${why}. ⛔ Do not re-send blind.`); return { ...out, failed, exit: exitForResult(result) }; } @@ -314,12 +336,17 @@ export async function runRerunFailed(options, deps = {}) { const PROBE_RUN = 39713789453; -/** An offline board for the probe and the self-test: a run and its two attempts' job lists, every read recorded. */ -function fakeBoard({ run = {}, attempt1 = null, attempt2 = null, after = null } = {}) { +const PROBE_PULL = 25155; +const PROBE_HEAD = '25b3a6c4bff905d462d4ff44030e5465c342254a'; + +/** An offline board for the probe and the self-test: a run, its pull and its two attempts' job lists, every read recorded. */ +function fakeBoard({ run = {}, pull = {}, attempt1 = null, attempt2 = null, after = null } = {}) { const calls = []; const REPO = RELAY_REPO; - // `run: null` is a run the platform does not answer for; an object patches the measured run. - const base = run === null ? null : { id: PROBE_RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', head_sha: '25b3a6c4bff905d462d4ff44030e5465c342254a', pull_requests: [{ number: 25155 }], ...run }; + // `run: null` is a run the platform does not answer for; an object patches the measured run, which EMBEDS a copy of its pull. + const base = run === null ? null : { id: PROBE_RUN, name: 'CI', path: '.github/workflows/ci.yml', event: 'pull_request', run_attempt: 1, status: 'completed', conclusion: 'failure', head_sha: PROBE_HEAD, pull_requests: [{ number: PROBE_PULL, head: { sha: PROBE_HEAD } }], ...run }; + // `pull: null` is a pull the platform does not answer for; an object patches the pull's OWN answer (its current head). + const pullAnswer = pull === null ? null : { number: PROBE_PULL, state: 'open', head: { sha: PROBE_HEAD }, ...pull }; const jobs1 = attempt1 ?? [ { id: 119611140808, name: 'Test Core (3/6)', run_attempt: 1, status: 'completed', conclusion: 'failure' }, { id: 119611140839, name: 'Test Core (1/6)', run_attempt: 1, status: 'completed', conclusion: 'success' }, @@ -338,6 +365,7 @@ function fakeBoard({ run = {}, attempt1 = null, attempt2 = null, after = null } const json = runReads > 1 && after && base ? { ...base, ...after } : base; return json === null ? { status: 404, json: { message: 'Not Found' }, detail: 'Not Found', call: `GET ${p}` } : { status: 200, json, detail: '', call: `GET ${p}` }; } + if (p === `/repos/${REPO}/pulls/${PROBE_PULL}`) return pullAnswer === null ? { status: 404, json: { message: 'Not Found' }, detail: 'Not Found', call: `GET ${p}` } : { status: 200, json: pullAnswer, detail: '', call: `GET ${p}` }; if (p === `/repos/${REPO}/actions/runs/${PROBE_RUN}/attempts/1/jobs`) return jobs1 === 'unread' ? { status: 403, json: { message: 'Forbidden' }, detail: 'Forbidden', call: `GET ${p}` } : { status: 200, json: { total_count: jobs1.length, jobs: jobs1 }, detail: '', call: `GET ${p}` }; if (p === `/repos/${REPO}/actions/runs/${PROBE_RUN}/attempts/2/jobs`) return { status: 200, json: { total_count: jobs2.length, jobs: jobs2 }, detail: '', call: `GET ${p}` }; return { status: 404, json: { message: 'Not Found' }, detail: 'Not Found', call: `GET ${p}` }; @@ -363,7 +391,8 @@ export async function relayMissProbe({ route, send }) { const SELF_TEST_BATTERIES = Object.freeze({ 'the command line: --run a positive integer, --repo in the one organization, the board repo by default, anything else refused': 6, - 'the fences at the seat, advisory: the measured run passes all four; each failing fence refused by name with nothing sent; a run or job list that cannot be read, or one with no failed job, sends nothing': 8, + // 8 → 11 with the head fence: a stale head, a run naming no pull, a pull that cannot be read. + "the fences at the seat, advisory: the measured run passes all five, its head the pull's current head by the pull's own answer; each failing fence refused by name with nothing sent — a stale head and a run naming no pull among them; a run, pull or job list that cannot be read, or one with no failed job, sends nothing": 11, 'relay-only: a direct route, or a route that errs, refuses with zero writes and no dispatch; the dispatch route packs ONE action, run_id alone; --dry-run prints it and sends nothing': 5, "the outcomes: a success run is read back at the seat naming attempt 2's re-run job ids; a failed run re-reads the run and names what the fences say now, exit 5; no run or none completed is UNCONFIRMED, exit 6; never a second dispatch": 6, 'the wiring: this file issues no write of its own, imports the fences rather than restating them, and declares itself relay-only': 3, @@ -407,6 +436,8 @@ export async function selfTest() { return { ...r, calls: b.calls, sent, text: lines.join('\n') }; }; const RUN_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}`; + const PULL_PATH = `GET /repos/${RELAY_REPO}/pulls/${PROBE_PULL}`; + const MOVED = '9f0e1d2c3b4a59687766554433221100ffeeddcc'; const A1_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}/attempts/1/jobs`; const A2_PATH = `GET /repos/${RELAY_REPO}/actions/runs/${PROBE_RUN}/attempts/2/jobs`; @@ -422,18 +453,22 @@ export async function selfTest() { } // ── the fences at the seat ──────────────────────────────────────────────── - battery('the fences at the seat, advisory: the measured run passes all four; each failing fence refused by name with nothing sent; a run or job list that cannot be read, or one with no failed job, sends nothing'); + battery("the fences at the seat, advisory: the measured run passes all five, its head the pull's current head by the pull's own answer; each failing fence refused by name with nothing sent — a stale head and a run naming no pull among them; a run, pull or job list that cannot be read, or one with no failed job, sends nothing"); { const ok = await drive({}); - t('accepted: the measured run passes the four fences — the run read, then attempt 1\'s jobs — and the line names the failed jobs and the pull request head', [ok.exit, ok.calls.slice(0, 2), ok.text.includes('attempt 1 failed `Test Core (3/6)`, `Test Core`'), ok.text.includes('head 25b3a6c · pull #25155')], [EXIT_OK, [RUN_PATH, A1_PATH], true, true], ok.text); - const fenced = async (run, fence) => { - const r = await drive({}, { board: { run } }); + t('accepted: the measured run passes the five fences — the run read, then its pull by the pull\'s own GET, then attempt 1\'s jobs — and the line names the failed jobs, the pull request head and that it is the pull\'s current head', [ok.exit, ok.calls.slice(0, 3), ok.text.includes('attempt 1 failed `Test Core (3/6)`, `Test Core`'), ok.text.includes('head 25b3a6c · pull #25155'), ok.text.includes("the run's head is pull #25155's current head")], [EXIT_OK, [RUN_PATH, PULL_PATH, A1_PATH], true, true, true], ok.text); + const fenced = async (run, fence, board = {}) => { + const r = await drive({}, { board: { run, ...board } }); return [r.exit, r.sent.length, r.calls.length, r.text.includes(`fence \`${fence}\``) && r.text.includes('Nothing sent')]; }; - t('⛔ fence workflow, refused at the seat: a run of lint.yml — exit 2, no dispatch, no further read', await fenced({ path: '.github/workflows/lint.yml' }, 'workflow'), [EXIT_USAGE, 0, 1, true]); - t('⛔ fence event, refused at the seat: a merge_group run', await fenced({ event: 'merge_group' }, 'event'), [EXIT_USAGE, 0, 1, true]); - t('⛔ fence once, refused at the seat: a run already at attempt 2 — a second failure is real', await fenced({ run_attempt: 2 }, 'once'), [EXIT_USAGE, 0, 1, true]); - t('⛔ fence failed, refused at the seat: a run that succeeded', await fenced({ conclusion: 'success' }, 'failed'), [EXIT_USAGE, 0, 1, true]); + t('⛔ fence workflow, refused at the seat: a run of lint.yml — exit 2, no dispatch, no further read', await fenced({ path: '.github/workflows/lint.yml' }, 'workflow'), [EXIT_USAGE, 0, 2, true]); + t('⛔ fence event, refused at the seat: a merge_group run', await fenced({ event: 'merge_group' }, 'event'), [EXIT_USAGE, 0, 2, true]); + t('⛔ fence once, refused at the seat: a run already at attempt 2 — a second failure is real', await fenced({ run_attempt: 2 }, 'once'), [EXIT_USAGE, 0, 2, true]); + t('⛔ fence failed, refused at the seat: a run that succeeded', await fenced({ conclusion: 'success' }, 'failed'), [EXIT_USAGE, 0, 2, true]); + t('⛔ fence head, refused at the seat: a stale head — the pull was pushed to since the run, by the pull\'s own answer, while the run\'s embedded copy still agrees', await fenced({}, 'head', { pull: { head: { sha: MOVED } } }), [EXIT_USAGE, 0, 2, true]); + t("⛔ fence head, refused at the seat by name: a run naming no pull request (pull_requests empty — a fork pull request's run) — and no pull is read", await fenced({ pull_requests: [] }, 'head'), [EXIT_USAGE, 0, 1, true]); + const pullUnread = await drive({}, { board: { pull: null } }); + t('a pull that cannot be read leaves the head fence unjudged: exit 3, no dispatch, NOT MEASURED', [pullUnread.exit, pullUnread.sent.length, pullUnread.text.includes('the head fence was not judged') && pullUnread.text.includes('NOT MEASURED')], [EXIT_PREREQUISITE, 0, true], pullUnread.text); const unread = await drive({}, { board: { run: null } }); t('a run that cannot be read is NOT MEASURED: exit 3, no dispatch', [unread.exit, unread.sent.length, unread.text.includes('NOT MEASURED')], [EXIT_PREREQUISITE, 0, true]); const noJobs = await drive({}, { board: { attempt1: 'unread' } }); @@ -446,7 +481,7 @@ export async function selfTest() { battery('relay-only: a direct route, or a route that errs, refuses with zero writes and no dispatch; the dispatch route packs ONE action, run_id alone; --dry-run prints it and sends nothing'); { const direct = await drive({}, { route: DIRECT_ROUTE }); - t('⛔ a direct route is a PREREQUISITE refusal: exit 3, no dispatch, the reads only, the line saying RELAY-ONLY and never a direct write', [direct.exit, direct.sent.length, direct.calls, direct.text.includes('RELAY-ONLY') && direct.text.includes('Not falling back')], [EXIT_PREREQUISITE, 0, [RUN_PATH, A1_PATH], true]); + t('⛔ a direct route is a PREREQUISITE refusal: exit 3, no dispatch, the reads only, the line saying RELAY-ONLY and never a direct write', [direct.exit, direct.sent.length, direct.calls, direct.text.includes('RELAY-ONLY') && direct.text.includes('Not falling back')], [EXIT_PREREQUISITE, 0, [RUN_PATH, PULL_PATH, A1_PATH], true]); const erring = await drive({}, { route: { requested: 'dispatch', transport: 'dispatch', reason: 'x', error: 'OS_FLEET_TRANSPORT=dispatch but the relay is not live.', session: SESSION } }); t('⛔ a route that errs is a PREREQUISITE refusal, its error printed, no dispatch', [erring.exit, erring.sent.length, erring.text.includes('the relay is not live')], [EXIT_PREREQUISITE, 0, true]); const sent = await drive({}); @@ -461,11 +496,11 @@ export async function selfTest() { battery("the outcomes: a success run is read back at the seat naming attempt 2's re-run job ids; a failed run re-reads the run and names what the fences say now, exit 5; no run or none completed is UNCONFIRMED, exit 6; never a second dispatch"); { const landed = await drive({}, { board: { after: { run_attempt: 2, status: 'queued', conclusion: null } } }); - t('⭐ success: exit 0 after ONE dispatch, the run and attempt 2\'s jobs read back, the line naming the re-run job id', [landed.exit, landed.sent.length, landed.calls.slice(2), landed.text.includes('attempt 2 · `Test Core (3/6)` job 119642047246 queued'), landed.reading?.jobs], [EXIT_OK, 1, [RUN_PATH, A2_PATH], true, [{ name: 'Test Core (3/6)', id: 119642047246, status: 'queued' }]], landed.text); + t('⭐ success: exit 0 after ONE dispatch, the run and attempt 2\'s jobs read back, the line naming the re-run job id', [landed.exit, landed.sent.length, landed.calls.slice(3), landed.text.includes('attempt 2 · `Test Core (3/6)` job 119642047246 queued'), landed.reading?.jobs], [EXIT_OK, 1, [RUN_PATH, A2_PATH], true, [{ name: 'Test Core (3/6)', id: 119642047246, status: 'queued' }]], landed.text); const unreadAfter = await drive({}, { board: { after: { run_attempt: 1 } } }); t('…a reading that does not see attempt 2 at the seat is SAID, and the exit stays 0 — the executor\'s read-back is the verdict, this is a reading', [unreadAfter.exit, unreadAfter.text.includes('a reading, not a verdict')], [EXIT_OK, true]); const failed = await drive({}, { outcome: 'failure', board: { after: { run_attempt: 2, status: 'in_progress', conclusion: null } } }); - t('⛔ a failed relay run: exit 5, ONE dispatch, the run re-read and the fence that fails NOW named — here a person re-ran it first', [failed.exit, failed.sent.length, failed.text.includes('FAILED') && failed.text.includes('fence `once`: the run reads 2')], [EXIT_PLATFORM_REFUSAL, 1, true], failed.text); + t('⛔ a failed relay run: exit 5, ONE dispatch, the run and its pull re-read and the fence that fails NOW named — here a person re-ran it first', [failed.exit, failed.sent.length, failed.text.includes('FAILED') && failed.text.includes('fence `once`: the run reads 2')], [EXIT_PLATFORM_REFUSAL, 1, true], failed.text); const failedHolding = await drive({}, { outcome: 'failure' }); t('…and when the fences still hold, the line says to read the relay run, never to re-send', [failedHolding.exit, failedHolding.text.includes('read the relay run') && failedHolding.text.includes('Do not re-send blind')], [EXIT_PLATFORM_REFUSAL, true]); const noRun = await drive({}, { outcome: 'no-run' }); @@ -515,7 +550,7 @@ export async function selfTest() { return 1; } console.log( - `✓ rerun-failed self-test: ${cases.length} cases pass across ${declared.length} batteries — the four fences judged at the seat as an advisory refusal ` + + `✓ rerun-failed self-test: ${cases.length} cases pass across ${declared.length} batteries — the five fences, the pull's current head among them, judged at the seat as an advisory refusal ` + 'naming each failing one, nothing sent; relay-only, a direct route refused with zero writes; ONE dispatch of ONE action carrying run_id alone; ' + "a success read back naming attempt 2's job ids, a failure re-read and explained, and no run UNCONFIRMED — never a second dispatch.", ); From 49804f28cf0d70fdd253dcbcc012d7c3ee4f0ac2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 22 Oct 2026 08:23:47 +0000 Subject: [PATCH 6/6] chore(pm): check:pm-fleet-write-dispatch also runs rerun-failed.mjs --self-test The wrapper's self-test reaches CI through the script lint.yml already runs, with no workflow edit. Claude-Session: https://claude.ai/code/session_01FM29ZqLTrnPNXxijioZwMp Co-authored-by: Claude --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index c059066f759..e28703fbf82 100644 --- a/package.json +++ b/package.json @@ -96,7 +96,7 @@ "check:pm-close-cards": "node scripts/pm/close-cards.mjs --self-test", "check:pm-fleet-write-validate": "node scripts/pm/fleet-write/validate.mjs --self-test", "check:pm-fleet-write-execute": "node scripts/pm/fleet-write/execute.mjs --self-test", - "check:pm-fleet-write-dispatch": "node scripts/pm/fleet-write/dispatch.mjs --self-test", + "check:pm-fleet-write-dispatch": "node scripts/pm/fleet-write/dispatch.mjs --self-test && node scripts/pm/rerun-failed.mjs --self-test", "check:pm-board-snapshot": "node scripts/pm/board-snapshot.mjs --self-test", "check:pm-changeset-deadline-census": "node scripts/pm/changeset-deadline-census.mjs --self-test", "check:pm-governed-merges": "node scripts/pm/check-governed-merges.mjs --self-test",