Skip to content

fix(docs): name v0.25.0 in README and gate release-claim drift - #657

Open
toasterbook88 wants to merge 5 commits into
mainfrom
fix/release-claim-freshness
Open

toasterbook88 wants to merge 5 commits into
mainfrom
fix/release-claim-freshness

Conversation

@toasterbook88

@toasterbook88 toasterbook88 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Problem

v0.25.0 was tagged on 2026-11-16, but README.md still pinned v0.24.0 in all three current-version install commands and in the Status table, and SECURITY.md still declared v0.24.x the supported line. The release notes had no v0.25.0 upgrade guide, and v0.24.2 was already published at the time — so the docs were two releases behind.

Every gate was green. hack/verify-doc-facts.sh §5 proves each released tag has a CHANGELOG.md entry, and hack/verify-repo-truth.sh rejects unpublished tags and stale current release is claims — but nothing compared the install pins or the Status table against the tags present in the checkout. An operator following Quick start installed a superseded binary, and the release flow never surfaced it.

Change

Fix the drift

  • README.md: pin the three current-version commands to v0.25.0; point the upgrade note at a v0.25.0 guide and move v0.24.0 into the older-release list; correct the Status table version, tag commit (16eeb3b → 8578cb2a) and supported security line.
  • SECURITY.md: supported line v0.25.x.
  • docs/release-notes.md: add the v0.25.0 upgrade guide, derived from the merged CHANGELOG entry — cloud stream idle guard, consult_model, desk leases failing closed, A2A terminal-state refusal, daemon supervision check, durable axis update.

Close the gap so it cannot recur

  • hack/verify-release-claims.sh — new local, network-free guard. The tags in the checkout are the authority. It checks the three claims that present themselves as install this now: the first AXIS_VERSION=vX.Y.Z in README.md; the Status table Latest release row (version and tag commit); and the supported security line, which README.md and SECURITY.md must agree on.
  • hack/verify-release-claims-tests.sh — fixture-driven regression tests. Wired into verify-doc-facts.sh, which ci.yml, release.yml, and hack/ci-preflight.sh already run.

Two deliberate scope limits:

  • Two-tag window. README.md may name the newest release tag or the one immediately before it. release.yml runs this gate in its test job with fetch-depth: 0, so on a tag push the new tag is in the checkout while the tree still documents the previous release — and the README cannot cite the tag's own commit, which does not exist until the commit is made. One release of lag keeps publishing possible; two do not. The drift this PR fixes is still caught, because v0.24.0 was two behind v0.25.0.
  • Historical material stays free. The rollback example and the "Upgrading from an older release?" links name older tags on purpose, and a prerelease tag (v0.26.0-rc.1) never demands promotion.

Review round

Both automated reviewers flagged the same --short defect on the first head, and Devin flagged a second defect that would have blocked publishing. Both were checked in code and reproduced.

Finding Real? Resolution
git rev-parse --short yields 7 chars in CI and 8 locally, so the commit comparison rejected a correct citation (TestDriver, Devin) Yes — the first CI run failed on it 363f0dc0: the reference is the full 40-character commit, README.md may cite any 7-to-40-character prefix, and a truncated reference now fails with its own message instead of rejecting every citation
The release.yml test job runs this gate on a tag push, so the new tag would be rejected and GoReleaser would never run (Devin) Yes — reproduced by tagging v0.25.1 on the tree: the gate returned 1 d41db1f2: the two-tag window above, with the cited commit validated against the claimed tag
The test suite resolved tags from the repository when the commit override was omitted (TestDriver) Yes d41db1f2: the suite passes an explicit tag/commit map, so it never reads the repo's own tags and cannot fail in a tagless checkout

Reviewers' heads: both reviewed 492856c8. The head has moved twice since.

Verification

  • The guard fails on the pre-fix tree: verify-release-claims: README.md Quick start pins v0.24.0, but the tags say the newest release (v0.25.0) or the one before it (v0.24.2). It passes after the fix.
  • Release-window regression, reproduced in a clone with tags: with v0.25.1 tagged on this tree the gate passes and reports newest tag v0.25.1; with only v0.24.0–v0.25.0 present and the README reverted to v0.24.0, it fails.
  • git rev-parse --short returns 7 characters in that clone (the CI shape) and the gate passes — the earlier CI failure does not recur.
  • hack/ci-preflight.sh — green end to end on 363f0dc0: build, make test, make test-race, make coverage (total 78.8%), verify-public-boundary, verify-deadcode, verify-import-tiers, verify-specs, verify-repo-truth, verify-doc-facts, actionlint, lifecycle-check.

Notes

  • 505 changed lines across 6 files: 413 of them are the two new scripts and their tests. Larger than the ~400 guideline; flagging it rather than hiding it. Splitting the gate from the docs fix would leave main red, since the gate correctly fails on the current tree.
  • No behaviour change outside the docs and the gate itself.
  • Ready for review; not merged.

v0.25.0 was tagged on 2026-11-16 but README.md still pinned v0.24.0 in the
three install commands and in the Status table, and SECURITY.md still
declared v0.24.x the supported line. Every gate passed: verify-doc-facts
proves each tag has a CHANGELOG entry, and verify-repo-truth rejects
unpublished tags and stale "current release is" claims, but nothing
compared the install pins or the Status table against the tags present in
the checkout. An operator following Quick start installed a superseded
binary, and the release flow never surfaced it.

- README: pin the three current-version commands to v0.25.0, point the
  upgrade note at a v0.25.0 guide and push v0.24.0 into the older-release
  list, and correct the Status table version, tag commit and supported
  security line.
- SECURITY.md: supported line v0.25.x.
- docs/release-notes.md: add the v0.25.0 upgrade guide, derived from the
  merged CHANGELOG entry (cloud stream idle guard, consult_model, desk
  leases failing closed, A2A terminal-state refusal, daemon supervision
  check, durable axis update).
- hack/verify-release-claims.sh: new local, network-free guard that the
  first AXIS_VERSION pin, the Latest-release row (version and tag commit),
  and the supported security line all name the newest release tag.
  Historical material (rollback example, older-release links) stays free,
  and a prerelease tag never demands promotion. Annotated tags are
  dereferenced with ^{commit} because README quotes the commit, not the
  tag object.
- hack/verify-release-claims-tests.sh: fixture-driven regression tests for
  the happy path, each stale claim, the prerelease and no-tag cases, and
  the missing-claim shapes. Wired into verify-doc-facts.sh, which CI, the
  release workflow, and ci-preflight already run.

@toasterbook88 toasterbook88 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AXIS Grok gate

Verdict: PASS-WITH-RISK

PR #657 — fix(docs): name v0.25.0 in README and gate release-claim drift
Head: 492856c85bb1fa996e1347f1566d2b9e06652ce2
Files: README.md, SECURITY.md, docs/release-notes.md, hack/verify-doc-facts.sh, hack/verify-release-claims.sh (new), hack/verify-release-claims-tests.sh (new)

Invariant hits: none.

  • Docs + local network-free verification script only. No binary behavior change, no advisory surface, no HITL/confirm path, no cache badge change, no secret/ssh/fleet/spawn/self-authorize surface.
  • New guard uses checkout tags as authority (grep + git tag / rev-parse --short ^{commit}), scope-limited to the three “install this now” claims, prereleases ignored, historical material free. Wired into existing verify-doc-facts.sh path.
  • Aligns with AGENTS.md §2 (releases via tag workflow; never claim version from memory) and public-boundary rules.

CI status: in progress (Test & Build, govulncheck, Analyze). auto-merge skipped. No completed failures at review time.

Recommended operator action: Wait for CI green, then merge if the release-claim gate matches the intended freshness contract. No code changes required on this SHA.

Review event: COMMENT only. No APPROVE.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread hack/verify-release-claims.sh Outdated
# ^{commit} dereferences an annotated tag object to the commit it points at;
# README's "tag commit" quotes the commit (e.g. v0.24.0 -> 16eeb3b), never
# the tag object's own SHA.
latest_commit="$(git rev-parse --short "${latest}^{commit}" 2>/dev/null || true)"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Correct release commit fails validation

With Git's default abbreviation, latest_commit becomes 8578cb2, rejecting the correct README claim 8578cb2a. Consequently, verify-doc-facts.sh fails on this PR without overrides.

Suggested change
latest_commit="$(git rev-parse --short "${latest}^{commit}" 2>/dev/null || true)"
latest_commit="$(git rev-parse "${latest}^{commit}" 2>/dev/null || true)"

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread hack/verify-doc-facts.sh
# and SECURITY.md still declaring v0.24.x supported, and every gate passed.
# The guard is separate so its regression tests can drive it with fixtures.
./hack/verify-release-claims-tests.sh
./hack/verify-release-claims.sh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Release publication blocked by freshness gate

When a new release tag is pushed, verify-release-claims.sh rejects the previous release retained by pre-tag validation. The publication job requires this gate, so publication cannot proceed.

Learn more

The release Test Gate runs this check before publication, and the publication job depends on that gate. Before tagging, the new check requires the README to name the previous release. After tagging the same commit, the check instead requires the new release. Updating the README before tagging fails pre-tag validation; updating it afterward cannot repair the tagged tree. Requiring the new tag's commit inside that same commit also creates a self-reference.

Example: With v0.25.0 available, a v0.25.1 preparation commit passes while README names v0.25.0. Pushing v0.25.1 on that commit makes the Test Gate reject v0.25.0, preventing GoReleaser from publishing.

Recommended fix: Make freshness validation release-phase-aware. During a tag build, validate operator claims against the preceding published release, or move the new-release freshness requirement to a post-publication check and documentation update. Add a regression covering pre-tag validation and tag-triggered publication of the same tree.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread hack/verify-release-claims.sh Resolved
Comment thread hack/verify-release-claims-tests.sh Resolved
@testdriverai

testdriverai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The branch has been rebased onto main (60eba8a0) cleanly. All verification scripts and tests continue to pass:

  • hack/verify-release-claims.sh resolves full 40-character commit SHAs, validates cited prefixes, and enforces the two-release window.
  • hack/verify-release-claims-tests.sh passes synthetic tag-to-commit mappings and runs completely self-contained.
  • hack/verify-doc-facts.sh passes end-to-end.

The PR remains clean and ready to merge.

📋 Walkthrough (6 files)
File Change
README.md Pin workstation install commands and Status table to v0.25.0 at commit 8578cb2, with supported security line v0.25.x.
SECURITY.md Update supported release policy table from v0.24.x to v0.25.x.
docs/release-notes.md Add v0.25.0 upgrade guide detailing stream idle guards, consult_model, desk leases, A2A terminal-state refusal, and daemon supervision.
hack/verify-doc-facts.sh Wire verify-release-claims-tests.sh and verify-release-claims.sh into Section 6 of doc-facts guardrails.
hack/verify-release-claims.sh Verify Quick start install pin, Status table release version/commit, and security lines against latest release tags with a two-tag release window and full-SHA prefix validation.
hack/verify-release-claims-tests.sh Fixture-driven regression tests verifying two-tag window handling, stale claims, tag commits, security line drift, and prerelease handling with synthetic tag and commit files.

Automatic re-reviews are paused on this PR after 3 reviews. Mention @testdriverai to get a fresh review of the latest commits.

Was this review useful? React with 👍 or 👎 to help TestDriver improve.


Want help from the team that built TestDriver? Explore TestDriver for Business

AXIS Contributor added 2 commits November 17, 2026 18:02
…gate

The first CI run of the new gate failed while the same check passed
locally: `git rev-parse --short` returns seven characters in a small CI
clone and eight in a developer's repository, so comparing README's
eight-character `tag commit` against a short reference rejected a correct
citation. The reference is now the full 40-character commit, README may
cite any 7-to-40-character prefix of it, and a reference that is not a
full SHA fails with its own message instead of rejecting every citation.

Verified in a shallow clone where `git rev-parse --short` reports seven
characters (the CI shape): the gate passes and reports the deterministic
eight-character form. Regression tests cover an eight-character citation
against a full reference, the seven-character form, a citation shorter
than seven characters, and a truncated reference.
`.github/workflows/release.yml` runs verify-doc-facts.sh in its `test` job
with `fetch-depth: 0`, so on a tag push the new tag is already in the
checkout while the tree still documents the previous release. Requiring
README to name the newest tag failed that job on every release, and the
README cannot cite the tag's own commit, which does not exist until the
commit is made. Verified by tagging v0.25.1 on the current tree: the gate
returned 1, which would have blocked GoReleaser.

README may now name the newest release tag or the one immediately before
it, and the cited `tag commit` is checked against that claimed tag rather
than against the newest one. One release of lag is allowed because the
tag push and the follow-up docs change cannot be the same commit; two are
not. The drift this PR fixes still fails: when v0.25.0 was tagged with
v0.24.2 already published, README named v0.24.0, which is two behind.

The supported security line now keys off the claimed release instead of
the newest tag, which is what keeps the window open across a minor bump.
The redundant freshness test on the Status row is gone: the row must
equal the Quick start pin, which the first check already compared.

Regression tests pass an explicit tag/commit map, so the suite no longer
reads the repository's own tags and cannot fail in a tagless checkout.
@testdriverai

testdriverai Bot commented Nov 17, 2026

Copy link
Copy Markdown

⏸️ Skipped this review: 3/3 PR reviews used this hour for this organization. Push a new commit after Tue, 17 Nov 2026 23:50:02 GMT and I'll review it.

@toasterbook88

Copy link
Copy Markdown
Owner Author

Both findings from the first review round are fixed. Recording the resolution so the threads are answered:

1. --short / prefix asymmetry (major, TestDriver + Devin). Real — the first CI run failed on it. The reference is now the full 40-character commit (git rev-parse <tag>^{commit}, never --short, whose length depends on how many objects the checkout holds). README may cite any 7-to-40-character prefix of it, and a truncated reference now fails with its own message instead of rejecting every correct citation. Fixed in 363f0dc0.

2. run_guard falling back to local git tags (minor, TestDriver). Real in principle, and fixed by construction in d41db1f2: the override variable is gone, and every case passes an explicit tag→commit map, so the suite never reads the repository's own tags. In a tagless checkout it now cannot fail for that reason.

3. The release path (Devin, verify-doc-facts.sh:316). Real and the most serious of the three: release.yml runs this gate in its test job with fetch-depth: 0, so a tag push has the new tag in the checkout while the tree still documents the previous release — and the README cannot cite the tag's own commit, which does not exist until the commit is made. That would have failed the gate and blocked GoReleaser on every release.

Reproduced by tagging v0.25.1 on this tree: the gate returned 1. Fixed in d41db1f2 — README may name the newest release tag or the one immediately before it, and the cited tag commit is validated against the claimed tag rather than the newest one. One release of lag keeps publishing possible; two do not. The drift this PR fixes is still caught, because v0.24.0 was two behind v0.25.0.

Re-verified after the fix, in a clone whose git rev-parse --short reports seven characters (the CI shape):

  • with v0.25.1 tagged on the tree → passes, reports newest tag v0.25.1
  • with the README reverted to v0.24.0 and v0.24.2/v0.25.0 present → fails
  • hack/ci-preflight.sh green on d41db1f2; CI green on d41db1f2

Both reviews above are anchored to 492856c8; the head has moved twice since.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants