Repository navigation
feat(capability): evidence-backed capability record - #659
Conversation
…d demotion Add internal/capability and the axis capability command (list, show, check, record, ceiling, verify, export). A capability earns a level (unproven, advisory, supervised, trusted) one step at a time, up to an operator-set ceiling that starts at advisory. - Only passes from Axis own checkers count. axis capability check runs the probe itself; the first checker, facts.hardware, compares the AXIS fact collector with a syscall reading of the same machine. Passes reported with axis capability record count as inconclusive. - Every failure counts, whoever reports it; only a failure with grounded evidence (a named live probe or a hashed file) is critical. - Only the axis actor holds a record until actor identity is verified. - Policy floors: promote_after >= 10, min_rate >= 0.9, window >= promote_after. Each journal entry records the thresholds used. - Every observation and ceiling change is a SHA-256-chained line in journal.jsonl, synced before state.json is written. An interrupted write is counted on the next write without reusing a sequence number. axis capability verify replays the journal against the state. - Level changes are emitted as capability.promoted, capability.demoted and capability.ceiling.changed events. Nothing reads a level to allow or refuse an action. The package docs state the limit: anything running as the same user can rewrite journal and state together, so the journal head must be anchored, for example in backups, before any level grants authority. Reviewed in two rounds by other agents (correctness, independent re-run, adversarial); their findings are fixed and covered by tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
axis capability check gains --node, --all-nodes and --except. In node mode a check compares the node entry in the published AXIS snapshot with a read-only shell probe run on that machine through the AXIS SSH transport (a local shell for this machine). Machines AXIS reports unreachable, or that the probe cannot reach, are inconclusive. facts.gpu compares the NVIDIA GPUs AXIS lists with the GPUs the NVIDIA driver shows under /proc/driver/nvidia, and fails any GPU entry whose model is an error message. It is inconclusive where no NVIDIA driver is loaded. Run against a live cluster it fails on a node whose nvidia-smi cannot reach its driver: parseNvidiaSMIOutput in internal/facts/local_gpu.go takes the one-column error line as a GPU name tagged cuda. That parser is left as it is here so the record shows the failure. Drop four internal/transport LocalExecutor methods from the dead-code allowlist; the probe now reaches them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d to end A critical failure recorded with a hashed file demotes to unproven; the same failure without --evidence-sha256 counts as an ordinary one. record, ceiling, show and verify --format json|yaml are parsed. Closes the two command-level gaps from the second independent re-run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by an adversarial pass against 347a7ad5 and aecc7c21, each reproduced on a live cluster in a throwaway home before the fix: 1. Burst: 20 local checks in 83 s reached advisory. A promotion now also needs the streak to cover min_span (policy, default 6h, floor 1h). 2. Selective checking: after facts.gpu failed on one machine, 47 checks of a healthy machine alone reached advisory. A failure from an Axis checker now opens its subject on the record, and no promotion happens while any subject is open; only a later checker pass on that same subject closes it. Inconclusive checks and outside failures do not. 3. --all-nodes --except <failing node> had the same effect; closed by 2. 4. --node ALPHA was recorded as node:ALPHA, not node:alpha. Node subjects now use the snapshot spelling, lowercased, and the probe dials the snapshot name. 5. An outside failure with a made-up file hash counted as critical. Only Axis checkers can fail critically now; an outside critical failure is one level. Lowering the ceiling is the way to drop further. The store also stamps every observation time itself, so a caller cannot backdate entries to fake the span. Reports carry earliest_promotion and open_failures; list and show explain both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no NVIDIA GPU Agreement on absence is a checked claim that placement relies on, and it is the only way a machine that once listed a phantom GPU can close its open failure: an inconclusive result never closes one. Test fixtures now use neutral GPU names instead of real product models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Written after the code, out of order under AGENTS.md section 8. It has no spec review yet, so verify-specs fails on rule 3 until a fresh-context review is recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… subjects like --node From the fresh-context spec review of PR 659 (approve-with-changes): F1 (major): an open failure closed only through a checker pass on the same subject, so a machine that was retired, renamed or never checked again blocked promotion for good, and only a journal edit (which breaks verify) could clear it. axis capability resolve <capability> <subject> --reason journals an EntryResolve that closes one open subject, keeps the reason, emits capability.failure.resolved and never raises a level. F2 (minor): a check without --node named its subject after the OS hostname while --node checks use the snapshot name, so on a machine whose names differ the two never met. Local checks now take the snapshot name of the node whose recorded hostname is this machine, falling back to the hostname when there is no snapshot or no match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fresh-context spec review, approve-with-changes; both findings fixed in 74c41518. Design now names axis capability resolve and how a local check names this machine. verify-specs passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebase onto the maintainer merge of main rewrote 74c41518 as e55568c; the spec now cites the commit that is on the branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- --node <this machine> dialed itself over SSH when the OS hostname matched neither the config name nor the config hostname. The probe now also uses a local shell when the snapshot recorded this machine's hostname for the node (probeRunnerFor). - resolve matched subjects exactly, so node:ALPHA did not close the open node:alpha. Node subjects are written in their canonical form. - No test asserted that resolve emits capability.failure.resolved; TestCapabilityResolveCommand now reads it from the event log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four minor findings, fixed in 452535d and in this file. The review is by the author and is labelled so; whether it counts toward section 8 is left open for the maintainer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…readable GPU list is inconclusive From the spec review of PR 659 by hermes (revise): - F4: transitions.jsonl is appended after the state is saved, so a failed append lost that level change from the log for good, silently. verify now replays the journal's transitions and compares them with the log, and verify --repair-transitions rebuilds the log from the journal without touching the journal or the state. TestVerifyFindsAndRepairsALostTransition injects the failed write. - F5: the spec named a concurrent-writers test that did not exist. TestConcurrentWritersNeverLoseOrRepeatAnEntry runs four stores on one directory, 25 writes each, and checks 100 unique sequence numbers and a clean verify; it passes under -race. - Q1: a driver GPU entry this user cannot read made facts.gpu fail. The probe now reports it and the check is inconclusive. - Q2: open failures outlive ceiling changes by design; TestCeilingChanges now pins that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verdict revise; findings 1 to 3 were already fixed, 4 and 5 and both questions are fixed in the previous commit. verify-specs needs the re-review that a revise verdict asks for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each finding and question rerun at 6ace141 in a fresh clone, with two mutations showing the new tests fail when the fixes are removed. By the author at the maintainer direction, and labelled so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Devin Review found 6 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
toasterbook88
left a comment
There was a problem hiding this comment.
AXIS Grok gate
Verdict: PASS-WITH-RISK
PR: #659 feat(capability): evidence-backed capability record (draft: AGENTS.md §8 spec review pending)
Head SHA: 926fc8e
Files touched: 24 (+4512/-7). New internal/capability package + cmd/axis/capability.go (list/show/check/record/ceiling/resolve/verify/export), spec docs/superpowers/specs/2026-11-17-capability-record-design.md, AGENTS.md CLI count + layer note, tests, events.
Invariant hits: none hard. Record is explicitly advisory (no surface reads a level to allow/refuse an action; ceiling default advisory). Checkers (facts.hardware, facts.gpu) compare against published snapshot or local collector over existing SSH transport; no secret_read, ssh_bypass, fleet_exec, spawn_subagent, self-authorize, or unattended production config write. Local journal/state under ~/.axis/capability/ only. HITL/confirm paths untouched. Cache reads for --node use published snapshot (source/age inherited from existing plane).
CI status: Test & Build success, govulncheck success, CodeQL success, Analyze success. auto-merge skipped. mergeable_state: blocked.
Residuals / risks:
- AGENTS.md §8: code written before spec (order accepted by maintainer 2026-11-18). Spec reviews recorded (fresh-context approve-with-changes + Hermes revise + author re-review). Independent diff review requested multiple times, none started; only author-labeled diff review recorded. PR body notes this and asks maintainer to rule; independent diff not waived.
- Same-user rewrite of journal+state is an explicit non-goal (verify detects cheaper tampering only). Future wiring of levels into autonomy/A2A would need its own spec and must not treat this record as authority.
- Large feature (>>150 non-test .go lines across cmd/axis + internal/capability).
Recommended operator action: Do not merge until an independent fresh-context diff review (per §8, two for a change this size) is completed and recorded, or the maintainer explicitly waives it in writing on the PR. Confirm no other surface consumes capability levels. Advisory only; no APPROVE.
- facts.gpu passed when the counts matched but the models did not. It now compares model names, ignoring order, case and spacing; on two NVIDIA machines the driver and the collector spell them identically. - The node probe counted online CPUs with getconf while the collector counts the CPUs it may use with nproc, so an affinity mask could fail a healthy node. The probe now uses nproc, falling back to getconf. - A reading the probe did not take (df or getconf failing) compared as zero and failed the node. It is now inconclusive; a real mismatch among the readings that were taken still fails. - verify compared five fields of each transition, so an edited reason, time or threshold passed. It now compares whole transitions. - min_span with seconds was truncated to whole minutes, so promotion could come early. The policy file now refuses a span that is not whole minutes, and a span set in code rounds up. - An unknown --format printed a table (or JSON from export) without an error. Every capability command now refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Model names for facts.gpu, nproc and unmeasured readings for facts.hardware, whole-minute min_span, whole-transition verify, and the --format values every command accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
⏸️ Skipped this review: 3/3 PR reviews used this hour for this organization. Push a new commit after Wed, 18 Nov 2026 02:34:31 GMT and I'll review it. |
…R 659 From the fresh-context review (OpenAI GPT, run by the Codex agent) of b7afa93, verdict revise: - F1: when the local probe stopped early (statfs failing, say), the check threw away the readings it had already taken, so a measured CPU mismatch was inconclusive instead of a failure. The readings taken are now compared: a mismatch among them fails, otherwise the result is inconclusive and names the probe error. Two new TestFactsHardwareCheckerVerdicts cases; both fail on the old code. - F2: check --format json printed the observation before the store stamped it, so its time, actor, origin and counted were empty. Record and RecordCheck now also return the observation as committed, and the check command prints that. TestCapabilityCheckJSONMatchesTheJournal compares it with the journal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verdict revise, two findings, both fixed in the previous commit. verify-specs asks for the re-review that a revise verdict needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The pull request introduces the The implementation is clean, rigorously guarded by policy floors and file locks, cleanly handles crash catch-up recovery, and includes thorough unit and concurrency test coverage. The latest commit records the diff re-review in the design specification in accordance with the AGENTS.md §8 process. Overall verdict: clean and ready to merge. 📋 Walkthrough (24 files)
Was this review useful? React with 👍 or 👎 to help TestDriver improve. Want help from the team that built TestDriver? Explore TestDriver for Business |
…ver a pass From fresh-context diff review B (OpenAI GPT, run by the Codex agent) of b7afa93, finding 1 (major): with /proc/driver/nvidia/gpus present but not listable (mode 0100), the glob matched nothing, the probe reported the driver with zero GPUs, and facts.gpu earned a counted pass without seeing the GPU list. The probe now marks the list unreadable, so the check is inconclusive, when the directory cannot be read and listed, when a GPU's directory has no readable information file, or when that file names no model. TestTheNodeProbeNeverPassesAGPUListItCannotRead runs the real probe script against a stand-in directory for each case (the unlistable case is the reviewer's repro); all three fail on the old script. Review B's other two findings are the ones reviewer A also found, fixed in d3b57c8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Approve at 6ace141, no findings. The diff gate still waits for reviewer B's re-review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewers A and B approve the cumulative diff at b7b1933 with no findings. Earlier revise verdicts are kept above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Summary Prepare v0.26.0: version constant, CHANGELOG entry, the summary goldens that embed the version, and the regenerated current-state facts block. No code changes. v0.26.0 rather than v0.25.1: since v0.25.0, main gained features (`axis capability` #659, deferred MCP tool loading #649, the delegation brief before spawn #648), and this repo bumps the minor version for those. - `internal/buildinfo/version.go`: `0.26.0`. - `CHANGELOG.md`: `## v0.26.0 (2026-11-18)` covering the ten PRs merged since v0.25.0 (#648, #649, #654, #655, #656, #657, #658, #659, #662, #663). - Goldens: `summary_empty_state`, `summary_corrupt_state`, `summary_live_snapshot` and `summary_reachability` (the last embeds the build version on a line that also carries `Cache Age`). - `docs/current-state.md`: regenerated by `hack/refresh-current-state.sh` under a throwaway `HOME` (repo version, coverage 79.4%). README, `SECURITY.md` and the release notes stay on v0.25.0 until the tag exists: the release-claim gate allows the newest tag or the one before it, so the tag push passes, and a follow-up PR points them at v0.26.0 and its tag commit, as #657 did for v0.25.0. Checks: `make test`, `make test-install` (41 assertions), `gofmt -l`, `go vet`, `hack/verify-repo-truth.sh` and `hack/verify-doc-facts.sh` all pass. After merge: rerun both gates on the merge commit, run the Release dry run on main, then tag that commit. ## Spec Spec: exempt (release preparation, no behaviour change). ## Review findings Declines and open questions: - none open. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/toasterbook88/axis/pull/664" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: AXIS Contributor <axis-dev@example.invalid> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Adds
internal/capabilityandaxis capability(list,show,check,record,ceiling,verify,export): an evidence-backed record of how far each capability has earned trust, withlevels
unproven,advisory,supervisedandtrusted. Nothing reads a level to allow or refuse anaction; the record is advisory.
axis capability checkruns the probe itself; checkersare
facts.hardwareandfacts.gpu, on this machine or against the published snapshot with--node/--all-nodesover the existing SSH transport. Passes reported withrecordcount asinconclusive.
keeps its machine open, and no promotion happens while any machine is open.
(default 6h, floor 1h), and stops at an operator ceiling (default
advisory).~/.axis/capability/journal.jsonl,synced before
state.json; an interrupted write is counted on the next write;verifyreplays thejournal against the state.
Spec
Spec:
docs/superpowers/specs/2026-11-17-capability-record-design.mdAGENTS.md §8 was not followed in order: the code was written before the spec. The fresh-context
spec review has since been done and recorded in the spec (approve-with-changes, both findings fixed in
e55568c), and
verify-specspasses. This PR stays a draft until the two diff reviews a change this sizeneeds are in. Asking the
maintainer to rule: waive the order, or require the spec review and the two diff reviews (this is a
larger feature) before review proceeds. Fresh-context reviews with
docs/superpowers/review-prompt.mdhave been requested.
Review findings
Reviews so far were by other agents given a written ask, not the review-prompt.md protocol, so none is
recorded in the spec's
## Reviewsection. Their findings and resolutions:Declines and open questions:
crash could leave a counted pass with no log line. fixed in be4954e (journal synced before state).
(catch-up from the journal before assigning the next number).
Policies.Forreturned unresolved entries; zero thresholds would promote onevery pass. fixed in be4954e.
several flags were untested. fixed in be4954e and fb73ea9.
policy made a level cheap;
--actorwrote any actor's record. fixed in be4954e (checker-onlypasses, every failure counts, policy floors, no
--actor).be in normal form).
ceilingcall dropped caught-up entries. fixed in be4954e.--evidence-sha256and structured output were untested end to end.fixed in fb73ea9.
past a standing failure;
--exceptthe failing machine did the same; subject spelling splitrecords; a made-up hash made an outside failure critical. fixed in 007f52f (time span, open
failures per machine, canonical subjects, checker-only critical, store-stamped times).
facts.gpufailure, since aninconclusive check never closes one. fixed in dae3609 (agreement on "no NVIDIA GPU" is a pass).
again blocked promotion for good, with no operator path. fixed in e55568c (
axis capability resolve, journaled with its reason).--nodechecks by snapshotname. fixed in e55568c (local checks use the snapshot name of this machine).
--nodenaming this machine dialed it over SSH when its OS hostnamematched neither config field. fixed in 452535d (local shell when the snapshot recorded this
machine's hostname for the node).
resolvematched subjects exactly, sonode:ALPHAdid not closenode:alpha. fixed in 452535d.capability.failure.resolvedevent. fixed in 452535d.--exceptwas missing from the spec. fixed in e6f8e23.maintainer on this PR, cited in the spec Status line.
7436eca (verify compares it with the journal replay; verify --repair-transitions rebuilds it).
(four concurrent stores, passes under -race).
outliving ceiling changes is intended and now tested.
the new tests fail without the fixes: recorded in the spec, verdict approve. verify-specs passes.
Testing
./hack/hermetic-go-test.sh ./... -count=1 -timeout 180s: pass.make lint,verify-doc-facts.sh,verify-repo-truth.sh,verify-deadcode.sh,verify-import-tiers.go,verify-public-boundary.sh,verify-specs.go: pass.AXIS_HOMEdirectories against a ten-node cluster:facts.hardwarepassed onevery reachable node;
facts.gpufailed on one node whose snapshot lists annvidia-smifailuresentence as a GPU (fixed separately in the
fix(facts)PR); each attack in item 9 was reproducedbefore its fix and replayed after it.
🤖 Generated with Claude Code