feat(ship-check): tool-definition reviewer agent, skill, and surface-diff script

[aliasunder]

What this adds

A report-only reviewer for MCP tool definitions, dispatched on demand. It reads the tool list a server sends (tools/list, saved to a file) the way a client receives it, and compares it with the list from before a change.

• plugins/ship-check/agents/tool-definition-reviewer.md: a sixth ship-check agent. Tools are Read, Grep, Glob, and Bash (Bash only to run the script). It never edits, commits, or posts.
• plugins/ship-check/skills/tool-definition-review/SKILL.md: the rubric and the checks.
  • A cold read marks each in-scope tool on six rubric dimensions.
  • A diff read reports text changed in tools outside the stated intent, facts the change dropped, description text that repeats a schema description, and failures the handler can return that the description does not list (and listed failures nothing produces).
  • The error check writes one line for each failure it traced, marked listed or MISSING, so the comparison with the description is visible in the report.
  • One dispatch does both reads on a small change. A Pass: cold or Pass: diff line and a Review only: list split a larger one, and one dispatch takes at most eight tools through the diff read.
  • A report is partial, never complete, while any tool is unreviewed or untraced, and it names those tools. If the script cannot run, the report is failed; the reviewer does not compare the files by hand.
• scripts/surface-diff.ts in that skill, run with Bun: the comparisons that have one right answer, printed as JSON for the agent to judge.
  • Changed, added, removed, and unchanged tools, with key-order-only changes kept apart.
  • A line or schema sentence added to (or removed from) two or more tools, grouped as one shared edit.
  • Description text that repeats a parameter's schema description (40 characters or more), labelled when the base already had it.
  • Sizes, and whether a snapshot's server instructions or prompts changed.
  • --variants lists the tools another configuration's file words differently; --show prints a tool's definition as readable text, because a surface file keeps each description on one long JSON line.
  • It rejects input it cannot trust: an unrecognised shape, a tool without a name or input schema, duplicate names, a tool name that could carry shell syntax (the reviewer passes names on a command line), one page of a paginated list, or flags that do not combine.

What this does not change

The ship-check pipeline does not dispatch the new agent, and pr-review's tool-definition dimension is unchanged. Whether the reviewer becomes part of the pipeline is a separate decision that waits on a comparison against the existing reviewer on a real change.

Supporting changes

• .github/workflows/test.yml: runs bun test plugins on pushes to main and on pull requests.
• Release workflows: the plugin and skill archives leave __tests__ out, every action is pinned to a commit, and the job that only validates versions keeps no token in its checkout.
• Docs: AGENTS.md (structure tree and a convention for bundled scripts), the root README, the plugin README, and the two manifest descriptions now list six agents and eight skills in ship-check. The root README's install and adapting steps name the Bun prerequisite and both vault-cortex tool prefixes.

How it was checked

• bun test plugins passes; the script tests check runs it on every push.
• Two mutation checks: removing the key sort fails the two key-order tests, and raising the overlap threshold by one fails the boundary test.
• Run on two real snapshots of a 33-tool server, the script reports 29 changed and 4 unchanged tools and groups one sentence added to 14 tools as a single shared edit.
• The agent was dispatched against a tool list with five planted changes (an unintended edit, a deleted fact, a moved fact, a removed error entry, an invented error entry) and reported all five correctly, and against another server's tool list with no base.
• A first run on a 29-tool change skipped the error-entry check for most tools and still reported complete. The eight-tool limit, the status rule, and the tracing steps in the skill come from that run. Re-run in batches of eight, the same change produced the three error entries known to be missing.
• A reviewer whose shell refused the script compared the files by hand and left 20 of 33 tools uncompared, which is why a script that cannot run now ends the review as failed.
• A later run with the script working stopped at eight tools and reported partial as the skill requires, but for one tool it traced six failures and reported none missing while two had no entry. The one-line-per-failure format comes from that run.
• The release zip exclude pattern was run locally and leaves no __tests__ entry in the archive.

The script has no dependencies and is not type-checked in CI; the tests run it under Bun.

🤖 Generated with Claude Code

[umm-actually]

Pin create-github-app-token to a commit SHA in release workflows
Low severity · ci · high confidence

.github/workflows/auto_release.yml:72 — beyond the diff's line ranges, in code the changes touch or depend on.

auto_release.yml and manual_release.yml use actions/create-github-app-token@v3, a mutable tag for a third-party action that issues release credentials, while the new test.yml pins every action to a full commit SHA with a version comment. The repo's own pr-review skill requires third-party actions to be SHA-pinned, and the release pipeline is the most privileged workflow.

Failure scenario: The action maintainer force-pushes a new v3 tag (or the action repo is compromised); the next tag push runs attacker-controlled code in the release job, which holds contents: write and an app token with write permission.

Suggested fix

Pin to the full commit SHA with a version comment as test.yml does — uses: actions/create-github-app-token@<full-sha> # v3.x.x — and pin the remaining floating actions/checkout@v7 lines in both release workflows for consistency.

---

umm-actually · deepseek/deepseek-v4-flash-0731

[umm-actually]

umm-actually re-reviewed at 3a74463

No new findings (11 tracked finding(s) across all runs).

---

umm-actually · minimax/minimax-m3

[aliasunder]

On the unpinned actions in the release workflows: fixed in cd348fa. auto_release.yml and manual_release.yml now pin actions/checkout to 3d3c42e (v7.0.1) and actions/create-github-app-token to bcd2ba4 (v3.2.0), the same commits umm_review.yml and test.yml use. That was five floating references across the two files.

---

Generated by Claude Code