diff --git a/docs/plans/fix-ci-lint-evals-prettier/investigation.md b/docs/plans/fix-ci-lint-evals-prettier/investigation.md new file mode 100644 index 00000000..e435a5ec --- /dev/null +++ b/docs/plans/fix-ci-lint-evals-prettier/investigation.md @@ -0,0 +1,240 @@ +# Investigation: PR #219 lint job fails on unformatted `evals/` files + +Investigation report. Read the Summary, then approve the Planned Fix or push back. + +## Summary + +- **Root Cause:** PR #219 inherits a lint failure from `main`: the Skillwalker evals (Skillwalker is the external tool that runs them) merged in #218 were committed without running Prettier, so the Prettier hook rewrites 85 files under `evals/` and fails the job (E1, E3, E4). +- **Fix:** Run the pinned Prettier over `evals/` and commit the result on `main`, so `main` goes green and PR #219 picks up the fix when it updates from `main`. +- **Why Correct:** In a clean checkout with CI's pinned tools, formatting `evals/` makes all 13 lint hooks pass (E7). +- **Validation Outcome:** Validation confirmed the root cause and fix, showed the reformat is whitespace-only, and refuted the draft's reason for rejecting a `.prettierignore` entry, which has been corrected (V3, V4). +- **Remaining Risks:** Confidence is high; the one unverified assumption is that Skillwalker, the external eval runner, does not read `evals/` files byte for byte. + +## Problem Statement + +The `lint` job on PR #219 fails, and the PR's own changes are not the cause. + +- **Symptom:** `npm run lint` exits 1. The Prettier hook reports `Failed` with "files were modified by this hook". Every other hook passes, and the `test` job in the same run passes (E1). +- **Expected:** The lint job passes, since every file PR #219 changes is already formatted (E2). +- **Conditions:** Any CI run on a commit that includes #218 (`abba73a`) fails the same way. That includes `main` itself (E4) and every branch cut from it since, including PR #219 (E3). +- **Impact:** `main` has been red since #218 merged on 2026-09-22, and every open PR based on it will show a failing lint check until the fix lands. + +## Root Cause Analysis + +### Root Cause + +The `evals/` tree added in #218 was never run through Prettier, and nothing in the lint setup exempts it, so the Prettier hook reformats 85 of its files on every CI run and fails. + +### Detailed Analysis + +The lint job runs Prettier in write mode over every Markdown, JSON, YAML, and JavaScript file in the repo. A hook that changes a file counts as a failure (E5). `.prettierignore` exempts old plans, research, and vendored assets, but not `evals/` (E5). + +Commit `abba73a` (#218) added 247 tracked files under `evals/` (E3). Of those, 85 are not in the shape Prettier wants (E6): + +- Prompt files end without a final newline. +- Rubric files are missing blank lines Prettier inserts, such as between a heading and the list below it. +- `tests.json` files hold one-line objects wider than the 120-column print width, which Prettier breaks onto multiple lines. +- Two scaffold plan files hold Markdown tables that Prettier pads into aligned columns. + +None of these changes alter content. They are whitespace and layout only. + +`main`'s own CI run on `abba73a` failed with the same Prettier error (E4). PR #219 branched from `abba73a` (E3), so it carries the same 85 files. The failing file list in the PR's log names only `evals/` paths and none of the 25 files the PR changes (E1, E2). + +## Planned Fix + +### Approach + +Format `evals/` with the repo's pinned Prettier and commit the result on `main`, then update PR #219 from `main`. + +### Changes + +#### `evals/**` (85 files) + +- **Change:** Reformat with Prettier. Whitespace and layout only (E6). +- **Evidence:** (E1), (E6), (E7) +- **Standards:** `.prettierrc.json` and the Prettier hook in `.pre-commit-config.yaml`. +- **Details:** Run these steps from a checkout of `main`: + 1. Install the pinned tools so Prettier is 3.9.6, the version CI uses. + 2. Format the evals tree. + 3. Confirm the full lint passes. + 4. Commit and push, or open a small PR to `main`. + 5. Merge or rebase `main` into `opus-5-5-plugin-builder-guidance` to re-run PR #219's checks. + + ```sh + npm ci + npx prettier --write evals + npm run lint + git add evals + git commit -m "chore(lint): format the Skillwalker evals with Prettier" + ``` + +If the fix needs to land on PR #219 first, commit the same change on the PR branch instead. The cost is that the PR's diff grows by 85 unrelated files. + +### Alternative considered: ignore `evals/` instead of formatting it + +Adding `evals/` to `.prettierignore` also turns the job green, with no other change. The file-hygiene hooks already skip Markdown and JSON everywhere, so nothing else would touch the missing final newlines (V4). + +Formatting is still the better fix. The `.prettierignore` header says Markdown in this repo "is the product," and every entry it exempts is a static archive, a vendored asset, or a file Prettier cannot parse. The eval prompts and rubrics are none of those. They are live content an agent reads, so they belong under the same formatting standard as the skills they test. + +Choose the ignore only if you learn that Skillwalker reads `evals/` files byte for byte (see Remaining Risks). + +``` +# .pre-commit-config.yaml:41-48, on all four hygiene hooks +exclude: '\.(md|markdown|json|ya?ml|jsx?|mjs|cjs)$' +``` + +### Tell reviewers the fix commit is whitespace-only + +The fix commit shows 433 insertions and 265 deletions across 85 files. The two scaffold files under `evals/iterative-plan-review/scaffolds/ruby-security-app/plans/` realign every row of a 49-row table, which can look like content loss in a collapsed or filtered diff view (V3). + +Say in the commit or PR description that the change is Prettier formatting only, and suggest `git diff -w` to review it. + +## Evidence Summary + +### E1: The PR's lint job fails only in the Prettier hook, on `evals/` files + +- **Source:** CI run 35880222315, job 107246445126 (PR #219, `lint`) +- **Finding:** + ``` + prettier.................................................................Failed + - hook id: prettier + - files were modified by this hook + ... + shellcheck...............................................................Passed + (the other 11 hooks)......................................................Passed + ##[error]Process completed with exit code 1. + ``` + Prettier's per-file output lists 85 files without `(unchanged)`. Every one is under `evals/`. The `test` job in the same run passed. +- **Relevance:** Pins the failure to Prettier rewriting `evals/`. + +### E2: PR #219 touches no file under `evals/` + +- **Source:** `gh pr view 219 --json files` +- **Finding:** 25 changed files, all under `docs/plans/opus-5-5-plugin-builder-guidance/` and `han-plugin-builder/`. +- **Relevance:** The PR's own changes are not the cause. + +### E3: `evals/` arrived on `main` in #218, and PR #219 branched from that commit + +- **Source:** `git log -- evals`, `git merge-base HEAD origin/main` +- **Finding:** + ``` + abba73a adding Skillwalker evals for skills, agents, etc. (#218) + merge-base(HEAD, origin/main) = abba73a + git ls-files evals | wc -l = 247 + ``` +- **Relevance:** The failing files came from `main`, not from the PR. + +### E4: `main` has failed the same way since #218 + +- **Source:** `gh run list --branch main --workflow ci.yml`, run 35780701011 +- **Finding:** + ``` + failure adding Skillwalker evals for skills, agents, etc. (#218) 2026-09-22 + success chore(release): v5.5.0 2026-09-15 + ``` + Run 35780701011 reports `prettier...Failed`. +- **Relevance:** The failure predates PR #219 and is present on `main`. + +### E5: The Prettier hook rewrites files, and `evals/` is not exempt + +- **Source:** `.pre-commit-config.yaml:17-23`, `.prettierignore`, `package.json:7` +- **Finding:** + ```yaml + - id: prettier + entry: npx prettier --write --ignore-unknown + types_or: [markdown, json, yaml, javascript] + ``` + `npm run lint` runs `prek run --all-files`. `.prettierignore` lists `docs/plans/`, `docs/research/`, `han-reporting/skills/html-summary/assets/`, one HTML template, `node_modules/`, and `package-lock.json`. +- **Relevance:** Any unformatted `evals/` file fails the job. + +### E6: Prettier's changes to `evals/` are whitespace and layout only + +- **Source:** diff of a Prettier-formatted copy of `evals/` against the committed tree +- **Finding:** + ``` + prompts/*.md add missing final newline + rubrics/*.md insert blank lines (e.g. after a heading, before a list) + tests.json expand one-line objects wider than 120 columns + scaffolds/**.md pad table columns into alignment + ``` +- **Relevance:** Formatting changes no eval's content. + +### E7: With CI's pinned tools, formatting `evals/` makes lint pass + +- **Source:** throwaway `git worktree` at `HEAD`, after `npm ci` +- **Finding:** + ``` + npx prettier --version 3.9.6 + npx prettier --list-different evals 85 files + npx prettier --write evals && npm run lint + -> all 13 hooks Passed, exit 0 + ``` +- **Relevance:** Confirms the fix and that no other lint failure hides behind this one. + +## Validation Results + +An adversarial validator re-ran every evidence item against live CI data and two clean worktrees. It confirmed the root cause and the fix, and refuted one claim about the alternative fix. + +### Counter-Evidence Investigated + +#### V1: Was the causal chain from #218 to PR #219 real? + +- **Hypothesis:** E1 to E4 are stale or wrong. +- **Investigation:** Re-read the PR #219 job log, the PR file list, the merge base, and `main`'s run history with `gh`. +- **Result:** Confirmed. +- **Impact:** Stronger than first written. PR #218 failed lint in all 4 of its pre-merge CI runs (35740115461, 35765129642, 35766777178, 35773692481) and merged with the red check. `main` has no branch protection (`gh api repos/testdouble/han/branches/main/protection` returns 404), so nothing blocked the merge. + +#### V2: Does formatting `evals/` make lint pass? + +- **Hypothesis:** The fix misses a failure. +- **Investigation:** Clean worktree at `78a98e4`, `npm ci`, format `evals/`, run `npm run lint`. +- **Result:** Confirmed. 85 files changed, and all 13 hooks passed. +- **Impact:** None. + +#### V3: Does the reformat change any content? + +- **Hypothesis:** Prettier renumbers an ordered list, trims padding inside inline code, or alters JSON or code fences. The `.prettierignore` header names the first two as transforms Prettier cannot turn off. +- **Investigation:** Compared all 12 changed `tests.json` files as parsed JSON, every inline code span, every ordered-list marker, and every fenced code block before and after. Checked a prompt file's new final newline at the byte level. +- **Result:** Confirmed. No content changed. The 85 files break down as 63 prompts, 8 rubrics, 12 `tests.json`, and 2 scaffolds. +- **Impact:** Added the reviewer note to the Planned Fix. The validator briefly misread the scaffold table diff as dropped rows before a full diff showed 49 rows before and after. + +#### V4: Does ignoring `evals/` need a second config change? + +- **Hypothesis:** The draft claimed the `fix end of files` hook would fail in Prettier's place if `evals/` were ignored. +- **Investigation:** Read `.pre-commit-config.yaml:41-48`, then added only `evals/` to `.prettierignore` on the unformatted tree and ran `npm run lint`. +- **Result:** Refuted. All 13 hooks passed. The hygiene hooks already skip `.md` and `.json` files. +- **Impact:** Rewrote the alternative's rejection reason (see Adjustments Made). + +#### V5: Is another failure hiding behind this one? + +- **Hypothesis:** The PR's own files carry a lint problem masked by the `evals/` noise. +- **Investigation:** `npx prettier --list-different .` over the whole repo, plus the real CI hook results. +- **Result:** Confirmed clean. No file outside `evals/` needs formatting, and the `test` job passes. +- **Impact:** None. + +#### V6: Could the fix pass locally but fail in CI? + +- **Hypothesis:** Prettier version drift or a stale prek cache. +- **Investigation:** `.github/workflows/ci.yml` runs `npm ci`, which installs Prettier 3.9.6 from `package-lock.json`. The prek cache holds tool installs keyed on `.pre-commit-config.yaml`, not per-file results. +- **Result:** Refuted as a risk. +- **Impact:** None. + +### Adjustments Made + +- **V4:** The "Alternative considered" section now says ignoring `evals/` would work alone. It gives the real reason to format instead: eval prompts and rubrics are live agent-facing content, not the archives and vendored files `.prettierignore` exists for. +- **V3:** Added a note to tell reviewers the fix commit is whitespace-only. +- **V1:** Recorded that #218 merged with a failing check into an unprotected `main`. + +### Confidence Assessment + +- **Confidence:** High +- **Remaining Risks:** + - Skillwalker, the external tool that runs these evals, is not in this repo, so nothing here shows how it reads `evals/`. If it compares any file byte for byte, the reformat could change a result. No evidence either way. Run one eval after the fix to check. + - `main` has no branch protection, so another PR can merge with a red lint check the same way #218 did. Out of scope for this fix, but it is how the failure reached `main`. + +## Coding Standards Reference + +| Standard | Source | Applies To | +| ------------------------------------------------------------------------ | ----------------------------------------- | ------------------- | +| Prettier formats Markdown, JSON, YAML, JS at print width 120, prose preserved | `.prettierrc.json`, `.pre-commit-config.yaml` | every `evals/` file | +| Lint runs with npm-pinned tools, identical in CI and locally | `.github/workflows/ci.yml`, `package.json` | the fix's commands | diff --git a/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/change-decision-log.md b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/change-decision-log.md new file mode 100644 index 00000000..d53366cb --- /dev/null +++ b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/change-decision-log.md @@ -0,0 +1,489 @@ +# Change Decision Log: Opus 5.5 plugin-builder guidance update + + + +## Trivial decisions + +- D-13: Gotchas as a reference-file category — defined once in `skill-reference-files.md`'s category list as + failure points found in use and distinct from a checklist, and named with a link in `progressive-disclosure.md` + (review finding JD-13). — Referenced in plan: Surface Delta. +- D-14: Verification skills — a new subsection in `success-criteria-and-testing.md`, separated from testing the skill + itself. — Referenced in plan: Surface Delta. +- D-16: Helper-script libraries — added to `hardening-fuzzy-vs-deterministic.md`'s Deterministic Steps in + language-neutral words (review finding JD-14), with the permission-prompt caveat cross-referenced to + `script-execution-instructions.md`. — Referenced in plan: Surface Delta. + +## Full decisions + +### D-1: Add Opus 5.5 to the per-model guide and every file that restates its model list + +- **Question:** How does the plugin come to name Opus 5.5, and what does it say about Opus 5 guidance that 5.5 inherits? +- **Decision:** Add Opus 5.5 to `per-model-authoring.md`, bump its "last checked" stamp to the date the build runs, + and state that Opus 5.5 inherits the Opus 5 guidance except where the guide says otherwise. Mirror the new model list + in `guidance/SKILL.md:48-50`, `assets/guidance-portable-SKILL.md:27-29`, `assets/rule-index-body.md:166-171`, and + `specialization-and-model-selection.md:89`. Keep "Fable 5" as written. +- **Rationale:** The guide dates itself before Opus 5.5 shipped and names no Opus 5.5 behavior. SOURCE 4 says "Existing + Claude Opus 5 prompts should perform well without changes," so saying that 5.5 inherits the Opus 5 advice is both + accurate and the smallest change. The four mirrors are independent copies, so missing one leaves a stale router or rule + index in every repo that vendors the guidance. +- **Evidence:** C-1, C-2; SOURCE 4 opening paragraph; precedent commit `59398fc`, which swept the same four surfaces for + Opus 5. +- **Behavior impact:** Preserving. Every existing recommendation still stands. Readers see an extra model named, and the + router routes the same questions to the same file. +- **Rejected alternatives:** + - Rename Fable 5 to Fable 5.1 throughout — rejected because the four sources mention Fable 5.1 once, about biology + safeguards, and say nothing about its instruction style (group 1 validation). + - Write a standalone Opus 5.5 guide — rejected because it would duplicate the Opus 5 content SOURCE 4 says still + applies. +- **Revisit criterion:** A Fable 5.1 review (see Cut for Scope), or an Anthropic page stating that an Opus 5 + recommendation no longer holds on 5.5. +- **Dissent (if any):** None. +- **Settles delta entry:** S-1 +- **Dependent decisions:** D-2, D-3, D-4, D-6, D-18 +- **Referenced in plan:** What Changes, In One Paragraph; Surface Delta; Change Units + +### D-2: The reasoning-echo warning covers Opus 5.5 and names the refusal category + +- **Question:** Does the reasoning-echo warning stay Fable 5 only? +- **Decision:** Extend the section to Opus 5.5 and name the `reasoning_extraction` refusal category. Replace + "single-source, not independently corroborated" with "single-vendor, stated on the Fable 5 page, the Opus 5.5 page, + and the Opus 5.5 blog post". Update the rule-index description that names "the Fable 5 reasoning-echo refusal" to match. +- **Rationale:** Opus 5.5 can refuse an author's "explain your reasoning" instruction outright, a functional failure + rather than a style mismatch. SOURCE 1 and SOURCE 4 both say so. +- **Evidence:** C-3; SOURCE 1 "Don't ask it to show its reasoning in the reply"; SOURCE 4 "Safeguard refusals". +- **Behavior impact:** Preserving. The existing test for spotting the pattern is unchanged; it now applies to two models. +- **Rejected alternatives:** + - Describe the claim as independently corroborated — rejected because every source is an Anthropic publication + (group 1 validation, V8). +- **Revisit criterion:** A source saying the refusal no longer applies on either model. +- **Dissent (if any):** None. +- **Settles delta entry:** S-2 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-3: One clause on Opus 5.5 thinking, not a new section + +- **Question:** How much does the guide say about thinking on Opus 5.5? +- **Decision:** Add one clause to the thinking-mode bullet: Opus 5.5 always thinks and cannot disable it, so the + XML-tag-leak caveat, which applies only when thinking is disabled on Opus 5, does not apply to it. +- **Rationale:** The bullet already turns an API fact into author-facing advice. The only thing an author needs to know + about 5.5 is that the leak caveat does not apply. +- **Evidence:** C-4; SOURCE 4 "Prompts written for thinking disabled". +- **Behavior impact:** Preserving. +- **Rejected alternatives:** + - Carry SOURCE 4's four-step migration list for thinking-disabled integrations — rejected because it covers API request + fields (`thinking.display`, `max_tokens`) that a skill author does not set. +- **Revisit criterion:** Claude Code exposing a thinking toggle to skills or agents. +- **Dissent (if any):** None. +- **Settles delta entry:** S-3 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-4: Tell authors to re-test a pinned effort value on Opus 5.5 + +- **Question:** What does an author who pinned `effort` in frontmatter need to know? +- **Decision:** Add to the effort bullet: the same effort name means more thinking on Opus 5.5 than on Opus 5; + Anthropic's platform docs put 5.5's API default at `medium` (Opus 5's is `high`) and report `medium` on 5.5 matching + or exceeding Opus 5 at `high`; and `xhigh` and `max` are for measured gains. The default is attributed to the API docs + because Claude Code's own session default is not stated in any source (review finding JD-9). A pinned value tuned + for Opus 5 should be re-tested rather than carried over. Add a one-line pointer to the section "Other settings the + model differences affect" beside the `effort` row in `skill-frontmatter-fields.md` and in `agent-external-files.md`. +- **Rationale:** Skill and agent frontmatter can pin effort, so a value chosen for Opus 5 silently produces longer, + costlier turns on 5.5. The pointers put the warning where an author first sees the field. +- **Evidence:** C-5; SOURCE 4 "Calibrate effort"; SOURCE 1 "To change how much it thinks in Claude Code, change effort." +- **Behavior impact:** Preserving. Additive. +- **Rejected alternatives:** + - Recommend a specific effort level for skills — rejected because SOURCE 4 says to test levels against your own evals, + and the plugin's specialization guide already owns tier choice. +- **Revisit criterion:** A later model changing the default again. +- **Dissent (if any):** None. +- **Settles delta entry:** S-4 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-5: Keep the limiting-phrase warning and separate vague limits from a concrete bar + +- **Question:** Does the warning against "only report X" review instructions stay, given that SOURCE 1 recommends one? +- **Decision:** Keep the warning and its report-then-filter default for vague limits ("high-severity", "be + conservative"). Add that a limit is acceptable when it names a bar a reader could check (a consequence such as "would block + the merge", or a rubric the skill itself carries) and requires evidence for each item. Quote SOURCE 1's example whole, + evidence clause included: "List only problems you'd block the merge for. For each one, give the file and line, why it's + wrong, and how to show it fails." This test and its two examples are the plan's contract 3, shared with D-18 (review + finding JD-6). +- **Rationale:** No source says Opus 5.5 is less literal than Opus 5, so the warning's premise still holds. SOURCE 1's + prompt differs from the vague examples in two ways the model can act on: the bar is observable, and each finding must + carry proof. +- **Evidence:** C-6; SOURCE 1 "Ask it to review the code"; group 3 validation (K-14). +- **Behavior impact:** Changing. A reader who used to avoid every limiting phrase now learns one shape of limit that is + acceptable. The user was asked on 2026-09-23 and answered: "go with recommendation for this one" (option A: keep the + warning and add the distinction). +- **Rejected alternatives:** + - Keep the warning unchanged (option B) — rejected by the user, and it leaves the primary source's recommended prompt + reading as a mistake. + - Drop the warning for Opus 5.5 (option C) — rejected because no source supports it. + - Distinguish reusable skill instructions from one-off prompts (a validator's proposal) — rejected because a skill + author only ever writes reusable instructions, so the distinction gives them nothing to act on. +- **Revisit criterion:** A source stating Opus 5.5 under-reports with a concrete bar, or no longer follows limits + literally. +- **Dissent (if any):** None. +- **Settles delta entry:** S-5 +- **Dependent decisions:** D-18 +- **Referenced in plan:** Target State; Surface Delta; Behavior Changes; Review Findings + +### D-6: Add the Opus 5.5 sources, labeled by tier, without touching the research report + +- **Question:** What goes in the per-model Sources section? +- **Decision:** Add the Opus 5.5 prompting page as an Anthropic docs entry. Add SOURCE 1 labeled as an Anthropic blog + post with its author and date. Extend the existing "postdates that research report" sentence to cover the Opus 5.5 + material. Leave `docs/research/model-specific-guidance-for-skills.md` and `multi-agent-economics.md`'s Opus 5 URL + unchanged. +- **Rationale:** The section already distinguishes evidence tiers. The research report is a dated artifact that the + guide already marks as superseded. `multi-agent-economics.md`'s Opus 5 URL still supports the claim it is cited for. +- **Evidence:** C-7; group 1 validation (K-28, K-11). +- **Behavior impact:** Preserving. +- **Rejected alternatives:** + - Rewrite the research report — rejected because `docs/research/` holds point-in-time reports. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-6 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-7: Name the stops a long autonomous stretch should and should not make, and protect the collaborative stops + +- **Question:** How does the guidance answer Opus 5.5 ending a turn early with a report? +- **Decision:** Add a subsection after "Human Gates in Workflow Steps" in `workflow-patterns.md`. For a stretch of a + skill meant to run without the user, such as a fan-out or a long tool-calling loop, the step names the stops it does + not want: a summary that announces the next step instead of taking it, an offer to continue, and a list of choices + that block nothing. It also names the stops it does want: no work can move without the user, a destructive or + irreversible action is next, or a human gate is reached. The subsection says plainly that human gates, interview + turns, and `han-core`'s collaborative-stop boundaries are wanted stops, and that the keep-going instruction does not + belong in a skill whose design is to hand control back. Add one sentence to `per-model-authoring.md` naming the Opus + 5.5 behavior and linking to the subsection. +- **Rationale:** SOURCE 1 and SOURCE 4 both describe the behavior and both say naming the stops works. Both also say to + leave the keep-going instruction out of human-in-the-loop work ("For pair programming, you may want the opposite"; + "leave the addition out of human-in-the-loop applications"). Most Han skills are that shape, so an unscoped rule would + weaken them. +- **Evidence:** C-8; SOURCE 1 "Tell it which stops you want"; SOURCE 4 "Unattended agentic runs"; group 2 validation + (K-5). +- **Behavior impact:** Preserving. Additive. No existing gate or stop rule changes. +- **Rejected alternatives:** + - A blanket keep-going rule for all skills — rejected because it contradicts both sources' human-in-the-loop carve-out + and `han-core/references/collaborative-stop-rule.md`. + - Put the whole pattern in `per-model-authoring.md` — rejected because the pattern holds on any model that stops + early. The guide carries only the model fact. +- **Revisit criterion:** A report of a Han skill stopping mid-stretch on Opus 5.5 despite the guidance, or of the + guidance being pasted into a collaborative skill. +- **Dissent (if any):** None. +- **Settles delta entry:** S-7 +- **Dependent decisions:** D-8 +- **Referenced in plan:** Surface Delta; Risks + +### D-8: State the finish line and keep the task list in a file, only for unbounded steps + +- **Question:** Does every skill need a stated finish line and a task file? +- **Decision:** No. Add one subsection to `workflow-patterns.md` for steps that iterate over a set whose size is not + known up front: every file, every endpoint, every finding. Such a step states its completion condition in checkable + terms, keeps its checklist in a file it updates as items finish, and reads that file rather than the conversation to + decide what remains. Cross-reference `skill-composition.md`'s continuation rule and `multi-agent-economics.md`. +- **Rationale:** A fixed flowchart skill already ends at its last step (`plugin-entity-taxonomy.md`). The sources' finish + line and TASKS.md advice targets open-ended, multi-part work, which inside a skill is the unbounded step. +- **Evidence:** C-9; SOURCE 1 "Say what 'done' looks like" and "Keep the task list in a file"; SOURCE 4 "keep the task's + parts in a checklist the model updates, such as a to-do tool or a file"; group 2 validation (K-6, K-7). +- **Behavior impact:** Preserving. Additive. +- **Rejected alternatives:** + - A finish line on every skill — rejected as restating what the flowchart already guarantees. + - SOURCE 4's checker model and continuation cap — rejected because both need a custom harness a skill author does not + have. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-8 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-9: Check fanned-out evidence and wait for background dispatches + +- **Question:** What do multi-agent skills need that the delegation rules do not say? +- **Decision:** In `multi-agent-economics.md`, add a rule for fan-out: when a skill gives each unit of work to its own + subagent, the skill checks each returned finding's evidence before accepting it, then produces one consolidated + result. State the line against the existing self-check ban: checking the evidence a subagent cites against its source is + allowed; dispatching another agent to redo the skill's own reasoning is not (review finding JD-8). Under + "Practical Implications for Skills," add that a skill does not treat its run as complete while a dispatch it launched + with `run_in_background: true` is still running, and label that item as resting on a single source. Add a + cross-reference beside the `background` row in `agent-external-files.md`. +- **Rationale:** SOURCE 1 gives the fan-out pattern as a worked prompt. Han already dispatches in the background, at + `han-coding/skills/automated-test-planning/SKILL.md:134`. +- **Evidence:** C-10; SOURCE 1 FIG C; SOURCE 4 "If something the model started is still running, such as a background + command or a subagent, don't treat the task as done yet"; group 2 and group 3 validation (K-8, K-9). +- **Behavior impact:** Preserving. Additive. +- **Rejected alternatives:** + - Tie the wait rule to the frontmatter `background: true` field only — rejected because no Han agent uses that field; + the `Agent` tool parameter is the mechanism in use. + - A continuation cap — rejected as harness-only. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-9 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta + +### D-10: Agents say what they could not confirm and where they looked + +- **Question:** Where does "mark anything you couldn't confirm" belong? +- **Decision:** Add a second rule to `agent-building-guidelines/graceful-degradation.md`. When a research or analysis + agent cannot establish a claim even with every tool available, it says so on the claim and names where it looked. + Reuse the file's "note the limitation" wording. Update `agent-builder` Step 6 item 8 so its one-line summary names both + rules, since the review re-reads that file. +- **Rationale:** The existing rule fires only on a missing tool. The source's case, where the agent looked and found + nothing, is a different trigger. +- **Evidence:** C-11; SOURCE 1 "Ask it to mark what it couldn't confirm"; group 3 validation (K-12). +- **Behavior impact:** Changing for `agent-builder` users, whose final review checks one more condition. The guide + itself only gains a rule. Operator decision recorded in D-23. +- **Rejected alternatives:** + - Land it in the skill-side `graceful-degradation.md` — rejected because that file covers environment detection, not + agent output. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-10 +- **Dependent decisions:** D-23 +- **Referenced in plan:** Surface Delta; Behavior Changes + +### D-11: Add a limit to "be specific" + +- **Question:** Should the instruction-writing guide tell authors when not to be specific? +- **Decision:** Add a rule beside "Be specific and actionable" in `writing-effective-instructions.md`. Spell out + conventions that break something when missed. Leave out instructions for routine details the model gets right + unprompted, rather than writing blanket rules for them. The wording avoids "judgment", which `plugin-entity-taxonomy.md` + reserves for agents (review finding JD-7). The rule governs how much detail an instruction carries, not whether a step is fixed. Skills stay + flowchartable, per `plugin-entity-taxonomy.md`. Link the taxonomy's flowchart test and the per-model middle path rather than restating either. +- **Rationale:** The baseline articles report that newer models are over-constrained by blanket rules, and that + Anthropic removed such rules from Claude Code with no measured loss. The existing counterweight covers only an unknown + target model. +- **Evidence:** C-12; SOURCE 2 "Avoid railroading Claude"; SOURCE 3 "Then: Give Claude rules / Now: Let Claude use + judgement" and "Avoid making them overconstrained, except in highly important areas"; group 4 validation (K-16). +- **Behavior impact:** Changing. Readers of the specificity rule now meet a stated limit on it. The user was asked on + 2026-09-23 and answered: "go with recommendation" (option A: add the counterweight). +- **Rejected alternatives:** + - Leave the rule as it is (option B) — rejected by the user. +- **Revisit criterion:** Evidence that skills written under the new rule lose conventions they needed. +- **Dissent (if any):** A validator warned that the wording could blur the line between a skill and an agent. The + decision limits the rule to instruction detail for that reason. +- **Settles delta entry:** S-11 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta; Behavior Changes; Risks + +### D-12: Name "restates a default" as removable, in the list the builder review reads + +- **Question:** Where does "don't state the obvious" land, and what does it change? +- **Decision:** Add one bullet to `progressive-disclosure.md`'s "When to remove entirely" list: instructions restating + what the model already does by default, such as "read the code before changing it." Link `context-hygiene.md`'s + "every token must earn its place" rule, which already states the principle. +- **Rationale:** `context-hygiene.md` states the principle but the `skill-builder` review does not cite it. + `progressive-disclosure.md` is the file that review names, so the example lands where it is applied. +- **Evidence:** C-13; SOURCE 2 "Don't state the obvious"; SOURCE 3 "Avoid stating 'the obvious'"; group 4 validation + (K-17). +- **Behavior impact:** Changing for `skill-builder` users: its Step 6 review will now cut such instructions from the + skills it builds (review finding JD-4). Operator decision recorded in D-23. +- **Rejected alternatives:** + - A new rule in `context-hygiene.md` — rejected because the principle is already there and the builder review would + not see it. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-12 +- **Dependent decisions:** D-23 +- **Referenced in plan:** Surface Delta; Behavior Changes + +### D-15: Keep skill state in the plugin data directory, documented as a section, not a new file + +- **Question:** Where does guidance on first-run setup and skill memory live, and where does the state itself live? +- **Decision:** Add a section beside the environment-variable table in `plugin-json-options.md` covering two uses of + `${CLAUDE_PLUGIN_DATA}` for a skill shipped in a plugin. The first is first-run setup: read a saved answer, ask with + `AskUserQuestion` when it is absent (under the existing rule that keeps that tool out of `allowed-tools`), and save + it. The second is run memory: an append-only log or JSON file read at the next run. The section also says: + - the skill's own directory is under `${CLAUDE_PLUGIN_ROOT}` and is replaced on update, so nothing is written there; + - `userConfig` is the choice for a value known at install time; + - a file the user edits by hand belongs where the user edits it; + - a skill checked into `.claude/skills/`, including one vendored by `/guidance init`, has no plugin data directory. + + `allowed-tools-AskUserQuestion.md` links to the section. +- **Rationale:** SOURCE 2 describes both patterns with a `config.json` in the skill directory, and points to + `${CLAUDE_PLUGIN_DATA}` for memory. For an installed plugin, the skill directory does not survive an update. The two + facts that make this work already sit in `plugin-json-options.md`'s environment-variable table, so a section there + adds the least structure. No Han skill uses the data directory today, which argues for the smaller form. +- **Evidence:** C-16, C-21; SOURCE 2 "Think through the setup" and "Help Claude remember"; group 5 validation (K-21, + K-22); review findings JD-1, JD-2, R5. +- **Behavior impact:** Preserving. Additive. +- **Rejected alternatives:** + - A new `skill-persistent-state.md` file (the draft's choice) — rejected in review because it added a file, a + rule-index line, and a builder-routing question for content two table rows already anchor. + - Two separate files — rejected because both would explain the same directory. + - Follow SOURCE 2 literally and use `config.json` in the skill directory — rejected because the plugin's own + reference says that path "Changes on plugin update." +- **Revisit criterion:** The section outgrows its host file, or a Han skill starts saving state. +- **Dissent (if any):** The group 5 validator recommended a new file. The review round reversed that. +- **Settles delta entry:** S-15 +- **Dependent decisions:** — +- **Referenced in plan:** Target State; Surface Delta; Deferred (YAGNI); Review Findings + +### D-17: One sentence on skill-scoped guardrail hooks + +- **Question:** How much hooks guidance does the plugin add? +- **Decision:** Extend the `hooks` row in `skill-frontmatter-fields.md` with one sentence: a hook declared there lasts + only for the skill's run, which suits a guardrail wanted only while that skill is active, such as blocking destructive + shell commands. Link the official hooks reference for the schema. Defer the usage-logging hook. +- **Rationale:** SOURCE 2 names the on-demand guardrail, and the plugin mentions skill hooks only as a bare table row. + No Han plugin or skill ships a hook, so anything larger fails the same evidence test as the deferred hooks guide. The + usage-logging hook lives in plugin or settings configuration, not in the frontmatter inventory. +- **Evidence:** C-18; SOURCE 2 "Use on-demand hooks" and "Measuring skills"; group 5 validation (K-24); review + findings JD-5, R6. +- **Behavior impact:** Preserving. Additive. +- **Rejected alternatives:** + - A new hooks guide covering the full schema (validator's proposal) — deferred. + - A section with both the guardrail and the usage-logging patterns (the draft's choice) — narrowed in review. +- **Revisit criterion:** A Han skill ships a hook, or an author asks how to measure how often a skill is used. +- **Dissent (if any):** The group 5 validator recommended the larger guide. +- **Settles delta entry:** S-17 +- **Dependent decisions:** — +- **Referenced in plan:** Surface Delta; Deferred (YAGNI); Review Findings + +### D-18: The builders read the per-model guide when a target model is named, and always check for leftovers + +- **Question:** Should `skill-builder` and `agent-builder` route to and review against `per-model-authoring.md`? +- **Decision:** Add a row to each builder's decision table: "Instructions for a named target model → + `per-model-authoring.md`." Add one item to each builder's Step 6 review: the finished files carry no model-specific + leftovers. That means no "think step by step" or "think carefully" line, no instruction to reproduce reasoning in the + reply, no pinned `effort` without a reason stated in the skill or agent body, and no vague limiting phrase in a review + step, judged by the plan's contract 3 with both of its examples quoted in each Step 6 (review finding JD-6). +- **Rationale:** Without this, the Opus 5.5 guidance reaches only people who ask the `guidance` skill directly. The + check applies the guide's own model-agnostic default rather than contradicting it. +- **Evidence:** C-19, C-20; group 2 validation (K-25). +- **Behavior impact:** Changing. Builder runs read one more file when a target model comes up, and the final review + checks one more item. The user was asked on 2026-09-23 and answered: "go with recommendation" (option A: add it to both + builders). +- **Rejected alternatives:** + - Leave the builders unchanged (option B) — rejected by the user. + - Also make `agent-builder` cite `writing-effective-instructions.md` — cut for scope; the sources do not drive it. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-18 +- **Dependent decisions:** D-23 +- **Referenced in plan:** Surface Delta; Behavior Changes; Review Findings + +### D-19: The source set and its precedence + +- **Question:** Which sources does the evaluation run against, and which wins when they conflict? +- **Decision:** SOURCE 1 is primary and overrides the others. SOURCE 2 is the operator-named baseline. SOURCE 3 (linked + as related from both operator-named articles) and SOURCE 4 (Anthropic's own Opus 5.5 prompting page) are related + baseline sources. Entries resting only on SOURCE 2 or SOURCE 3 are labeled baseline gaps rather than Opus 5.5 changes. +- **Rationale:** The operator said to include "related sources in this eval as a baseline, but letting the article on + getting the most out of opus 5.5 override anything from this one." SOURCE 4 is the platform page for the same model. +- **Evidence:** Operator message recorded in `scope-boundary.md`. +- **Behavior impact:** — +- **Rejected alternatives:** + - Treat SOURCE 4 as primary because it is the platform docs — rejected because the operator named SOURCE 1. The two + do not conflict on any retained entry. +- **Revisit criterion:** — +- **Dissent (if any):** A validator noted that SOURCE 1 is a blog post outranking platform docs. No conflict arose. +- **Settles delta entry:** — +- **Dependent decisions:** D-6 +- **Referenced in plan:** Why This Change + +### D-20: Candidates rejected by adversarial validation + +- **Question:** Which proposed changes did not survive? +- **Decision:** Eight of the twenty-eight merged candidates are not in the plan: + - **Time-budget signal for agent teams (K-10)** — needs a harness that appends elapsed time to each message; a skill + author has no such surface. SOURCE 4 only. + - **Delegation eagerness on Opus 5.5 (K-11)** — no source says 5.5 reaches for subagents more or less readily; SOURCE + 4's claim is about endurance. + - **Marking pasted content as untrusted (K-13)** — an API-level tagging technique with a random ID a skill cannot + generate. SOURCE 4 only. + - **The nine skill categories (K-19)** — `skill-decomposition.md:20-39` already gives the single-concern test with a + worked example; SOURCE 2 presents the categories as a portfolio audit. + - **Updating the `claude-opus-5` example ID (K-26)** — the sentence illustrates that pinned IDs go stale, which is + still true. + - **"Repeat a reminder" versus "don't repeat yourself" (K-27)** — different mechanisms: a recency reminder in one + document versus duplicate tool instructions in two places. + - **Examples constrain newer models (K-15)** — deferred; see the plan's Deferred section. + - **A full hooks schema guide (part of K-24)** — deferred; see D-17. +- Also confirmed already covered, so not candidates: removing "think carefully" lines (covered by the thinking bullet's + "do not write 'think step by step' prompt hacks"), `${CLAUDE_PLUGIN_DATA}` as a variable (`plugin-json-options.md`), + skill-scoped `hooks` as a field (`skill-frontmatter-fields.md:49`), composing skills by name + (`skill-composition.md`), and descriptions written for the model (`skill-description-frontmatter.md`). +- **Rationale:** Each item failed the evidence, applicability, or coverage test in its validator's report. +- **Evidence:** Group 1 to 5 validation reports; C-20. +- **Behavior impact:** — +- **Rejected alternatives:** — +- **Revisit criterion:** Per item, as named in Deferred. +- **Dissent (if any):** A validator called K-19's rejection a judgment call. +- **Settles delta entry:** — +- **Dependent decisions:** — +- **Referenced in plan:** Deferred (YAGNI); Review Findings + +### D-21: The target state came from the validators' landing recommendations, not a separate architect pass + +- **Question:** Does the run dispatch `han-core:software-architect` for the target state, as the skill's Step 4 + prescribes? +- **Decision:** No. The target state is file placement of guidance text. Each validator already named a landing file + with evidence and checked it for duplication, and the structural map (C-2, C-21) shows which files move together. +- **Rationale:** An architect pass would re-derive placement from the same evidence with no module or interface design to + add. The deviation is disclosed here and in the plan. +- **Evidence:** Group 1 to 5 validation reports; the structural-analyst report. +- **Behavior impact:** — +- **Rejected alternatives:** + - Dispatch the architect anyway — rejected as cost without new information. +- **Revisit criterion:** The review round finding a placement conflict the validators missed. +- **Dissent (if any):** None. +- **Settles delta entry:** — +- **Dependent decisions:** — +- **Referenced in plan:** Target State + +### D-22: Pin every widened file's rule-index description + +- **Question:** Which rule-index descriptions change, and to what? +- **Decision:** Update the description of every file whose scope this plan widens, using the text in the plan's + contract 2: Per-Model Authoring Guidance, Workflow Patterns, Multi-Agent Economics, Graceful Degradation (agents), + Success Criteria and Testing, Writing Effective Instructions, Skill Reference Files, Hardening: Fuzzy vs. + Deterministic, and plugin.json Schema Reference. +- **Rationale:** The rule index is vendored into consuming repos and is how the model chooses which file to read. A + description that omits a file's new content sends a reader with that question elsewhere. +- **Evidence:** C-21; `rule-index-body.md:39-41`, `:46-51`, `:58-63`, `:97-99`, `:125-135`, `:143-145`; review + findings JD-3, JD-11. +- **Behavior impact:** Preserving. Descriptions widen to match content; no link or routing changes. +- **Rejected alternatives:** + - Update only the two lines the draft pinned — rejected in review because six other descriptions would mislead. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** — (applies across S-1, S-7 to S-11, S-13 to S-16) +- **Dependent decisions:** — +- **Referenced in plan:** Target State; Review Findings + +### D-23: Builder reviews that pick up S-10 and S-12 + +- **Question:** The `agent-builder` and `skill-builder` final reviews re-read guidance files this plan changes. Does the + operator accept that those reviews will now check that research agents mark unconfirmed claims (S-10), and cut + instructions that restate model defaults (S-12)? +- **Decision:** Both builder reviews apply the new guidance. `agent-builder` checks that a research or analysis agent + marks claims it could not confirm (S-10), and `skill-builder` cuts instructions that restate model defaults (S-12). + The operator was asked on 2026-09-23 and answered: "option A". +- **Rationale:** Both are behavior changes that the review round surfaced (JD-4) after the three approved escalations. + They are asked together because they share one mechanism: a builder review applies whatever the guidance file it + re-reads says. +- **Evidence:** C-11, C-13, C-19; review finding JD-4. +- **Behavior impact:** Changing. See the plan's Behavior Changes. +- **Rejected alternatives:** + - Add the guidance but carve the builders out (option B) — rejected by the operator; it adds text and leaves the two + surfaces disagreeing. + - Drop S-10 and S-12 (option C) — rejected by the operator. +- **Revisit criterion:** — +- **Dissent (if any):** None. +- **Settles delta entry:** S-10, S-12 +- **Dependent decisions:** — +- **Referenced in plan:** Behavior Changes; Open Items; Review Findings + diff --git a/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/current-state-findings.md b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/current-state-findings.md new file mode 100644 index 00000000..25892737 --- /dev/null +++ b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/current-state-findings.md @@ -0,0 +1,382 @@ +# Current State Findings: Opus 5.5 plugin-builder guidance update + +## Provenance + +No prior findings report existed. This run produced the findings itself, on 2026-09-23, over the area +`han-plugin-builder/`: + +- `han-core:structural-analyst`: which files restate one another, what `guidance init` vendors, and where dates and model + names are pinned. +- `han-core:behavioral-analyst`: what `skill-builder`, `agent-builder`, and `guidance` read at run time, and how the + builders' own instructions behave. +- Four `han-core:gap-analyzer` runs, one per slice of the plugin's files. Each compared its slice against the source + digest and grepped the whole plugin before reporting a gap. +- Five `han-core:adversarial-validator` runs. Together they tested every merged candidate against counter-evidence. + +The analysis ran against four sources. The operator named the first two. The run added the third and fourth as related +sources; the context-engineering article is linked from both operator-named articles. + +| ID | Source | Role | +| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | +| SOURCE 1 | [Getting the most out of Opus 5.5 in Claude and Claude Code](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/), Addy Osmani, 2026-09-22 | Primary; overrides the others | +| SOURCE 2 | [Lessons from building Claude Code: How we use skills](https://claude.dev/blog/lessons-from-building-claude-code-how-we-use-skills/), Thariq Shihipar, 2026-06-03 | Baseline (operator-named) | +| SOURCE 3 | [The new rules of context engineering for Claude 5 generation models](https://claude.dev/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models/), Thariq Shihipar, 2026-07-24 | Baseline (related post) | +| SOURCE 4 | [Prompting Claude Opus 5.5](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5), Anthropic platform docs | Baseline (related source) | + +All four are Anthropic publications. Where two of them agree, the agreement is single-vendor, two publications. It is +not independent corroboration. + +## Project Context + +- **Stack:** Markdown (skill, agent, and reference content) and Bash (`skills/guidance/scripts/init-guidance.sh`). No + build. +- **Conventions source:** `CLAUDE.md` (`## Project Discovery`, `## Conventions`); the voice profile at + `han-communication/references/writing-voice.md`. +- **ADRs found:** `docs/adr/0001-project-configurable-default-swarm-size.md`. It does not bear on this area. +- **Coding standards found:** none apply to guidance prose. Prettier runs with `printWidth: 120` and + `proseWrap: preserve` (`.prettierrc.json`). +- **Recent churn:** 44 commits touched the plugin in the last 90 days. The closest precedent is `59398fc` + (2026-07-31), "docs(guidance): update per-model authoring for Claude Opus 5". It retargeted `per-model-authoring.md` + to Sonnet 5, Opus 5, and Fable 5, swept the four surfaces that name the model trio, and added the delegation policy to + `multi-agent-economics.md`. `33ed427` (2026-08-20) brought reference files over 100 lines under a `## Contents` + convention. + +## Gaps + +- No automated test covers any guidance file. `test/` holds only `codex-packaging.bats` and `sanity.bats`, and neither + reads guidance content. A content change has nothing automated to break, and nothing automated to confirm it. +- This repo carries no vendored copy of the plugin-building skills. Consuming repos pick up changes when they re-run + `/guidance update`. +- Anthropic publishes a separate Fable 5.1 prompting page (`prompting-claude-fable-5-1`, HTTP 200 on 2026-09-23). This + run did not read it; see the plan's Cut for Scope. + +## Findings + +### C-1: The per-model guide names three models and was last checked before Opus 5.5 shipped + +- **Claim:** `per-model-authoring.md` covers Sonnet 5, Opus 5, and Fable 5 only, and dates itself 2026-07-31. No file in + the plugin names Opus 5.5. +- **Location:** `han-plugin-builder/skills/guidance/references/per-model-authoring.md:5` +- **Evidence:** + ```markdown + _Last checked against Anthropic's published guidance on 2026-07-31, for Sonnet 5, Opus 5, and Fable 5. ..._ + ``` +- **Raised by:** gap-analyzer (slice A, GA-1); structural-analyst S-12; a validator confirmed it (group 1). +- **Confidence:** Verified +- **Bears on:** S-1, D-1 + +### C-2: The model list is restated in four other files, and a change must reach all of them + +- **Claim:** The guidance router, its portable copy, the rule index, and the specialization guide each carry their own + copy of the model-list clause. `init-guidance.sh` vendors the portable copy and the rule index into consuming repos. +- **Location:** `skills/guidance/SKILL.md:48-50`; `skills/guidance/assets/guidance-portable-SKILL.md:27-29` + (byte-identical body); `skills/guidance/assets/rule-index-body.md:166-171`; + `skills/guidance/references/specialization-and-model-selection.md:89` +- **Evidence:** + ```markdown + - Writing the instructions for a target model (how Sonnet 5, Opus 5, and Fable 5 differ in following instructions, which + ``` + ```markdown + Opus 5, and Fable 5 differ in how they follow instructions, ... and the Fable 5 + reasoning-echo refusal to avoid. + ``` +- **Raised by:** structural-analyst S-1, S-10, S-17 +- **Confidence:** Verified +- **Bears on:** S-1 + +### C-3: The reasoning-echo warning is scoped to Fable 5 and called single-source + +- **Claim:** The guide warns only about Fable 5 refusing requests to reproduce internal reasoning, and says the warning + rests on one uncorroborated source. SOURCE 1 and SOURCE 4 both place the same refusal on Opus 5.5, under the + `reasoning_extraction` category. +- **Location:** `per-model-authoring.md:35-41`, `:83`, `:85` +- **Evidence:** + ```markdown + On Fable 5, an instruction that tells the model to reproduce or transcribe its own internal thinking into its visible answer can be refused outright. + ``` + SOURCE 4: "Requests that push the model to reproduce its internal reasoning in the response text can be declined with + the `reasoning_extraction` category, which is new if you're coming from Claude Opus 5." +- **Raised by:** gap-analyzer GA-2; gap-analyzer (slice C) contradiction note; a validator confirmed it (group 1). +- **Confidence:** Verified +- **Bears on:** S-2 + +### C-4: The thinking-mode bullet describes disabling thinking on Opus 5, which Opus 5.5 does not allow + +- **Claim:** The bullet says Opus 5 can disable thinking at effort `high` or below, and ties an XML-tag-leak caveat to + that case. SOURCE 4 says Opus 5.5 does not accept disabled thinking. +- **Location:** `per-model-authoring.md:47` +- **Evidence:** + ```markdown + Opus 5 and Sonnet 5 have thinking on by default; on Opus 5 it can be disabled only at effort `high` or below. Fable 5 always has it on and cannot turn it off. + ``` +- **Raised by:** gap-analyzer GA-3; a validator narrowed it (group 1). +- **Confidence:** Verified +- **Bears on:** S-3 + +### C-5: The effort bullet predates Opus 5.5's default, and authors can pin effort in frontmatter + +- **Claim:** The effort bullet describes Opus 5 only. Skills and agents both accept an `effort` frontmatter value, so a + value tuned for Opus 5 carries over silently. SOURCE 4 says Opus 5.5 defaults to `medium` (Opus 5 defaults to `high`), + that `medium` on 5.5 matches or exceeds Opus 5 at `high`, and that 5.5 thinks more per turn at a given level. +- **Location:** `per-model-authoring.md:48`; `skill-building-guidance/skill-frontmatter-fields.md:46`; + `agent-building-guidelines/agent-external-files.md:112` +- **Evidence:** + ```markdown + | `effort` | No | `low`, `medium`, `high`, `xhigh`, or `max`. Overrides the session's effort level for this skill. + ``` +- **Raised by:** gap-analyzer GA-4, GC-1; a validator confirmed it (group 1). +- **Confidence:** Verified +- **Bears on:** S-4 + +### C-6: The guide warns against every "only report X" review limit, while SOURCE 1 recommends one with a concrete bar + +- **Claim:** The guide treats a limiting phrase in a review instruction as a failure mode, and its two examples are + vague ("high-severity", "be conservative"). SOURCE 1 recommends a limit that names a checkable bar and demands evidence + per item. No source says Opus 5.5 follows instructions less literally than Opus 5. +- **Location:** `per-model-authoring.md:31` +- **Evidence:** + ```markdown + A review instruction that says "only report high-severity issues" or "be conservative" is followed literally, and the run reports less than it found. + ``` + SOURCE 1: "List only problems you'd block the merge for. For each one, give the file and line, why it's wrong, and how + to show it fails." +- **Raised by:** gap-analyzer (slice C) contradiction note; a validator confirmed the tension (group 3). +- **Confidence:** Verified +- **Bears on:** S-5, D-5 + +### C-7: The per-model Sources section cites only the Opus 5-era pages + +- **Claim:** The Sources section lists the Opus 5, Sonnet 5, and Fable 5 prompting pages plus a research report titled + for the Opus 4.8 era. `multi-agent-economics.md` cites the same Opus 5 URL. +- **Location:** `per-model-authoring.md:81-91`; `multi-agent-economics.md:168`; + `docs/research/model-specific-guidance-for-skills.md:1` +- **Evidence:** + ```markdown + - [Prompting Claude Opus 5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5) + ``` +- **Raised by:** structural-analyst S-4, S-14, S-15; a validator narrowed it (group 1). +- **Confidence:** Verified +- **Bears on:** S-6 + +### C-8: Workflow guidance covers deliberate pauses, not a model that stops early to report + +- **Claim:** `workflow-patterns.md` covers human gates placed before irreversible actions. Nothing in the plugin covers + a run that ends its turn with a status report, an offer to continue, or a list of non-blocking choices. SOURCE 1 and + SOURCE 4 both describe that Opus 5.5 behavior. Both also say to leave the keep-going instruction out of + human-in-the-loop work, which is the shape of most Han skills and the whole of `han-core`'s collaborative-stop contract. +- **Location:** `skill-building-guidance/workflow-patterns.md:296-341`; `han-core/references/collaborative-stop-rule.md` +- **Evidence:** + ```markdown + **Human gates** are different: they pause execution to ask the user for confirmation before an irreversible + operation. + ``` +- **Raised by:** gap-analyzer GA-5, GB1-6, GB2-9; a validator narrowed it (group 2). +- **Confidence:** Verified +- **Bears on:** S-7 + +### C-9: No guidance names a finish line or a durable task list for long steps + +- **Claim:** The plugin has per-step validation gates and a planning-time "Expected Result" field. It has no rule for + stating a completion condition on a step that iterates over an unbounded set, and no rule for keeping that step's + progress in a file that survives context summarization. The two nearby rules cover other mechanisms: continuation after + a Skill call, and the skill definition surviving compaction. +- **Location:** `skill-building-guidance/skill-composition.md:81-85`; `skill-building-guidance/context-hygiene.md:191-199`; + `skill-building-guidance/use-case-planning.md:129-143` +- **Evidence:** + ```markdown + A Skill call mid-workflow is the moment the calling model is most likely to stop, treating the sub-skill's output as its + own final answer. + ``` +- **Raised by:** gap-analyzer GB1-5, GB1-7, GB2-8, GC-5; a validator narrowed it (group 2). +- **Confidence:** Verified +- **Bears on:** S-8 + +### C-10: Delegation guidance has no evidence check on returned subagent work and no wait for background dispatches + +- **Claim:** `multi-agent-economics.md` has three delegation rules: do not delegate small work, do not dispatch an agent + to check the dispatching skill's own work, and prefer one agent to several. None covers checking a fanned-out + subagent's evidence before accepting it, which SOURCE 1 recommends, or waiting for a background dispatch before calling + the run done, which SOURCE 4 recommends. Han skills do dispatch in the background. +- **Location:** `agent-building-guidelines/multi-agent-economics.md:33-48`, `:131`; + `han-coding/skills/automated-test-planning/SKILL.md:134` +- **Evidence:** + ```markdown + - **Do not dispatch an agent to verify or double-check the dispatching skill's own work.** This is a self-check, and + ``` + ```markdown + Launch the testing agents **in parallel** using the `Agent` tool with `run_in_background: true`. + ``` +- **Raised by:** gap-analyzer GC-2, GC-3, GB2-10, GB2-11; validators narrowed it (groups 2 and 3). +- **Confidence:** Verified +- **Bears on:** S-9 + +### C-11: Agent graceful-degradation covers a missing tool, not a claim the agent could not confirm + +- **Claim:** The agent rule fires when a tool is unavailable. It says nothing about an agent that had every tool, looked, + and still could not establish a claim. +- **Location:** `agent-building-guidelines/graceful-degradation.md:16-27` +- **Evidence:** + ```markdown + If the tool is not available, skip the step and note the limitation explicitly in the agent's output. + ``` +- **Raised by:** gap-analyzer GB1-8, GC-6; a validator narrowed it (group 3). +- **Confidence:** Verified +- **Bears on:** S-10 + +### C-12: The instruction-writing guide pushes toward specificity with no stated limit + +- **Claim:** "Be specific and actionable" has no counterweight in the instruction-writing guide. The per-model guide's + middle path (goal, reasons, and load-bearing constraints, with no exhaustive checklist) applies only when the target + model is unknown. +- **Location:** `skill-building-guidance/writing-effective-instructions.md:34`; `per-model-authoring.md:17-21` +- **Evidence:** + ```markdown + ### Rule: Be specific and actionable + ``` +- **Raised by:** gap-analyzer GB1-2; a validator narrowed it (group 4). +- **Confidence:** Verified +- **Bears on:** S-11, D-11 + +### C-13: "When to remove entirely" names only toolchain-enforced rules + +- **Claim:** The removal list the builder review reads covers rules a linter enforces, not instructions restating what + the model already does by default. `context-hygiene.md` states the general principle, but the builder review does not + cite it. +- **Location:** `skill-building-guidance/progressive-disclosure.md:111-116`; `skill-building-guidance/context-hygiene.md:36-40` +- **Evidence:** + ```markdown + - Rules already enforced by the toolchain (linters, formatters, CI checks). + ``` +- **Raised by:** gap-analyzer GB1-3; a validator narrowed it (group 4). +- **Confidence:** Verified +- **Bears on:** S-12 + +### C-14: No reference-file category holds accumulated gotchas + +- **Claim:** "Gotcha" appears nowhere in the plugin. The two lists of reference-file categories name templates, + checklists, rate tables, decision matrices, style guides, and canonical examples. `troubleshooting.md` covers failures + to build or load a skill, which is a different audience. +- **Location:** `skill-building-guidance/skill-reference-files.md:29-37`; `skill-building-guidance/progressive-disclosure.md:95-101` +- **Evidence:** + ```markdown + - **Canonical examples** that demonstrate conventions the skill enforces (2-3 representative "do this / not this" code + ``` +- **Raised by:** gap-analyzer GB1-4, GB2-3; a validator confirmed it (group 4). +- **Confidence:** Verified +- **Bears on:** S-13 + +### C-15: Testing guidance covers testing a skill, not building a skill that verifies a product + +- **Claim:** `success-criteria-and-testing.md` covers triggering tests, functional tests, and performance comparison of + the skill under construction. It has no pattern for a skill whose job is to drive a running product and assert its + state. +- **Location:** `skill-building-guidance/success-criteria-and-testing.md:30-181` +- **Evidence:** section headings `## Triggering Tests`, `## Functional Tests`, `## Performance Comparison`. +- **Raised by:** gap-analyzer GB2-2; a validator confirmed it (group 5). +- **Confidence:** Verified +- **Bears on:** S-14 + +### C-16: The persistent data directory is documented only as a plugin.json variable + +- **Claim:** `plugin-json-options.md` defines `${CLAUDE_PLUGIN_ROOT}` (changes on update) and `${CLAUDE_PLUGIN_DATA}` + (survives updates), plus `userConfig`. No skill-building guide tells an author where to keep a skill's first-run + answers or its run history. SOURCE 2 describes a `config.json` inside the skill directory. In an installed plugin, that + directory sits under `${CLAUDE_PLUGIN_ROOT}` and is replaced on update. +- **Location:** `claude-marketplace-and-plugin-configuration/plugin-json-options.md:72-88`, `:137-138` +- **Evidence:** + ```markdown + | `${CLAUDE_PLUGIN_ROOT}` | Absolute path to plugin install directory. Changes on plugin update. | + | `${CLAUDE_PLUGIN_DATA}` | Persistent state dir at `~/.claude/plugins/data/{id}/`. Survives updates. | + ``` +- **Raised by:** gap-analyzer GB2-4, GB2-5, GB1-9; a validator narrowed it (group 5). +- **Confidence:** Verified +- **Bears on:** S-15 + +### C-17: Script guidance treats every script as one fixed operation + +- **Claim:** Script guidance shows one script per step, invoked directly. Nothing covers a helper library in `scripts/` + that the model imports into a one-off script it writes. Script commands are kept out of `allowed-tools`, so a composed + script still prompts for permission. +- **Location:** `skill-building-guidance/hardening-fuzzy-vs-deterministic.md:32-41`; + `skill-building-guidance/script-execution-instructions.md` +- **Evidence:** `### Deterministic Steps` describes a script per step. The script-execution guide's "Why Scripts Should + Not Be in `allowed-tools`" section covers the permission prompt. +- **Raised by:** gap-analyzer GB2-6; a validator narrowed it (group 5). +- **Confidence:** Verified +- **Bears on:** S-16 + +### C-18: Hooks are documented as one-line table rows + +- **Claim:** The `hooks` field appears as a single row in the skill frontmatter table and in the plugin.json table. + `PreToolUse` appears nowhere in the plugin. +- **Location:** `skill-building-guidance/skill-frontmatter-fields.md:49`; + `claude-marketplace-and-plugin-configuration/plugin-json-options.md:49` +- **Evidence:** + ```markdown + | `hooks` | No | Lifecycle hooks scoped to this skill's run. + ``` +- **Raised by:** gap-analyzer GB2-7; a validator confirmed it (group 5). +- **Confidence:** Verified +- **Bears on:** S-17 + +### C-19: Neither builder skill reads or checks against the per-model guide + +- **Claim:** `skill-builder` and `agent-builder` read guidance through their own decision tables, not the guidance + router. Neither table nor either Step 6 review names `per-model-authoring.md`. +- **Location:** `skills/skill-builder/SKILL.md:36-54`, `:160-195`; `skills/agent-builder/SKILL.md:36-46`, `:157-187` +- **Evidence:** + ```markdown + ## Step 6: Full Guidance-Conformance Review + + This is the review pass the skill commits to. Re-read each governing document that applies to what you built and verify + the finished files against it, applying every fix directly. + ``` +- **Raised by:** behavioral-analyst B-1, B-2; a validator confirmed it (group 2). +- **Confidence:** Verified +- **Bears on:** S-18, D-18 + +### C-20: The builders' own instructions carry none of the patterns the sources say to remove + +- **Claim:** Neither builder contains a "think step by step" line, a request to reproduce reasoning, all-caps emphasis, + or a bare double-check step. Their Step 6 is a review scoped to named guidance files. Neither builder dispatches a + subagent, and each ends by asking whether to iterate. +- **Location:** `skills/skill-builder/SKILL.md`, `skills/agent-builder/SKILL.md`, `skills/guidance/SKILL.md` +- **Evidence:** behavioral-analyst B-11 to B-15 grepped for `think`, `step.by.step`, `reasoning`, `MUST|NEVER|ALWAYS`, + `verify|double-check`, and `Agent`/`Task` in `allowed-tools`, and found none of the patterns. +- **Raised by:** behavioral-analyst; gap-analyzer (slice C) "Confirmed no-gap findings". +- **Confidence:** Verified +- **Bears on:** no entry. This finding is why the builders' own bodies are not rewritten. + +### C-21: A new reference file needs a rule-index entry; the router finds it by listing the directory + +- **Claim:** The guidance router tells the model to list a subdirectory and read the file that applies, so a new file + under `skill-building-guidance/` is reachable with no router edit. The rule index lists every file by name, so a new + file needs one line there. `init-guidance.sh:74` copies `references/` wholesale. +- **Location:** `skills/guidance/SKILL.md:38-39`, `:52-56`; `skills/guidance/assets/rule-index-body.md:29-106`; + `skills/guidance/scripts/init-guidance.sh:74` +- **Evidence:** + ```markdown + 2. List the relevant subdirectory under `${CLAUDE_SKILL_DIR}/references/` to see the available documents, using the map + above. + ``` +- **Raised by:** structural-analyst S-9; this run's own sweep. +- **Confidence:** Verified +- **Bears on:** S-15 + +### C-22: The plugin's docs and README do not restate per-model content + +- **Claim:** `README.md` and `docs/skills/{guidance,skill-builder,agent-builder}.md` link to the per-model guide without + naming the model list. They need no edit for the model-list change. +- **Location:** `han-plugin-builder/README.md`; `han-plugin-builder/docs/skills/*.md` +- **Evidence:** structural-analyst S-8 grepped for "Sonnet 5", "Opus 5", and "Fable 5" and found no hits. +- **Raised by:** structural-analyst +- **Confidence:** Verified +- **Bears on:** S-18 (the builder docs describe Step 6 and may need one line; see the plan) + +## Findings No Agent Could Audit + +- **The live source pages.** The validators worked from this run's captured text. They could not refetch the four + sources to confirm that text matches the live pages. The captures were taken on 2026-09-23 with `curl`. +- **Whether authors hit these failures in practice.** No usage logs or session transcripts exist to show whether an + author has shipped a skill that stopped early or hit a `reasoning_extraction` refusal. Every entry rests on the sources' + own claims about model behavior. diff --git a/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/scope-boundary.md b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/scope-boundary.md new file mode 100644 index 00000000..4863b8c8 --- /dev/null +++ b/docs/plans/opus-5-5-plugin-builder-guidance/artifacts/scope-boundary.md @@ -0,0 +1,47 @@ +# Scope Boundary: Opus 5.5 plugin-builder guidance update + +## Work Item + +No work item exists. The operator's request, typed when invoking `/han-planning:plan-a-change` on 2026-09-23, and one +follow-up message are the only boundary this run has. + +## Stated Scope + +> evaluate the han-plugin-builder plugin, all of it's files, skills, guidance, docs, and everything else in it, against +> https://claude.dev/blog/getting-the-most-out-of-opus-5-5/ and give me a list of what needs to be updated. be sure to +> run adversarial validation against every suggestion to ensure it's applicable and not already covered. if there are +> changes required, make a branch and then commit the plan to the new branch. i'll determine when to start the work +> after that, if needed + +## Stated Exclusions + +None stated. + +## Operator-Stated Scope + +Follow-up message: + +> also include https://claude.dev/blog/lessons-from-building-claude-code-how-we-use-skills/ and related sources in this +> eval as a baseline, but letting the article on getting the most out of opus 5.5 override anything from this one, when +> it conflicts + +Confirmation turn answers ("yes to both questions"): + +1. The area is `han-plugin-builder/` in full. Repo-level surfaces that point into it (`docs/skills/README.md`, the + repo's own `.claude/` copies of the plugin-building skills) are out of scope except where a plugin change would make + one of them wrong. +2. The plan lives at `docs/plans/opus-5-5-plugin-builder-guidance/`, on a branch of the same name. + +Implied by the request: this run produces a plan only. No plugin file is edited. + +## Direction of Travel + +Not asked; the request does not name anything being deprecated or migrated away from. `Unanswered`. + +## Visual Material Received + +None received + +## Record Provenance + +Established by `plan-a-change` on 2026-09-23. No prior record was found. diff --git a/docs/plans/opus-5-5-plugin-builder-guidance/change-plan.md b/docs/plans/opus-5-5-plugin-builder-guidance/change-plan.md new file mode 100644 index 00000000..d6423f8b --- /dev/null +++ b/docs/plans/opus-5-5-plugin-builder-guidance/change-plan.md @@ -0,0 +1,562 @@ +# Change Plan: Opus 5.5 plugin-builder guidance update + +## Why This Change + +Anthropic shipped Opus 5.5, and `han-plugin-builder` still teaches authors to write for Sonnet 5, Opus 5, and Fable 5, +as checked on 2026-07-31. This is a **constraint arriving**: a new model whose behavior differs in ways that change what +a skill or agent author should write. + +The operator asked to evaluate the whole plugin against +[Getting the most out of Opus 5.5](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/). The operator also named +[How we use skills](https://claude.dev/blog/lessons-from-building-claude-code-how-we-use-skills/) and related sources as +a baseline, with the Opus 5.5 article winning on conflict ([D-19](artifacts/change-decision-log.md#d-19-the-source-set-and-its-precedence)). + +The plan cites four sources by number. SOURCE 1 is the Opus 5.5 article and SOURCE 2 is the skills article. SOURCE 3 is +[The new rules of context engineering for Claude 5 generation models](https://claude.dev/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models/), +linked as related from both. SOURCE 4 is Anthropic's +[Prompting Claude Opus 5.5](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5) +page. The findings file's Provenance section lists all four with dates. + +Every suggestion went through adversarial validation before it could enter this plan: 28 candidates went in, and 18 +entries came out ([D-20](artifacts/change-decision-log.md#d-20-candidates-rejected-by-adversarial-validation)). + +## What Changes, In One Paragraph + +After this change, the plugin's guidance names Opus 5.5 and says what an author does differently for it. Opus 5.5 can +refuse an instruction to show its reasoning. It cannot turn thinking off. Its effort levels mean more thinking than the +same names did on Opus 5, so a pinned effort needs re-testing. Everything else from Opus 5 carries over +([D-1](artifacts/change-decision-log.md#d-1-add-opus-55-to-the-per-model-guide-and-every-file-that-restates-its-model-list)). The workflow and delegation guides gain the patterns the new model's long +runs call for. They name the stops an autonomous stretch should and should not make, state a finish line and keep a +task file for unbounded steps, check each fanned-out subagent's evidence, and wait for background work. None of it +weakens Han's deliberate human checkpoints. The skill-building guides also pick up five patterns from the baseline +articles that the plugin lacked: gotchas, verification skills, keeping skill state across plugin updates, +helper-script libraries, and a skill-scoped guardrail hook. Finally, `skill-builder` and `agent-builder` start reading +and checking against the per-model guide, so the update reaches people who build through them. + +## Current State + +The per-model guide names three models, is dated 2026-07-31, and has four independent copies of its model list: the +router, the portable router, the rule index, and the specialization guide +([C-1](artifacts/current-state-findings.md#c-1-the-per-model-guide-names-three-models-and-was-last-checked-before-opus-55-shipped), +[C-2](artifacts/current-state-findings.md#c-2-the-model-list-is-restated-in-four-other-files-and-a-change-must-reach-all-of-them)). + +Its reasoning-echo warning covers only Fable 5 +([C-3](artifacts/current-state-findings.md#c-3-the-reasoning-echo-warning-is-scoped-to-fable-5-and-called-single-source)). + +Its thinking and effort bullets describe Opus 5 only +([C-4](artifacts/current-state-findings.md#c-4-the-thinking-mode-bullet-describes-disabling-thinking-on-opus-5-which-opus-55-does-not-allow), +[C-5](artifacts/current-state-findings.md#c-5-the-effort-bullet-predates-opus-55s-default-and-authors-can-pin-effort-in-frontmatter)). + +It warns against every limiting phrase in a review instruction, including the concrete one the Opus 5.5 article +recommends +([C-6](artifacts/current-state-findings.md#c-6-the-guide-warns-against-every-only-report-x-review-limit-while-source-1-recommends-one-with-a-concrete-bar)). + +The workflow and delegation guides cover deliberate human gates but not a model that stops early to report. They say +nothing about finish lines or task files for unbounded steps, about checking fanned-out evidence, or about waiting for +background dispatches ([C-8](artifacts/current-state-findings.md#c-8-workflow-guidance-covers-deliberate-pauses-not-a-model-that-stops-early-to-report), [C-9](artifacts/current-state-findings.md#c-9-no-guidance-names-a-finish-line-or-a-durable-task-list-for-long-steps), +[C-10](artifacts/current-state-findings.md#c-10-delegation-guidance-has-no-evidence-check-on-returned-subagent-work-and-no-wait-for-background-dispatches)). + +Neither builder skill reads the per-model guide +([C-19](artifacts/current-state-findings.md#c-19-neither-builder-skill-reads-or-checks-against-the-per-model-guide)). + +The builders' own instructions already avoid every pattern the sources say to remove, so their bodies need no rewrite +([C-20](artifacts/current-state-findings.md#c-20-the-builders-own-instructions-carry-none-of-the-patterns-the-sources-say-to-remove)). + +No automated test covers guidance content (see the findings file's Gaps), so every check in this plan is a grep or a +read. + +## Target State + +The target state is placement of guidance text in existing files. No new file is added +([D-15](artifacts/change-decision-log.md#d-15-keep-skill-state-in-the-plugin-data-directory-documented-as-a-section-not-a-new-file)). + +The adversarial validators set each entry's landing file, and the structural map shows which files must change +together. No separate architect pass ran +([D-21](artifacts/change-decision-log.md#d-21-the-target-state-came-from-the-validators-landing-recommendations-not-a-separate-architect-pass)). + +- **`per-model-authoring.md`** owns every model-specific fact, now including Opus 5.5. It does not own the long-run + patterns; it links to them. +- **`workflow-patterns.md`** owns the stop-naming pattern and the finish-line and task-file pattern for unbounded steps. + Both are model-agnostic. +- **`multi-agent-economics.md`** owns the fan-out evidence check and the background-wait rule. +- **`plugin-json-options.md`** owns where a plugin-shipped skill keeps state that must survive an update. +- **The two builder skills** route to `per-model-authoring.md` and check for model-specific leftovers. + +Three contracts must stay in agreement across files. + +**1. The model list.** `per-model-authoring.md:5` is the source. The router line in `guidance/SKILL.md` and +`guidance/assets/guidance-portable-SKILL.md` must stay byte-identical, and after the change reads: + +```markdown +- Writing the instructions for a target model (how Sonnet 5, Opus 5, Opus 5.5, and Fable 5 differ in following + instructions, which instructions to leave out, and how to calibrate length, narration, and scope) → + `${CLAUDE_SKILL_DIR}/references/per-model-authoring.md`. +``` + +`specialization-and-model-selection.md:89` names the same four models: "Sonnet 5, Opus 5, Opus 5.5, or Fable 5". + +**2. The rule-index descriptions.** `guidance/assets/rule-index-body.md` is vendored into every consuming repo and is +how the model decides which file to read. Every file whose scope this plan widens gets its description updated +([D-22](artifacts/change-decision-log.md#d-22-pin-every-widened-files-rule-index-description)). Keep each entry's link and wrap lines at 120 characters. The new +description text: + +| Entry | Description after the change | +| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Per-Model Authoring Guidance | How Sonnet 5, Opus 5, Opus 5.5, and Fable 5 differ in how they follow instructions, and how those differences change what you write: the model-agnostic default for an unknown target, the opposite-direction instruction-style split, the verification and re-check instructions to leave out, how to calibrate response length, narration, and scope, effort and thinking on Opus 5.5, and the reasoning-echo refusal on Fable 5 and Opus 5.5. Read when writing or hardening a skill or agent and tuning the instructions to a target model, not when choosing which model tier to run (see Specialization and Model Selection). | +| Workflow Patterns | Four structural patterns for organizing the steps inside a single skill, mapped to Anthropic's effective-agent patterns, plus how to name the stops an autonomous stretch should and should not make and how to bound a step that works through an unknown number of items. Read when designing or restructuring a skill's internal workflow, or when a skill stops early or loses track mid-run. | +| Multi-Agent Economics | The escalation cascade for deciding whether adding more agents is justified, given that each agent multiplies latency and token cost, plus the delegation policy a skill should state because the model's own default is eager, the evidence check a skill runs on each fanned-out subagent's result, and the rule against finishing while a background dispatch is still running. Read when a skill is considering dispatching multiple or parallel agents. | +| Graceful Degradation (agents) | How a dispatched agent should check tool availability inline and skip gracefully, so the orchestrating skill needs no defensive guards around the dispatch, and how a research or analysis agent marks a claim it could not confirm and says where it looked. Read when an agent's steps depend on git or other tools that may be missing, or when an agent reports findings. | +| Success Criteria and Testing | Three test types (triggering, functional, outcome) for knowing a skill works, plus the rule to test on the model tier the skill targets, and how to build a skill whose job is verifying a running product. Read when validating a skill before shipping it, or when building a verification skill. | +| Writing Effective Instructions | How to write the SKILL.md body so steps are specific, actionable, and reliably followed across sessions, and where to stop specifying. Read when a skill behaves inconsistently, skips steps, or improvises when it should follow a fixed process, or when its rules are over-constraining it. | +| Skill Reference Files | When and how to extract domain knowledge (templates, checklists, rate tables, gotchas) into a `references/` subdirectory loaded on demand. Read when a skill carries content that is knowledge rather than process steps. | +| Hardening: Fuzzy vs. Deterministic | The framework for classifying each skill step as fuzzy (keep as an LLM instruction) or deterministic (extract to a script), including a helper library the model composes into one-off scripts. Read when hardening a skill for reliability or deciding what to script. | +| plugin.json Schema Reference | Full schema for `.claude-plugin/plugin.json`: required fields, metadata, component paths, dependencies, and experimental keys, plus where a plugin-shipped skill keeps state that must survive an update. Read when creating or editing a plugin manifest, or when a skill saves first-run answers or run history. | + +**3. The review-limit test.** S-5 and S-18 must agree on what separates a vague limit from an acceptable one +([D-5](artifacts/change-decision-log.md#d-5-keep-the-limiting-phrase-warning-and-separate-vague-limits-from-a-concrete-bar)). A limit is acceptable when it names a bar the reader could check (a +consequence such as "would block the merge", or a rubric the skill itself carries) and requires evidence for each item. +The two examples both files use: + +```text +Acceptable: List only problems you'd block the merge for. For each one, give the file and line, why it's wrong, and how +to show it fails. + +Vague: Only report high-severity issues. +``` + +## Surface Delta + +Every path below is under `han-plugin-builder/skills/` unless it says otherwise. Entries S-1 to S-10 and S-18 are +driven by the Opus 5.5 sources. Entries S-11 to S-17 are baseline gaps from the two earlier articles +([D-19](artifacts/change-decision-log.md#d-19-the-source-set-and-its-precedence)). + +### S-1: `per-model-authoring.md` model list and its four mirrors — Re-scoped + +**Target state.** `per-model-authoring.md` covers Sonnet 5, Opus 5, Opus 5.5, and Fable 5, with a "last checked" stamp +dated to the build. It states that Opus 5.5 inherits the Opus 5 guidance except where the guide says otherwise. The +headings "Instructions to leave out on Opus 5" and "Calibrating length, narration, and scope on Opus 5" name Opus 5 and +Opus 5.5. The router line, the portable router line, the rule-index entry, and `specialization-and-model-selection.md:89` +match contracts 1 and 2 in Target State. + +**Behavior.** Preserving. Every existing recommendation stands, and the router sends the same questions to the same +file. + +**Why.** The guide and its copies predate Opus 5.5 ([C-1](artifacts/current-state-findings.md#c-1-the-per-model-guide-names-three-models-and-was-last-checked-before-opus-55-shipped), +[C-2](artifacts/current-state-findings.md#c-2-the-model-list-is-restated-in-four-other-files-and-a-change-must-reach-all-of-them)). + +**Decision.** [D-1](artifacts/change-decision-log.md#d-1-add-opus-55-to-the-per-model-guide-and-every-file-that-restates-its-model-list) + +### S-2: `per-model-authoring.md` reasoning-echo section — Re-scoped + +**Target state.** The section, headed to name both Fable 5 and Opus 5.5, covers both models and names the +`reasoning_extraction` refusal category. Its evidence note reads as single-vendor across three Anthropic publications, +not as uncorroborated. + +**Behavior.** Preserving. The test for spotting the pattern is unchanged. + +**Why.** Both Opus 5.5 sources place the refusal on 5.5 ([C-3](artifacts/current-state-findings.md#c-3-the-reasoning-echo-warning-is-scoped-to-fable-5-and-called-single-source)). + +**Depends on.** S-1. + +**Decision.** [D-2](artifacts/change-decision-log.md#d-2-the-reasoning-echo-warning-covers-opus-55-and-names-the-refusal-category) + +### S-3: `per-model-authoring.md` thinking-mode bullet — Re-scoped + +**Target state.** The bullet says Opus 5.5 always thinks and cannot disable it. The XML-tag-leak caveat is stated as an +Opus 5 thinking-disabled artifact that does not arise on 5.5. + +**Behavior.** Preserving. + +**Why.** [C-4](artifacts/current-state-findings.md#c-4-the-thinking-mode-bullet-describes-disabling-thinking-on-opus-5-which-opus-55-does-not-allow). + +**Depends on.** S-1. + +**Decision.** [D-3](artifacts/change-decision-log.md#d-3-one-clause-on-opus-55-thinking-not-a-new-section) + +### S-4: `per-model-authoring.md` effort bullet, with pointers from both frontmatter tables — Re-scoped + +**Target state.** The effort bullet says the same effort name means more thinking on Opus 5.5 than on Opus 5. It says +Anthropic's platform docs put Opus 5.5's API default at `medium`, where Opus 5's is `high`, and report `medium` on 5.5 +matching or beating Opus 5 at `high`. It says `xhigh` and `max` are for measured gains, and that a pinned `effort` tuned +for Opus 5 should be re-tested rather than carried over. The `effort` rows in +`skill-building-guidance/skill-frontmatter-fields.md` and `agent-building-guidelines/agent-external-files.md` each link +to the section "Other settings the model differences affect" in `per-model-authoring.md`. + +**Behavior.** Preserving. Additive. + +**Why.** Authors can pin effort, and what an effort name means changed ([C-5](artifacts/current-state-findings.md#c-5-the-effort-bullet-predates-opus-55s-default-and-authors-can-pin-effort-in-frontmatter)). + +**Depends on.** S-1. + +**Decision.** [D-4](artifacts/change-decision-log.md#d-4-tell-authors-to-re-test-a-pinned-effort-value-on-opus-55) + +### S-5: `per-model-authoring.md` limiting-phrase paragraph — Re-scoped + +**Target state.** The paragraph still warns that a vague limit ("only report high-severity issues", "be conservative") +is followed literally and under-reports. It still recommends reporting everything and filtering in a separate step for +vague limits. It adds the test from contract 3 and quotes the acceptable example whole, evidence clause included. + +**Behavior.** Changing. Readers learn one acceptable shape of limiting phrase. Settled by the operator (option A). + +**Why.** The primary source recommends a limit of that shape, and no source says 5.5 is less literal +([C-6](artifacts/current-state-findings.md#c-6-the-guide-warns-against-every-only-report-x-review-limit-while-source-1-recommends-one-with-a-concrete-bar)). + +**Depends on.** S-1. + +**Decision.** [D-5](artifacts/change-decision-log.md#d-5-keep-the-limiting-phrase-warning-and-separate-vague-limits-from-a-concrete-bar) + +### S-6: `per-model-authoring.md` Sources section — Re-scoped + +**Target state.** Sources lists the Opus 5.5 prompting page as Anthropic docs and the Opus 5.5 article as an Anthropic +blog post with author and date. The "postdates that research report" sentence covers the Opus 5.5 material. +`docs/research/model-specific-guidance-for-skills.md` is unchanged. + +**Behavior.** Preserving. + +**Why.** [C-7](artifacts/current-state-findings.md#c-7-the-per-model-sources-section-cites-only-the-opus-5-era-pages). + +**Depends on.** S-2 to S-5. + +**Decision.** [D-6](artifacts/change-decision-log.md#d-6-add-the-opus-55-sources-labeled-by-tier-without-touching-the-research-report) + +### S-7: `workflow-patterns.md` stop-naming subsection, plus one sentence in `per-model-authoring.md` — Added + +**Target state.** A subsection after "Human Gates in Workflow Steps", listed in the file's Contents, covers a stretch of +a skill meant to run without the user. It names the unwanted stops: a summary that announces the next step without +taking it, an offer to continue, and a list of choices that block nothing. It names the wanted stops: no work can move +without the user, a destructive or irreversible action is next, or a human gate is reached. It states that human +gates, interview turns, and collaborative-stop boundaries are wanted stops, and that a skill built to hand control back +carries no keep-going instruction. It links to `han-core/references/collaborative-stop-rule.md` for that claim. +`per-model-authoring.md` gains one sentence naming Opus 5.5's early-stop behavior and linking to the subsection. + +**Behavior.** Preserving. No existing gate or stop rule changes. + +**Why.** [C-8](artifacts/current-state-findings.md#c-8-workflow-guidance-covers-deliberate-pauses-not-a-model-that-stops-early-to-report). + +**Depends on.** S-1, for the per-model sentence. + +**Decision.** [D-7](artifacts/change-decision-log.md#d-7-name-the-stops-a-long-autonomous-stretch-should-and-should-not-make-and-protect-the-collaborative-stops) + +### S-8: `workflow-patterns.md` finish-line and task-file subsection for unbounded steps — Added + +**Target state.** A subsection, listed in Contents, covers steps that work through a set whose size is not known up +front. Such a step states its completion condition in checkable terms, keeps its checklist in a file it updates as items +finish, and reads that file to decide what remains. It cross-references `skill-composition.md`'s continuation rule and +`multi-agent-economics.md`, and says a fixed flowchart step needs neither. + +**Behavior.** Preserving. Additive. + +**Why.** [C-9](artifacts/current-state-findings.md#c-9-no-guidance-names-a-finish-line-or-a-durable-task-list-for-long-steps). + +**Depends on.** S-7, which sits beside it and supplies the stop vocabulary. + +**Decision.** [D-8](artifacts/change-decision-log.md#d-8-state-the-finish-line-and-keep-the-task-list-in-a-file-only-for-unbounded-steps) + +### S-9: `multi-agent-economics.md` fan-out and background rules, plus a pointer in `agent-external-files.md` — Re-scoped + +**Target state.** `multi-agent-economics.md` carries a fan-out rule. A skill that gives each unit of work to its own +subagent checks the evidence each subagent cites against its source before accepting the finding. It then produces one +consolidated result. The rule states the line against the existing self-check ban: checking a subagent's cited evidence +is allowed; dispatching another agent to redo the skill's own reasoning is not. "Practical Implications for Skills" says +a skill does not treat its run as complete while a `run_in_background: true` dispatch it launched is still running, and +labels that item single-source. The `background` row in `agent-external-files.md` links to that item. + +**Behavior.** Preserving. Additive. + +**Why.** [C-10](artifacts/current-state-findings.md#c-10-delegation-guidance-has-no-evidence-check-on-returned-subagent-work-and-no-wait-for-background-dispatches). + +**Decision.** [D-9](artifacts/change-decision-log.md#d-9-check-fanned-out-evidence-and-wait-for-background-dispatches) + +### S-10: `agent-building-guidelines/graceful-degradation.md` unconfirmed-claim rule, plus `agent-builder` Step 6 item 8 — Re-scoped + +**Target state.** The agent graceful-degradation guide has two rules. The first: skip the step and note the limitation +when a tool is missing. The second: a research or analysis agent marks a claim it could not confirm and names where it +looked. The `agent-builder` Step 6 item 8 summary names both. + +**Behavior.** Changing, for `agent-builder` users: the final review checks one more condition. The guide itself only +gains a rule. See Behavior Changes. + +**Why.** [C-11](artifacts/current-state-findings.md#c-11-agent-graceful-degradation-covers-a-missing-tool-not-a-claim-the-agent-could-not-confirm). + +**Decision.** [D-10](artifacts/change-decision-log.md#d-10-agents-say-what-they-could-not-confirm-and-where-they-looked) + +### S-11: `writing-effective-instructions.md` limit on specificity — Added + +**Target state.** A rule beside "Be specific and actionable" says to spell out conventions that break something when +missed. It also says to leave out instructions for routine details the model gets right unprompted, rather than +writing blanket rules for them. The rule governs how much detail an instruction carries, not whether a step is fixed. +It links to `plugin-entity-taxonomy.md`'s flowchart test and to the per-model middle path rather than restating either. + +**Behavior.** Changing. The specificity rule now carries a stated limit. Settled by the operator (option A). + +**Why.** [C-12](artifacts/current-state-findings.md#c-12-the-instruction-writing-guide-pushes-toward-specificity-with-no-stated-limit). + +**Decision.** [D-11](artifacts/change-decision-log.md#d-11-add-a-limit-to-be-specific) + +### S-12: `progressive-disclosure.md` "When to remove entirely" — Re-scoped + +**Target state.** The list names a second removable kind: instructions restating what the model already does by +default, such as "read the code before changing it." It links to `context-hygiene.md`'s "every token must earn its +place" rule. + +**Behavior.** Changing, for `skill-builder` users: its Step 6 review reads this list, so it will now cut such +instructions from the skills it builds. See Behavior Changes. + +**Why.** [C-13](artifacts/current-state-findings.md#c-13-when-to-remove-entirely-names-only-toolchain-enforced-rules). + +**Decision.** [D-12](artifacts/change-decision-log.md#d-12-name-restates-a-default-as-removable-in-the-list-the-builder-review-reads) + +### S-13: Gotchas category in `skill-reference-files.md`, named in `progressive-disclosure.md` — Re-scoped + +**Target state.** `skill-reference-files.md`'s category list defines gotchas: failure points found in use that the +skill tells the model to avoid. The definition says gotchas grow over the skill's life and differ from a checklist +(what to verify) and a canonical example (what a convention looks like). `progressive-disclosure.md`'s extraction list +names gotchas and links to that definition. + +**Behavior.** Preserving. Additive. + +**Why.** [C-14](artifacts/current-state-findings.md#c-14-no-reference-file-category-holds-accumulated-gotchas). + +**Decision.** [D-13](artifacts/change-decision-log.md#trivial-decisions) + +### S-14: `success-criteria-and-testing.md` verification-skill subsection — Added + +**Target state.** A subsection covers building a skill whose job is to verify a running product, as distinct from +testing a skill. The skill pairs with an external driver such as browser automation or a terminal tool, asserts state +at each step rather than once at the end, and can record the run so a person sees what was tested. + +**Behavior.** Preserving. Additive. + +**Why.** [C-15](artifacts/current-state-findings.md#c-15-testing-guidance-covers-testing-a-skill-not-building-a-skill-that-verifies-a-product). + +**Decision.** [D-14](artifacts/change-decision-log.md#trivial-decisions) + +### S-15: `plugin-json-options.md` section on keeping skill state across updates — Added + +**Target state.** A section beside the environment-variable table covers two uses of `${CLAUDE_PLUGIN_DATA}` for a +skill shipped in a plugin. First-run setup: read a saved answer, ask with `AskUserQuestion` when it is absent, and +save it. Run memory: an append-only log or JSON file read at the next run. The section says: + +- The skill's own directory is under `${CLAUDE_PLUGIN_ROOT}` and is replaced on update, so nothing is written there. +- `userConfig` is the choice when the value is known at install time. +- A file the user is expected to edit by hand belongs where the user edits it, not in the data directory. +- A skill checked into a repo's `.claude/skills/`, including one vendored by `/guidance init`, has no plugin data + directory. It keeps its state in the repository or asks each run. + +`allowed-tools-AskUserQuestion.md` links to the section. + +**Behavior.** Preserving. Additive. + +**Why.** [C-16](artifacts/current-state-findings.md#c-16-the-persistent-data-directory-is-documented-only-as-a-pluginjson-variable). + +**Decision.** [D-15](artifacts/change-decision-log.md#d-15-keep-skill-state-in-the-plugin-data-directory-documented-as-a-section-not-a-new-file) + +### S-16: `hardening-fuzzy-vs-deterministic.md` helper-library shape — Re-scoped + +**Target state.** Deterministic Steps says `scripts/` can hold a library of helper functions, with their gotchas in the +functions' comments. The model loads that library into a one-off script it writes for the question at hand. It notes +that each composed script still prompts for permission, linking to `script-execution-instructions.md`. + +**Behavior.** Preserving. Additive. + +**Why.** [C-17](artifacts/current-state-findings.md#c-17-script-guidance-treats-every-script-as-one-fixed-operation). + +**Decision.** [D-16](artifacts/change-decision-log.md#trivial-decisions) + +### S-17: `skill-frontmatter-fields.md` `hooks` row — Re-scoped + +**Target state.** The `hooks` row adds one sentence: a hook here lasts only for the skill's run, which suits a guardrail +you want only while that skill is active, such as blocking destructive shell commands. The row links to the official +hooks reference for the schema. + +**Behavior.** Preserving. Additive. + +**Why.** [C-18](artifacts/current-state-findings.md#c-18-hooks-are-documented-as-one-line-table-rows). + +**Decision.** [D-17](artifacts/change-decision-log.md#d-17-one-sentence-on-skill-scoped-guardrail-hooks) + +### S-18: `skill-builder` and `agent-builder` per-model routing and review item — Re-scoped + +**Target state.** Each builder's decision table routes "instructions for a named target model" to +`per-model-authoring.md`. Each builder's Step 6 review checks that the finished files carry no model-specific leftovers: + +- no "think step by step" or "think carefully" line; +- no instruction to reproduce reasoning in the reply; +- no pinned `effort` without a reason stated in the skill or agent body; +- no vague limiting phrase in a review step, judged by contract 3 with both of its examples quoted. + +**Behavior.** Changing. Builder runs read one more file when a target model comes up, and the review checks one more +item. Settled by the operator (option A). + +**Why.** [C-19](artifacts/current-state-findings.md#c-19-neither-builder-skill-reads-or-checks-against-the-per-model-guide). + +**Depends on.** S-1 to S-5, the content the builders route to. + +**Decision.** [D-18](artifacts/change-decision-log.md#d-18-the-builders-read-the-per-model-guide-when-a-target-model-is-named-and-always-check-for-leftovers) + +## Behavior Changes + +Five entries change what a reader or a builder user experiences. + +- **Review-limit advice (S-5).** An author reading the per-model guide used to be told to avoid every "only report X" + instruction. After the change, a limit that names a checkable bar and demands evidence per item is acceptable. Vague + limits are still warned against. Operator, 2026-09-23: "go with recommendation for this one." +- **Specificity advice (S-11).** An author reading the instruction-writing guide used to see "be specific" with no + limit. After the change, they are told to spell out conventions that break something when missed and to leave out + instructions for details the model gets right unprompted. Operator, 2026-09-23: "go with recommendation." +- **Builder runs (S-18).** For someone building a skill or agent, naming a target model now makes the builder read the + per-model guide. Their final review also flags leftovers such as a "think step by step" line or a pinned effort with + no stated reason. Operator, 2026-09-23: "go with recommendation." +- **The agent-builder review (S-10) and the skill-builder review (S-12).** Both builders' final reviews re-read + guidance files this plan changes. After the change, `agent-builder` checks that a research agent marks claims it could + not confirm, and `skill-builder` cuts instructions that restate what the model already does by default. Operator, 2026-09-23: "option A" ([D-23](artifacts/change-decision-log.md#d-23-builder-reviews-that-pick-up-s-10-and-s-12)). + +## Change Units + +Each unit leaves every file consistent with the others and passes `npm run lint`. + +### Unit 1: Per-model facts + +**What it does.** Brings the per-model guide and its four mirrors up to Opus 5.5. + +**Delta entries.** S-1, S-2, S-3, S-4, S-5, S-6, and the Per-Model Authoring row of contract 2. + +**How you know it worked.** No file still carries the three-model list, across line wraps: + +```bash +perl -0ne 'print "$ARGV\n" if /Sonnet 5,\s+Opus 5,?\s+(and|or)\s+Fable 5/' $(grep -rl "Fable 5" han-plugin-builder) +``` + +prints nothing. The router line matches between `guidance/SKILL.md` and `guidance/assets/guidance-portable-SKILL.md` +(`diff` on the extracted line). `grep -rln "Opus 5.5" han-plugin-builder` lists `per-model-authoring.md`, both +routers, `rule-index-body.md`, and `specialization-and-model-selection.md`. `npm run lint` passes. + +### Unit 2: Long runs and delegation + +**What it does.** Adds the stop-naming, finish-line, fan-out, background-wait, and unconfirmed-claim patterns, with +their rule-index rows from contract 2. + +**Delta entries.** S-7, S-8, S-9, S-10. + +**Ordering constraint.** After Unit 1, because S-7's per-model sentence lands in text Unit 1 rewrites. + +**How you know it worked.** `grep -n collaborative-stop-rule han-plugin-builder/skills/guidance/references/skill-building-guidance/workflow-patterns.md` +hits the new subsection. `git diff --stat main -- han-core han-coding han-planning` is empty, so no collaborative skill +changed. Both new subsections appear in the file's Contents. Every new cross-reference resolves to an existing +heading. `npm run lint` passes. + +### Unit 3: Instruction style and reference-file categories + +**What it does.** Adds the specificity limit, the "already does by default" removal bullet, and the gotchas category, +with their rule-index rows from contract 2. + +**Delta entries.** S-11, S-12, S-13. + +**How you know it worked.** The S-11 rule links to `plugin-entity-taxonomy.md`, and it does not use the word +"judgment", which the taxonomy reserves for agents. `grep -rn -i gotcha han-plugin-builder` hits the definition in +`skill-reference-files.md` and the link in `progressive-disclosure.md`. `npm run lint` passes. + +### Unit 4: Skill patterns from the baseline + +**What it does.** Adds verification skills, keeping state across updates, helper libraries, and the skill-scoped +guardrail note, with their rule-index rows from contract 2. + +**Delta entries.** S-14, S-15, S-16, S-17. + +**How you know it worked.** Each rule-index row in contract 2 matches the file. `npm run lint` passes. + +### Unit 5: Builder routing + +**What it does.** Points both builders at the per-model guide and adds the leftover check. + +**Delta entries.** S-18. + +**Ordering constraint.** After Unit 1, so the builders route to updated content, and after Unit 2, so `agent-builder`'s +Step 6 carries S-10's item 8 change alongside the new item. + +**How you know it worked.** `grep -n per-model-authoring han-plugin-builder/skills/*-builder/SKILL.md` hits the decision +table and Step 6 in each. Both examples from contract 3 appear in each Step 6. `npm run lint` passes. + +After Unit 5, run `/han-update-documentation` on the branch. The plugin's long-form docs describe the builders' review +in general terms (`docs/skills/skill-builder.md:32-34`, `docs/skills/agent-builder.md:30-33`) and stay accurate, so +that pass is a safety check, not a planned edit. + +The release version bump for `han-plugin-builder` belongs to `/han-release`, not to this plan. + +## Risks + +- **The stop-naming guidance gets pasted into a collaborative skill.** Six skills across three plugins rely on + `collaborative-stop-rule.md`, and a keep-going instruction in any of them would silently remove a human checkpoint. + S-7 names those stops as wanted in its own text. Unit 2 checks that the subsection links the rule and that no + collaborative skill changed. Nothing guards a later edit. +- **The specificity limit reads as permission to turn a skill's steps into judgment calls.** The guide is read on + nearly every skill-authoring task. S-11 limits the rule to instruction detail, avoids the word the taxonomy reserves + for agents, and links the flowchart test. A reviewer still reads the wording at Unit 3. +- **The builder review over-flags or under-flags.** S-18 runs on the next skill or agent anyone builds. A false positive + costs a rework. A false negative ships a leftover the review was meant to catch. Contract 3's two examples are the + guard. +- **A mirror or index description gets missed.** The model list lives in five files, and nine rule-index descriptions + change. Unit 1's multi-line check covers the model list. The index rows are checked by reading each against contract 2. +- **Consuming repos keep the old text.** Repos that ran `/guidance init` see the change only after `/guidance update`. + Name that in the release notes. +- **The sources change.** The article and the platform page are snapshots from 2026-09-23. The per-model guide's + "last checked" stamp is how a reader knows. + +## Deferred (YAGNI) + +- **Designing expressive script and tool interfaces instead of usage examples (K-15).** SOURCE 3 says examples + constrain newer models, but the claim is about tool definitions, not the convention examples the plugin teaches. SOURCE + 1 and SOURCE 4 both teach through concrete examples. Reopen if a Han skill's scripts grow parameters the model misuses, + or if an Anthropic page extends the claim to skill instructions. +- **A hooks guide, and the usage-logging hook pattern.** No Han plugin or skill ships a hook today. S-17 keeps one + sentence on the skill-scoped guardrail. Reopen when a Han skill ships a hook, or when an author asks how to measure how + often a skill is used. +- **A dedicated file for skill state.** S-15 lands as a section instead. Reopen if the section outgrows its host file, + or when a Han skill starts saving state. + +## Cut for Scope + +- **Re-checking the Fable 5 guidance against Fable 5.1.** Anthropic publishes a separate Fable 5.1 prompting page. None + of the four sources describes Fable 5.1's instruction style, and the operator's request is an evaluation against the + Opus 5.5 article and its related sources (`artifacts/scope-boundary.md`, Stated Scope). Cutting it means the guide + keeps describing Fable 5, not Fable 5.1, after this change. The operator can reinstate it. +- **Making `agent-builder` cite `writing-effective-instructions.md`.** The behavioral trace found that `agent-builder` + never cites the instruction-writing guide. The gap is real, but no source drives it, and the request is an evaluation + against the sources. Cutting it means agent authors using the builder still are not pointed at the general + instruction-writing rules. The operator can reinstate it. + +## Open Items + +- **Whether writing to `${CLAUDE_PLUGIN_DATA}` prompts for permission.** Non-blocking. S-15 should tell the author what + to expect on the first save. The build confirms it against the Claude Code permissions docs before writing that + sentence, and leaves the sentence out if the docs do not say. + +## Review Findings + +Two specialists reviewed the draft: `han-core:junior-developer` (16 findings, none blocking) and `han-core:risk-analyst` +(no critical or high risks; four medium, one low, one YAGNI candidate). Findings that changed the plan, merged where +both raised the same thing: + +- **The new state file was more structure than the reason justified** (junior-developer JD-1, JD-2; risk-analyst R5). + S-15 became a section in `plugin-json-options.md`, and it now covers vendored skills and user-edited config + ([D-15](artifacts/change-decision-log.md#d-15-keep-skill-state-in-the-plugin-data-directory-documented-as-a-section-not-a-new-file)). +- **Rule-index descriptions for widened files would go stale** (JD-3, JD-11). Pinned as contract 2 + ([D-22](artifacts/change-decision-log.md#d-22-pin-every-widened-files-rule-index-description)). +- **S-12 changes what `skill-builder` produces** (JD-4). Reclassified as Changing and sent to the operator with S-10 + ([D-23](artifacts/change-decision-log.md#d-23-builder-reviews-that-pick-up-s-10-and-s-12)). +- **The hook entry was the same evidence gap as the deferred hooks guide** (JD-5, R6). S-17 narrowed to one sentence + ([D-17](artifacts/change-decision-log.md#d-17-one-sentence-on-skill-scoped-guardrail-hooks)). +- **The review-limit test and the builder check could drift apart** (JD-6). Pinned as contract 3 + ([D-5](artifacts/change-decision-log.md#d-5-keep-the-limiting-phrase-warning-and-separate-vague-limits-from-a-concrete-bar), [D-18](artifacts/change-decision-log.md#d-18-the-builders-read-the-per-model-guide-when-a-target-model-is-named-and-always-check-for-leftovers)). +- **Unit 1's check could not see the wrapped rule-index copy** (R1, JD-15). Replaced with a multi-line match. +- **S-7 and S-11 relied on reading alone** (R2, R3). Unit 2 and Unit 3 gained mechanical checks. +- **Wording**: "judgment calls" became "routine details the model gets right unprompted" (JD-7). The fan-out rule's line + against the self-check ban is now stated (JD-8). The effort default is attributed to the API docs (JD-9). Headings + that name only Opus 5 or only Fable 5 are renamed (JD-10). Gotchas are defined once (JD-13). Helper-library wording is + language-neutral (JD-14). New subsections go in Contents (JD-16). +- **Closed with a citation**: JD-12, which asked for explicit doc entries. The long-form docs describe the review in + general terms and stay accurate (Unit 5 note). + +The reviewers marked `Unverified` any finding that rested on inputs they could not inspect (the live source pages). +None was blocking. The one that mattered, the effort default (JD-9), was checked against the captured SOURCE 4 text +and resolved by attributing the default to the API. diff --git a/docs/plans/opus-5-5-plugin-builder-guidance/research/claude-5-5-family-guidance.md b/docs/plans/opus-5-5-plugin-builder-guidance/research/claude-5-5-family-guidance.md new file mode 100644 index 00000000..f5f50766 --- /dev/null +++ b/docs/plans/opus-5-5-plugin-builder-guidance/research/claude-5-5-family-guidance.md @@ -0,0 +1,341 @@ +# Research: Is the plugin-builder guidance current for the Claude 5.5 models? + +This report checks whether the `han-plugin-builder` authoring guidance matches what Anthropic now publishes for the +Claude 5.5 models (Opus 5.5, Sonnet 5.5, Haiku 5.5) and for Claude Code's model settings. Evidence mode: strict. + +## Summary + +The guidance (called "the guide" below) gets Opus 5.5 mostly right. It is out of date in four places, though, and says +nothing about Sonnet 5.5 or Haiku 5.5. The most serious gap is the warning about asking a model to repeat its private +reasoning (the "reasoning-echo warning" below). That warning now covers every current model, not only the two the guide +names. It also covers more prompt shapes, including a +`reasoning` field in an agent's output format. The second gap is advice to cut "check your work" instructions. That +advice is right for Opus and wrong for Sonnet 5.5 and Haiku 5.5 at low effort. Third, Claude Code changed the order it +uses to pick a subagent's model. Fourth, the built-in Explore agent no longer runs on Haiku. The fix is to correct +those four, name the two missing models where a difference changes what an author writes, and point to the live +Claude Code docs for the settings that keep moving. + +Anthropic's own pages agree on all of this, and the key pages were re-read in full for this report. No outside source +confirms them, though. + +- **Confidence:** Medium +- **Web search:** used + +## Research Results + +### The reasoning-echo warning covers every current model + +The guide says only Fable 5 and Opus 5.5 refuse an instruction to reproduce their own thinking. It also calls the +`reasoning_extraction` category "new for anyone coming from Opus 5" (A18). Both claims are wrong now. The Opus 5 +prompting page has its own section on it: prompts, skills, and tool descriptions that ask Opus 5 to write out its +reasoning "may be declined with the `reasoning_extraction` refusal category" (A16). Anthropic's refusals page lists the +classifiers on Fable 5.1, Fable 5, Opus 5.5, Opus 5, Sonnet 5.5, and Haiku 5.5 (A15). The Sonnet 5.5 page names the +category directly and says to remove instructions that ask for reasoning in the response (A8, A10). What is new on Opus +5.5 is the biology safeguard, not this category (A1). + +Haiku 5.5 is the one unclear case. Its prompting page lists four refusal categories and leaves `reasoning_extraction` +out (A9). The refusals page puts Haiku 5.5 in scope (A15). Separately, the Haiku page says the model sometimes writes +reasoning-like text into its visible reply, and the fix is adaptive thinking at `medium` effort (A9). Writing as if +the refusal applies is the safe reading. + +The trigger is wider than "echo your thinking." Anthropic lists four shapes (A15): + +- a ``, ``, or scratchpad section the model fills in before it answers +- a `reasoning`, `thinking`, or `trace` field in JSON output or in a tool input +- private notes, or a running log of the model's reasoning +- reasoning asked for verbatim or in full + +The page says this wording "can sit in a system prompt, a skill, or a tool description" (A15). An agent output format +with a `reasoning` field is exactly this pattern. A short explanation, the evidence behind a result, or a summary of +actions taken stays fine (A15, A16). + +The builders' final review checks for this today, but only for the narrow "reproduce the model's reasoning" wording +(A19). A grep of the Han agents and skills found no `reasoning` field in any output format, so Han itself is not +affected. + +### Verification advice splits by model + +The biggest difference for an author is whether to tell the model to check its work. Opus 5 checks its own work +unprompted, and Anthropic says explicit verification instructions cause over-verification and should be removed (A16). +The guide carries that advice, scoped to Opus (A18). + +Sonnet 5.5 and Haiku 5.5 run the other way at low effort. Sonnet 5.5 "generally checks its work," but at `low` effort +it "sometimes reports a change as done without running a check" (A8). Haiku 5.5 does the same at `low` and `medium` +(A9). Both pages supply a paragraph to add: run a real check (tests, type-checker, build) before reporting a code change +done (A8, A9). Fable 5 also asks for explicit self-verification on long runs, with fresh-context verifier subagents +(A7). The guide has no warning against applying the Opus removal list to these models. + +### Each 5.5 model has its own early-stop and scope habits + +Both new models carry their predecessors' prompts over. Sonnet 5 prompts "should perform well without changes" on +Sonnet 5.5, and Haiku 4.5 prompts on Haiku 5.5 (A8, A9). There is no Haiku 5; the prior Haiku is 4.5 (A14). + +Their habits differ enough to matter for a skill that runs long: + +- **Sonnet 5.5** adds tests, docs, and small supporting files it was not asked for, at every effort level and more at + higher effort (A8). At `low` and `medium` on long agentic tasks it stops to check in more often (A8). At `xhigh` and + `max` it can start its own review rounds, "sometimes with subagents" (A8). Anthropic gives one paragraph for each + (A8). +- **Haiku 5.5** rarely stops early with a short system prompt. With a long coding-agent prompt at `low` it sometimes + hands the task back. In Anthropic's testing, moving from `low` to `medium` roughly halved early stops and more than + doubled output tokens (A9) [single-source]. + +Anthropic calls Haiku 5.5 built for classification, routing, extraction, and subagent work (A11), and the effort page +lists subagents as a `low`-effort use (A12). It is the first Haiku with effort levels (A9). + +Neither 5.5 page says whether the model reads instructions literally. The Sonnet 5 page does: it reads prompts +"literally and explicitly, particularly at lower effort levels" (A13) [single-source]. Carrying that to Sonnet 5.5 +rests on the carry-over statement, not on a 5.5 claim. + +### Claude Code changed how it picks a subagent's model + +The guide's resolution order puts the `CLAUDE_CODE_SUBAGENT_MODEL` environment variable first (A18). Claude Code now +uses this order (A4): + +1. The model Claude passes when it starts the subagent. +2. The agent definition's `model` frontmatter, where `inherit` means the main conversation's model. +3. The `CLAUDE_CODE_SUBAGENT_MODEL` environment variable. +4. The main conversation's model. + +The page says the environment variable came first before Claude Code v2.1.251 (A4). It now wins over everything only +when `CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1` is also set, which needs v2.1.257 or later (A4). + +Three more Claude Code facts in the guide have drifted: + +- **Omitted `model`.** The guide says an omitted `model` defaults to `inherit` (A18). The docs say an omitted `model` + falls through the order above, so the environment variable applies before the main model (A4). The guide's own + checklist already tells authors to set `model` explicitly, which limits the harm. +- **Explore.** The guide's built-in agent table says Explore runs on `haiku` (A18). It now runs on the main + conversation's model; when that model is Fable, it runs on the Opus alias with an Anthropic subscription or Console + account (A4). +- **Aliases.** The guide's alias table leaves out `fable`, which subagents accept (A4). Aliases also resolve by + provider: `sonnet` is Sonnet 5.5 on the Anthropic API but Sonnet 4.5 on Bedrock, and `haiku` is Haiku 5.5 only on the + Anthropic API (A3). The guide's reason to prefer an alias, that "it keeps tracking the current model," holds on the + Anthropic API and not everywhere. + +These Claude Code facts the guide already gets right: a skill's `model` lasts for one turn (A5), and `effort` accepts +`low`, `medium`, `high`, `xhigh`, and `max` in skill and agent frontmatter (A3, A4, A5). + +### Thinking and effort differ by model and by surface + +In Claude Code, thinking cannot be turned off on Opus 5.5, Sonnet 5.5, Haiku 5.5, or any Fable model (A3). On the API +the picture differs. Sonnet 5.5 rejects `disabled` and offers `between_tools` as its lowest setting, accepted at `high` +effort or below (A10, A8). Haiku 5.5 accepts `disabled` at `high` or below (A9). The guide's thinking bullet names +neither model (A18). Because skills built with this guidance run in Claude Code, the Claude Code rule is the one an +author needs, labeled as such. + +Effort defaults also differ by surface. In Claude Code, Opus 5.5, Sonnet 5.5, and Haiku 5.5 all default to `medium` +(A3). On the API, Sonnet 5.5 defaults to `high` (A10, A12). Sonnet 5.5's levels were recalibrated against Sonnet 5, so +the same name gives a different amount of thinking, and Anthropic says to re-run the effort sweep (A8, A10). The +guide's "re-test a pinned effort value" advice covers only the move from Opus 5 to Opus 5.5 (A18). + +Asking any of these models in a prompt to think less does not reliably reduce thinking; lowering effort does (A1, A8, +A9). The guide already says effort, not prompt wording, is the depth lever (A18). + +### What still holds for Opus 5.5 + +The guide's Opus 5.5 claims check out against the full prompting page. Existing Opus 5 prompts carry over unchanged +(A1). Thinking cannot be turned off (A1, A3). The API default effort is `medium`, where Opus 5's was `high`, and +`medium` on Opus 5.5 "matches or exceeds" Opus 5 at `high` on coding and knowledge-work evaluations (A1). Naming the +stops you do and do not want fixes early status-report stops on unattended runs (A1). + +One attribution is off. The guide says Anthropic's Opus 5.5 guidance "recommends" the "block the merge" review prompt +(A18). That prompt is in the blog post (A2), not on the prompting page (A1) [single-source]. + +### Fable 5.1 carries Fable 5 prompts over, and the guide overstates one Fable claim + +A Fable 5.1 prompting page now exists. It says Fable 5 prompts carry over unchanged and repeats the reasoning-echo +warning (A6). + +The guide says spelling out every step "with a checklist actively degrades its output" (A18). The Fable 5 page, read in +full, never uses the word "checklist." It says you can "steer most behaviors with a brief instruction rather than +enumerating each behavior by name," and that skills written for earlier models "are often too prescriptive" and "can +degrade output quality" (A7). + +### Smaller items to check before editing + +- `specialization-and-model-selection.md` cites two pages on older documentation hosts and uses the older "thinking + budget" term (A20). Whether those links still resolve was not checked. +- `success-criteria-and-testing.md` credits Anthropic with an unlinked claim that Haiku may need more explicit + instructions (A21). No current page found says this. The Haiku 5.5 page says to use `high` effort for strict + instruction-following (A9), which is a different claim. +- The guide's four-model list and its 2026-09-23 "last checked" date repeat in five files (A18, A20): + + ``` + per-model-authoring.md + guidance/SKILL.md + guidance-portable-SKILL.md + rule-index-body.md + specialization-and-model-selection.md + ``` + +## Options to Consider + +The question is mostly "what changed," but one real choice sits inside it: how to handle Claude Code facts that keep +moving. Alias targets, the built-in agents' models, and the resolution order all changed within three weeks of the +guide's last check (A3, A4). + +### O1: Pin the current Claude Code facts in the guide + +- **What it is:** Rewrite the alias table, the built-in agent table, and the resolution order to match today's docs. +- **Trade-offs:** An author reads the answer in place. The facts go stale again on the next Claude Code release, and + `guidance init` copies them into other repos, where they age unseen. +- **Rests on:** (A3), (A4) +- **Evidence status:** corroborated (same vendor) + +### O2: State the stable rule and link to the live docs + +- **What it is:** Keep only what an author acts on (set `model` explicitly; an operator setting can override it; + aliases resolve by provider), with a version note and a link to the Claude Code page for the current details. +- **Trade-offs:** It stays correct longer. An author has to follow a link for the specifics. +- **Rests on:** (A3), (A4) +- **Evidence status:** corroborated (same vendor) + +## Recommendation + +- **Recommendation:** Use O1 for the resolution order, which an author needs to reason about frontmatter, with the + v2.1.251 version note. Use O2 for the alias targets and the built-in agents' models. Then make these changes, most + consequential first: + 1. Widen the reasoning-echo section in `per-model-authoring.md` to every model on the refusals page, drop the "new + for anyone coming from Opus 5" line, and list the four trigger shapes. Widen the builders' review check to match, + in both builder skills and their docs (A15, A16, A8, A19). + 2. Scope the "leave out verification steps" advice to Opus by name. Add that Sonnet 5.5 at `low` effort and Haiku 5.5 + at `low` and `medium` may report a change done without checking, so a skill that runs on them should keep explicit + verification and keep-working text (A16, A8, A9). + 3. Correct the subagent resolution order and the omitted-`model` wording; add `fable` to the alias table; replace the + Explore row and the alias-tracking claim with the stable rule and a link (A4, A3). + 4. Add Sonnet 5.5 and Haiku 5.5 to the thinking and effort bullets, labeled for Claude Code: thinking cannot be + turned off, all 5.5 models default to `medium`, and Sonnet 5.5's effort names were recalibrated (A3, A8, A10). + 5. Add a short note on each 5.5 model's early-stop and scope habits to the length-and-scope section (A8, A9). + 6. Quote the Fable claim as Anthropic words it, note Fable 5.1, correct the "block the merge" attribution, and update + the model list and "last checked" date everywhere it repeats (A6, A7, A2, A18, A20). + 7. Label the Sonnet literalism claim as inherited from Sonnet 5, and add Haiku 5.5's stated subagent fit to the tier + guidance (A13, A11, A12). +- **Evidence basis:** Items 1 through 4 and 6 rest on two or more Anthropic pages that agree, re-read in full for this + report (corroborated, same vendor). Item 5 rests on the Sonnet 5.5 and Haiku 5.5 prompting pages, one each (A8, A9). + Item 7's literalism point rests on the Sonnet 5 page alone (A13) and is labeled inherited. The Haiku 5.5 + reasoning-echo coverage rests on the refusals page (A15) against the Haiku page's silence (A9), so the guide presents + it as the safe reading. Out of scope: the older research reports under `docs/research/` that quote the old resolution + order are historical records and stay as they are. + +## Validation + +### V1: The report describes the guide accurately + +- **Strategy:** Challenge the Evidence +- **Investigation:** Opened every cited repo location and quoted the guide's wording. +- **Result:** Confirmed +- **Impact:** None. The validator noted the guide's "always set `model` explicitly" checklist limits the harm of the + stale `inherit` default; the report now says so. + +### V2: The recommendation names every place the stale facts live + +- **Strategy:** Challenge the Fix +- **Investigation:** Grepped the repo for the model list and the old resolution order. +- **Result:** Partially Refuted +- **Impact:** Added `specialization-and-model-selection.md` to the model-list sweep, added the builders' review check to + item 1, and marked the `docs/research/` copies out of scope. + +### V3: Opus 5 being in scope on the refusals page does not prove it emits the category + +- **Strategy:** Challenge the Evidence +- **Investigation:** Re-fetched the full Opus 5 prompting page. Its "Reasoning in the response" section names the + `reasoning_extraction` category for Opus 5 directly (A16). +- **Result:** Refuted; the challenge does not hold +- **Impact:** The "new for anyone coming from Opus 5" line is wrong on direct evidence, not inference. Citations now + rest on A16. + +### V4: The resolution-order specifics were unquoted + +- **Strategy:** Challenge the Evidence +- **Investigation:** Re-fetched the subagents page. It states the four-step order, the pre-v2.1.251 behavior, the + `_FORCE` variable at v2.1.257, the omitted-`model` behavior, and the Explore rule (A4). +- **Result:** Confirmed once quoted +- **Impact:** Moved the version citations from A3 to A4 and added the version note to the recommendation. + +### V5: Thinking and effort claims mix surfaces + +- **Strategy:** Challenge the Evidence +- **Investigation:** Re-fetched the Sonnet 5.5 "what's new" page; it says `between_tools` "is accepted at `low`, + `medium`, and `high` effort" (A10). +- **Result:** Partially Refuted +- **Impact:** Labeled every thinking and effort fact by surface and made the Claude Code rule the recommended one. + +### V6: The options framing missed a real choice + +- **Strategy:** Challenge the Assumptions +- **Investigation:** The guide's own date stamp calls it a snapshot, and three Claude Code facts moved within three + weeks. +- **Result:** Refuted +- **Impact:** Added O1 and O2 and a split recommendation. + +### V7: The recommendation order and some wording over-reached + +- **Strategy:** Challenge the Fix +- **Investigation:** Re-fetched the Sonnet 5.5, Haiku 5.5, Fable 5, and Opus 5.5 pages in full. +- **Result:** Partially Refuted +- **Impact:** Reordered so the refusal and verification items lead. Changed "need" to "may report done without + checking." The Fable "checklist" absence now rests on a full read. Dropped "every" from the Opus 5.5 claim. + +### V8: Some citations did not support their claims + +- **Strategy:** Challenge the Evidence +- **Investigation:** Checked each citation against its registry line. +- **Result:** Partially Refuted +- **Impact:** Rewrote the registry lines for A1, A4, A8, A9, A16 so each states the facts it backs. + +### V9: The blog post may not be first-party, and the findings depend on few pages + +- **Strategy:** Challenge the Evidence-Gathering Integrity +- **Investigation:** The blog post sits on a different domain from the docs and was fetched as a summary. Removing it + changes only the attribution note in item 6. +- **Result:** Partially Refuted +- **Impact:** A2 is marked as not confirmed first-party. Items 1 and 3 rest on a few same-vendor pages, so confidence + stays Medium. No directive text aimed at the reader was found in the pages read in full. + +### V10: Two "stale" items were hypotheses + +- **Strategy:** Challenge the Evidence +- **Investigation:** Re-read the A20 and A21 claims. +- **Result:** Partially Refuted +- **Impact:** Moved both under "Smaller items to check before editing" and dropped the replacement source for A21. + +### Adjustments Made + +Validation reordered the recommendation, added the options section, widened the edit list, and replaced several +unquoted claims with direct quotes from re-fetched pages. The recommendation survived with those changes. + +### Confidence Assessment + +- **Confidence:** Medium +- **Remaining Risks:** Every source is Anthropic's, with no outside confirmation. No page shows a revision date. The + subagents and model-config pages were read to about 100,000 characters, not to the end. Whether Haiku 5.5 emits the + `reasoning_extraction` category is unresolved. Claude Code settings change often, so the pinned resolution order + needs its version note. + +## Sources + +| ID | Source | Link / location | Retrieved | Trust class | Summary (one line) | Evidence status | +| --- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| A1 | Prompting Claude Opus 5.5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5 | 2026-10-15 | web | Opus 5 prompts carry over; default effort `medium` "matches or exceeds" Opus 5 at `high`; thinking always on; biology safeguard new vs Opus 5; name the stops. | corroborated by A3, A12, A16 | +| A2 | Getting the most out of Opus 5.5 | https://claude.dev/blog/getting-the-most-out-of-opus-5-5/ | 2026-10-15 | web | Blog post carrying the "block the merge" review prompt; fetched as a summary; first-party status not confirmed. | single source (caveated) | +| A3 | Claude Code model configuration | https://code.claude.com/docs/en/model-config | 2026-10-15 | web | Aliases resolve by provider; Claude Code defaults all 5.5 models to `medium`; thinking cannot be turned off on 5.5 or Fable models. | corroborated by A4, A12 | +| A4 | Claude Code subagents | https://code.claude.com/docs/en/sub-agents | 2026-10-15 | web | Model order dispatch → frontmatter → env var → main; env var first before v2.1.251; `_FORCE` at v2.1.257; `fable` valid; Explore on main model. | corroborated by A3 | +| A5 | Claude Code skills | https://code.claude.com/docs/en/skills | 2026-10-15 | web | Skill `model` lasts one turn; skill `effort` accepts `low` through `max`. | corroborated by A3 | +| A6 | Prompting Claude Fable 5.1 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-1 | 2026-10-15 | web | Fable 5 prompts carry over to Fable 5.1; repeats the reasoning-echo warning for prompts, skills, and tool descriptions. | corroborated by A7, A15 | +| A7 | Prompting Claude Fable 5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5 | 2026-10-15 | web | Steer with a brief instruction; older skills "often too prescriptive" and "can degrade output quality"; no "checklist" wording; asks for explicit self-verification. | single source (caveated) | +| A8 | Prompting Claude Sonnet 5.5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5 | 2026-10-15 | web | Sonnet 5 prompts carry over; effort recalibrated; skips checks at `low`; checks in at `low`/`medium`; adds unasked files; reviewer subagents at `xhigh`/`max`. | corroborated by A10, A12, A15 | +| A9 | Prompting Claude Haiku 5.5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5 | 2026-10-15 | web | Haiku 4.5 prompts carry over; skips checks at `low`/`medium`; early stops on long prompts at `low`; `disabled` thinking at `high` or below; four refusal categories. | corroborated by A11, A12 | +| A10 | What's new in Claude Sonnet 5.5 | https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5 | 2026-10-15 | web | `disabled` returns 400; `between_tools` accepted at `low`/`medium`/`high`; API default effort `high`; effort recalibrated; names `reasoning_extraction`. | corroborated by A8, A12 | +| A11 | What's new in Claude Haiku 5.5 | https://platform.claude.com/docs/en/models/haiku-5-5/whats-new-haiku-5-5 | 2026-10-15 | web | Built for classification, routing, extraction, and subagents; 1M context. | corroborated by A9, A12 | +| A12 | Effort | https://platform.claude.com/docs/en/build-with-claude/effort | 2026-10-15 | web | All five levels on Sonnet 5.5 and Haiku 5.5; API defaults `high` for Sonnet 5.5 and `medium` for Haiku 5.5 and Opus 5.5; `low` suited to subagents. | corroborated by A3, A8, A9 | +| A13 | Prompting Claude Sonnet 5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5 | 2026-10-15 | web | Sonnet 5 reads prompts literally, especially at low effort; vague review limits cut recall. | single source (caveated) | +| A14 | Models overview | https://platform.claude.com/docs/en/models/overview | 2026-10-15 | web | Current lineup Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5; legacy includes Haiku 4.5; no Haiku 5. | corroborated by A10, A11 | +| A15 | Refusals and fallback | https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback | 2026-10-15 | web | Classifiers on Fable 5.1/5, Opus 5.5/5, Sonnet 5.5, Haiku 5.5; four `reasoning_extraction` trigger shapes; wording can sit in a skill or tool description. | corroborated by A6, A8, A16 | +| A16 | Prompting Claude Opus 5 | https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5 | 2026-10-15 | web | Remove verification instructions (over-verification); "Reasoning in the response" section names `reasoning_extraction` for Opus 5. | corroborated by A15 | +| A18 | per-model-authoring.md and siblings | `han-plugin-builder/skills/guidance/references/per-model-authoring.md:18,42,48,58,68-69`; `agent-building-guidelines/agent-model-selection.md:33-38,53-56,152` | n/a | codebase | The guide's model list, Fable checklist claim, review-prompt attribution, reasoning-echo scope, thinking/effort bullets, resolution order, alias and agent tables. | current-state anchor | +| A19 | Builder review checks | `han-plugin-builder/skills/skill-builder/SKILL.md:195`; `han-plugin-builder/skills/agent-builder/SKILL.md:187` | n/a | codebase | Builders' final review flags only "reproduce the model's reasoning" wording, not the wider trigger shapes. | current-state anchor | +| A20 | specialization-and-model-selection | `han-plugin-builder/skills/guidance/references/specialization-and-model-selection.md:9,81-82,206` | n/a | codebase | Uses "thinking budget" wording, cites older documentation hosts, and repeats the four-model list. | current-state anchor | +| A21 | success-criteria-and-testing.md | `han-plugin-builder/skills/guidance/references/skill-building-guidance/success-criteria-and-testing.md:24-27` | n/a | codebase | Credits Anthropic with an unlinked "Haiku may need more explicit instructions" claim. | current-state anchor | + +A17 (the prompting best-practices index) was dropped in consolidation: nothing in the final report rests on it. diff --git a/han-plugin-builder/docs/skills/agent-builder.md b/han-plugin-builder/docs/skills/agent-builder.md index 7bc23f18..219e221d 100644 --- a/han-plugin-builder/docs/skills/agent-builder.md +++ b/han-plugin-builder/docs/skills/agent-builder.md @@ -108,7 +108,11 @@ would call it. sonnet for structured procedures, haiku for fast lookups). If your sense of the work differs, say so. - **Trust the review pass.** The Step 6 conformance review fixes role-identity length, description budget, self-containment violations, and an over-broad tool set before you see them. That includes dropping the `Agent` tool - unless the agent's own protocol dispatches sub-agents, since dispatch flows from skills to agents by default. + unless the agent's own protocol dispatches sub-agents, since dispatch flows from skills to agents by default. It also + strips model-specific leftovers, such as a "think step by step" line, a `reasoning` field in the output format (which + any current model can refuse), a pinned `effort` with no stated reason, or a + vague limit like "only report high-severity issues" in a review step, and checks that a research or analysis agent + marks each claim it could not confirm. - **Wire up the caller.** An agent is dispatched by a skill. If the calling skill does not exist yet, the skill recommends [`/skill-builder`](./skill-builder.md) to build it. @@ -132,7 +136,7 @@ flowchartable process) with a single role; discover the target plugin, its sibli build the design tree in dependency order. Then it runs the interview loop one branch at a time; writes the single self-contained file; runs the full guidance-conformance review; and presents the result with the dispatch wiring. Each design-tree decision maps to a specific governing document (domain focus, description length, model selection, external -files, multi-agent economics, dispatch namespacing), read only when that decision is on the table. The review pass +files, multi-agent economics, dispatch namespacing, per-model authoring), read only when that decision is on the table. The review pass re-reads each document that applies and corrects the file directly rather than reporting problems back to you. ## Sources diff --git a/han-plugin-builder/docs/skills/skill-builder.md b/han-plugin-builder/docs/skills/skill-builder.md index 9ea85839..efbf06a8 100644 --- a/han-plugin-builder/docs/skills/skill-builder.md +++ b/han-plugin-builder/docs/skills/skill-builder.md @@ -101,7 +101,10 @@ cases. - **Push back during the interview.** Every recommendation is a proposal. If a recommended workflow pattern or tool set is wrong for your case, say so; the skill resolves dependent decisions from your redirect. - **Trust the review pass, then run the tests.** The Step 6 conformance review fixes description length, naming, - progressive-disclosure, and `allowed-tools` problems before you ever see them. After it lands, run the triggering and + progressive-disclosure, and `allowed-tools` problems before you ever see them. It also strips model-specific leftovers, + such as a "think step by step" line, a `reasoning` field in an output format (which any current model can refuse), a + pinned `effort` with no stated reason, or a vague limit like "only report + high-severity issues" in a review step. After it lands, run the triggering and functional tests it hands you against the model tier the skill targets. - **Plan for iteration.** Plugin entities rarely land in one pass. Expect 3-5 iterations; the skill says so and invites you to iterate on specific steps. @@ -126,7 +129,8 @@ The workflow runs in seven steps: capture the request and confirm a skill is the discover the target plugin and its conventions; and build the design tree in dependency order. Then it runs the interview loop one branch at a time; writes the skill files; runs the full guidance-conformance review; and presents the result with tests. Each design-tree decision maps to a specific governing document (use-case planning, naming -conventions, description frontmatter, progressive disclosure, workflow patterns, allowed-tools, and so on). It is read +conventions, description frontmatter, progressive disclosure, workflow patterns, allowed-tools, per-model authoring +when the skill targets a named model, and so on). It is read only when that decision is on the table. The review pass re-reads each document that applies to what was built and corrects the files directly rather than reporting problems back to you. diff --git a/han-plugin-builder/skills/agent-builder/SKILL.md b/han-plugin-builder/skills/agent-builder/SKILL.md index 4f7820ca..c8858f6a 100644 --- a/han-plugin-builder/skills/agent-builder/SKILL.md +++ b/han-plugin-builder/skills/agent-builder/SKILL.md @@ -43,6 +43,7 @@ Map from decision to governing document (read just-in-time): | Which frontmatter fields are valid (and which plugins ignore) | `agent-building-guidelines/agent-external-files.md` | | Degraded environments (no git, missing tools) | `agent-building-guidelines/graceful-degradation.md` | | Whether this agent is justified at all; how it gets dispatched | `agent-building-guidelines/multi-agent-economics.md`, `skill-building-guidance/agent-dispatch-namespacing.md` | +| Instructions for a named target model | `per-model-authoring.md` | | New plugin needed (plugin.json, marketplace.json) | `claude-marketplace-and-plugin-configuration/` and `templates/` | ## Operating Principles @@ -179,9 +180,16 @@ Cover at minimum: since dispatch flows from skills to agents; it carries the `Agent` tool only when its own protocol dispatches sub-agents. The `tools` allowlist is the minimum the work needs, each tool present only if the body uses it. 8. **Graceful degradation** (`agent-building-guidelines/graceful-degradation.md`) — every tool-dependent step checks - availability inline and notes the limitation when the tool is absent. + availability inline and notes the limitation when the tool is absent, and a research or analysis agent marks each + claim it could not confirm and names where it looked. 9. **Economic justification** (`multi-agent-economics.md`) — the agent clears the bar for existing: a single well-prompted agent or an instruction improvement to an existing agent would not do the job as well. +10. **Model-specific leftovers** (`per-model-authoring.md`) — no "think step by step" or "think carefully" line; no + instruction to reproduce the model's reasoning in its reply, and no output format with a `reasoning`, `thinking`, + or `trace` field, a scratchpad section, or a running reasoning log; no pinned `effort` without a reason stated in + the body; and no vague limiting phrase in a review step. A limit passes when it names a bar the reader could check and + asks for evidence on each item. "List only problems you'd block the merge for. For each one, give the file and + line, why it's wrong, and how to show it fails." passes; "Only report high-severity issues." does not. Apply the YAGNI discipline throughout: vocabulary terms, anti-patterns, tools, and frontmatter fields must each earn their place against the agent's actual job. Cut anything added "for completeness." diff --git a/han-plugin-builder/skills/guidance/SKILL.md b/han-plugin-builder/skills/guidance/SKILL.md index 804253ec..8b38bcde 100644 --- a/han-plugin-builder/skills/guidance/SKILL.md +++ b/han-plugin-builder/skills/guidance/SKILL.md @@ -45,8 +45,8 @@ specific file(s) you need: under `${CLAUDE_SKILL_DIR}/references/claude-marketplace-and-plugin-configuration/`. - Versioning, README structure, local development, the iterative development process, and specialization-versus-model-tier reasoning → the top-level files in `${CLAUDE_SKILL_DIR}/references/`. -- Writing the instructions for a target model (how Sonnet 5, Opus 5, and Fable 5 differ in following instructions, which - instructions to leave out, and how to calibrate length, narration, and scope) → +- Writing the instructions for a target model (how Sonnet, Haiku, Opus, and Fable models differ in following + instructions, which instructions to leave out, and how to calibrate length, narration, and scope) → `${CLAUDE_SKILL_DIR}/references/per-model-authoring.md`. - Copyable starter files → `${CLAUDE_SKILL_DIR}/references/templates/`. diff --git a/han-plugin-builder/skills/guidance/assets/guidance-portable-SKILL.md b/han-plugin-builder/skills/guidance/assets/guidance-portable-SKILL.md index 4e36db1b..ca7a7a80 100644 --- a/han-plugin-builder/skills/guidance/assets/guidance-portable-SKILL.md +++ b/han-plugin-builder/skills/guidance/assets/guidance-portable-SKILL.md @@ -24,8 +24,8 @@ specific file(s) you need: under `${CLAUDE_SKILL_DIR}/references/claude-marketplace-and-plugin-configuration/`. - Versioning, README structure, local development, the iterative development process, and specialization-versus-model-tier reasoning → the top-level files in `${CLAUDE_SKILL_DIR}/references/`. -- Writing the instructions for a target model (how Sonnet 5, Opus 5, and Fable 5 differ in following instructions, which - instructions to leave out, and how to calibrate length, narration, and scope) → +- Writing the instructions for a target model (how Sonnet, Haiku, Opus, and Fable models differ in following + instructions, which instructions to leave out, and how to calibrate length, narration, and scope) → `${CLAUDE_SKILL_DIR}/references/per-model-authoring.md`. - Copyable starter files → `${CLAUDE_SKILL_DIR}/references/templates/`. diff --git a/han-plugin-builder/skills/guidance/assets/rule-index-body.md b/han-plugin-builder/skills/guidance/assets/rule-index-body.md index fe49e8cf..41e0e380 100644 --- a/han-plugin-builder/skills/guidance/assets/rule-index-body.md +++ b/han-plugin-builder/skills/guidance/assets/rule-index-body.md @@ -44,11 +44,14 @@ Guidance for authoring and hardening `SKILL.md` files and their companion folder focused on what the current step needs. Read when deciding where a piece of content belongs, or when a SKILL.md is growing too large. - [Writing Effective Instructions](.claude/skills/plugin-guidance/references/skill-building-guidance/writing-effective-instructions.md) - — How to write the SKILL.md body so steps are specific, actionable, and reliably followed across sessions. Read when a - skill behaves inconsistently, skips steps, or improvises when it should follow a fixed process. + — How to write the SKILL.md body so steps are specific, actionable, and reliably followed across sessions, and where + to stop specifying. Read when a skill behaves inconsistently, skips steps, or improvises when it should follow a fixed + process, or when its rules are over-constraining it. - [Workflow Patterns](.claude/skills/plugin-guidance/references/skill-building-guidance/workflow-patterns.md) — Four - structural patterns for organizing the steps inside a single skill, mapped to Anthropic's effective-agent patterns. - Read when designing or restructuring a skill's internal workflow. + structural patterns for organizing the steps inside a single skill, mapped to Anthropic's effective-agent patterns, + plus how to name the stops an autonomous stretch should and should not make and how to bound a step that works through + an unknown number of items. Read when designing or restructuring a skill's internal workflow, or when a skill stops + early or loses track mid-run. - [Context Injection Commands](.claude/skills/plugin-guidance/references/skill-building-guidance/context-injection-commands.md) — The `` !`command` `` syntax that runs a shell command at skill load time and injects its output as runtime context. Read when a skill needs dynamic environment data (dates, git state, branch names) available to its steps. @@ -57,10 +60,11 @@ Guidance for authoring and hardening `SKILL.md` files and their companion folder fenced code blocks. Read when a skill runs shell scripts during its steps. - [Hardening: Fuzzy vs. Deterministic](.claude/skills/plugin-guidance/references/skill-building-guidance/hardening-fuzzy-vs-deterministic.md) — The framework for classifying each skill step as fuzzy (keep as an LLM instruction) or deterministic (extract to a - script). Read when hardening a skill for reliability or deciding what to script. + script), including a helper library the model composes into one-off scripts. Read when hardening a skill for + reliability or deciding what to script. - [Skill Reference Files](.claude/skills/plugin-guidance/references/skill-building-guidance/skill-reference-files.md) — - When and how to extract domain knowledge (templates, checklists, rate tables) into a `references/` subdirectory loaded - on demand. Read when a skill carries content that is knowledge rather than process steps. + When and how to extract domain knowledge (templates, checklists, rate tables, gotchas) into a `references/` + subdirectory loaded on demand. Read when a skill carries content that is knowledge rather than process steps. - [Context Hygiene](.claude/skills/plugin-guidance/references/skill-building-guidance/context-hygiene.md) — The attention-budget mechanism behind progressive disclosure and conciseness rules: why every irrelevant token degrades the model's attention on the rest. Read when justifying why content should be trimmed or moved out of a SKILL.md. @@ -96,7 +100,8 @@ Guidance for authoring and hardening `SKILL.md` files and their companion folder plugin.json and that skill directories must not carry README files. Read when creating or renaming any plugin entity. - [Success Criteria and Testing](.claude/skills/plugin-guidance/references/skill-building-guidance/success-criteria-and-testing.md) — Three test types (triggering, functional, outcome) for knowing a skill works, plus the rule to test on the model - tier the skill targets. Read when validating a skill before shipping it. + tier the skill targets, and how to build a skill whose job is verifying a running product. Read when validating a + skill before shipping it, or when building a verification skill. - [Documentation Maintenance](.claude/skills/plugin-guidance/references/skill-building-guidance/documentation-maintenance.md) — Why stale SKILL.md or reference content is active poison the model follows faithfully, and how to audit a skill so its docs match reality. Read when changing a skill's behavior or auditing existing skills. @@ -126,11 +131,14 @@ Guidance for authoring agent `.md` definitions. Agents are self-contained and ca explicitly not a factor. Read when setting or revisiting an agent's model. - [Multi-Agent Economics](.claude/skills/plugin-guidance/references/agent-building-guidelines/multi-agent-economics.md) — The escalation cascade for deciding whether adding more agents is justified, given that each agent multiplies - latency and token cost, plus the delegation policy a skill should state because the model's own default is eager. - Read when a skill is considering dispatching multiple or parallel agents. + latency and token cost, plus the delegation policy a skill should state because the model's own default is eager, the + evidence check a skill runs on each fanned-out subagent's result, and the rule against finishing while a background + dispatch is still running. Read when a skill is considering dispatching multiple or parallel agents. - [Graceful Degradation (agents)](.claude/skills/plugin-guidance/references/agent-building-guidelines/graceful-degradation.md) — How a dispatched agent should check tool availability inline and skip gracefully, so the orchestrating skill needs - no defensive guards around the dispatch. Read when an agent's steps depend on git or other tools that may be missing. + no defensive guards around the dispatch, and how a research or analysis agent marks a claim it could not confirm and + says where it looked. Read when an agent's steps depend on git or other tools that may be missing, or when an agent + reports findings. ## Plugin configuration files @@ -142,7 +150,8 @@ Schema references for the JSON manifests that define a plugin and its marketplac naming a new plugin or renaming an existing one. - [plugin.json Schema Reference](.claude/skills/plugin-guidance/references/claude-marketplace-and-plugin-configuration/plugin-json-options.md) — Full schema for `.claude-plugin/plugin.json`: required fields, metadata, component paths, dependencies, and - experimental keys. Read when creating or editing a plugin manifest. + experimental keys, plus where a plugin-shipped skill keeps state that must survive an update. Read when creating or + editing a plugin manifest, or when a skill saves first-run answers or run history. - [marketplace.json Schema Reference](.claude/skills/plugin-guidance/references/claude-marketplace-and-plugin-configuration/marketplace-json-options.md) — Schema for `.claude-plugin/marketplace.json`, the registry Claude Code reads to discover and install plugins. Read when adding a plugin to a marketplace or editing the manifest. @@ -163,12 +172,14 @@ Process guidance for building and evolving a plugin over its lifetime. — The research-backed rationale for why tighter specialization lets a smaller model at lower effort match a larger one on narrow tasks, without raising the capability ceiling. Read when reasoning about the specialization-versus-model-tier trade-off across skills and agents. -- [Per-Model Authoring Guidance](.claude/skills/plugin-guidance/references/per-model-authoring.md) — How Sonnet 5, - Opus 5, and Fable 5 differ in how they follow instructions, and how those differences change what you write: the - model-agnostic default for an unknown target, the opposite-direction instruction-style split, the verification and - re-check instructions to leave out, how to calibrate response length, narration, and scope, and the Fable 5 - reasoning-echo refusal to avoid. Read when writing or hardening a skill or agent and tuning the instructions to a - target model, not when choosing which model tier to run (see Specialization and Model Selection). +- [Per-Model Authoring Guidance](.claude/skills/plugin-guidance/references/per-model-authoring.md) — How the + Sonnet, Haiku, Opus, and Fable models (through the 5.5 releases and Fable 5.1) differ in how they follow instructions, + and how those differences change what you write: the model-agnostic default for an unknown target, the + opposite-direction instruction-style split, the verification instructions to leave out on Opus and keep on Sonnet 5.5 + and Haiku 5.5, how to calibrate response length, narration, and scope, early stops on Sonnet 5.5 and Haiku 5.5, effort + and thinking on the 5.5 models, and the reasoning-echo refusal on every current model. Read when writing or hardening a skill or agent and + tuning the instructions to a target model, not when choosing which model tier to run (see Specialization and Model + Selection). ## Templates and examples diff --git a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-external-files.md b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-external-files.md index 8fb36ab1..3ea75da5 100644 --- a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-external-files.md +++ b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-external-files.md @@ -97,22 +97,22 @@ Many agents set only `name`, `description`, `tools`, and `model`, but the [Subagents documentation](https://code.claude.com/docs/en/sub-agents) supports more. Only `name` and `description` are required. The others, briefly: -| Field | What it does | -| ----------------- | -------------------------------------------------------------------------------------------- | -| `tools` | Allowlist of tools the agent may use. Inherits all tools if omitted. | -| `disallowedTools` | Denylist applied on top of `tools`. A tool in both is removed. | -| `model` | Model alias, full model ID, or `inherit`. See [Model Selection](./agent-model-selection.md). | -| `permissionMode` | Permission posture (`default`, `acceptEdits`, `plan`, and so on). | -| `maxTurns` | Cap on agentic turns before the agent stops. | -| `skills` | Skills to preload at startup (full content injected). | -| `mcpServers` | MCP servers available to the agent. | -| `hooks` | Lifecycle hooks scoped to the agent. | -| `memory` | `user`, `project`, or `local` to enable cross-session persistent memory. | -| `background` | `true` to always run the agent as a background task. | -| `effort` | `low`/`medium`/`high`/`xhigh`/`max` effort override. | -| `isolation` | `worktree` to run the agent in a temporary git worktree. | -| `color` | Display color for the agent in the UI. | -| `initialPrompt` | First user turn auto-submitted when the agent runs as a main session via `--agent`. | +| Field | What it does | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `tools` | Allowlist of tools the agent may use. Inherits all tools if omitted. | +| `disallowedTools` | Denylist applied on top of `tools`. A tool in both is removed. | +| `model` | Model alias, full model ID, or `inherit`. See [Model Selection](./agent-model-selection.md). | +| `permissionMode` | Permission posture (`default`, `acceptEdits`, `plan`, and so on). | +| `maxTurns` | Cap on agentic turns before the agent stops. | +| `skills` | Skills to preload at startup (full content injected). | +| `mcpServers` | MCP servers available to the agent. | +| `hooks` | Lifecycle hooks scoped to the agent. | +| `memory` | `user`, `project`, or `local` to enable cross-session persistent memory. | +| `background` | `true` to always run the agent as a background task. A skill that dispatches it waits for its result before finishing; see [Multi-Agent Economics](./multi-agent-economics.md#practical-implications-for-skills). | +| `effort` | `low`/`medium`/`high`/`xhigh`/`max` effort override. Re-test a pinned value when the model changes; see [Per-Model Authoring](../per-model-authoring.md#other-settings-the-model-differences-affect-thinking-mode-effort-and-subagent-eagerness). | +| `isolation` | `worktree` to run the agent in a temporary git worktree. | +| `color` | Display color for the agent in the UI. | +| `initialPrompt` | First user turn auto-submitted when the agent runs as a main session via `--agent`. | ### Plugin agents ignore three of these (security boundary) diff --git a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-model-selection.md b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-model-selection.md index e7f984af..16d501c5 100644 --- a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-model-selection.md +++ b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/agent-model-selection.md @@ -31,11 +31,16 @@ The `model` field in agent frontmatter controls which AI model the agent uses. | `opus` | Most capable model. Deep reasoning and judgment. | | `sonnet` | Balanced capability and speed. | | `haiku` | Fastest model. Low-latency, lightweight tasks. | +| `fable` | Anthropic's Fable tier, above `opus`. | | `inherit` | Uses the same model as the user's main session. | -**Default behavior:** If `model` is omitted, the agent defaults to `inherit`. A full model ID (for example -`claude-opus-5`) is also accepted in place of an alias. Prefer the alias: it keeps tracking the current model in that -tier, where a pinned ID goes stale as new models ship. +**Default behavior:** If `model` is omitted, Claude Code falls through the resolution order below: an operator's +`CLAUDE_CODE_SUBAGENT_MODEL` applies first, then the main conversation's model. That is close to `inherit`, but not the +same, so set the field explicitly. A full model ID (for example `claude-opus-5-5`) is also accepted in place of an +alias. Prefer the alias: on the Anthropic API it tracks the newest model in that tier, where a pinned ID goes stale as +new models ship. On some other providers an alias resolves to an earlier model (for example, `sonnet` on Amazon Bedrock +was still Sonnet 4.5 as of 2026-10-15), so an alias names a tier, not a version. The current mapping by provider is in +[Claude Code's model configuration docs](https://code.claude.com/docs/en/model-config). **Syntax example:** @@ -50,13 +55,16 @@ model: sonnet **Resolution order.** When an agent runs, Claude Code resolves the model from the first source that is set: -1. The `CLAUDE_CODE_SUBAGENT_MODEL` environment variable. -2. The model chosen at dispatch time (Claude's own decision when it delegates). -3. The agent definition's `model` frontmatter (the value this doc is about). -4. The main conversation's model (the `inherit` default). +1. The model chosen at dispatch time (Claude's own decision when it delegates). +2. The agent definition's `model` frontmatter (the value this doc is about), where `inherit` selects the main + conversation's model. +3. The `CLAUDE_CODE_SUBAGENT_MODEL` environment variable. +4. The main conversation's model. -The frontmatter `model` is the level you control as an agent author, but an operator's env var or a dispatch-time choice -can override it. +The frontmatter `model` is the level you control as an agent author. A dispatch-time choice can override it. The +environment variable is only a default beneath it, unless the operator also sets `CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1`, +which runs every subagent on the environment variable's model and ignores the frontmatter. This order applies from +Claude Code v2.1.251; before that the environment variable came first. `_FORCE` needs v2.1.257 or later. **Scope: this rule governs agent definition files only, not skill dispatch.** "Always set model explicitly" applies to the `model:` frontmatter of an agent definition (`**/agents/**/*.md`). It does NOT mean a skill should pass a `model` @@ -110,6 +118,12 @@ Use `haiku`. Signs that an agent fits haiku: - Simple, repetitive operations at high volume - Tasks where latency matters more than depth +Anthropic describes Haiku 5.5 as built for classification, routing, extraction, and subagent work, which matches this +tier. Two of its habits shape the agent body. With a long prompt at `low` effort it sometimes stops early and hands the +task back, so keep a haiku agent's prompt short. At `low` and `medium` effort it sometimes reports a code change done +without checking it, so an agent that edits code should say to run a real check before reporting done. See +[Per-Model Authoring Guidance](../per-model-authoring.md#early-stops-and-scope-on-sonnet-55-and-haiku-55). + ### Summary Decision Table | Task Characteristic | Model | @@ -149,12 +163,16 @@ sonnet; open-ended synthesis agents weighing competing factors over unbounded in | Agent | Model | Rationale | | ----------------- | --------- | -------------------------------------------------------- | -| Explore | `haiku` | Fast, read-only codebase searches. Speed over depth. | +| Explore | `inherit` | Read-only codebase searches on the main session's model. | | Plan | `inherit` | Research for planning matches user's session model. | | General-purpose | `inherit` | Generic delegation. User's model choice carries through. | | statusline-setup | `sonnet` | Focused configuration task with a clear procedure. | | claude-code-guide | `haiku` | Fast Q&A lookups against Claude Code documentation. | +Built-in agents' models change between Claude Code releases. Explore ran on `haiku` until recently, and now runs on the +main conversation's model, or on the Opus alias when that model is Fable. Check the +[sub-agents docs](https://code.claude.com/docs/en/sub-agents) for the current table rather than relying on this one. + These examples reinforce the decision criteria: opus for synthesis and judgment, sonnet for structured procedures, haiku for fast lookups, inherit for generic tasks. @@ -176,5 +194,6 @@ for fast lookups, inherit for generic tasks. - [Claude Code Sub-agents Documentation](https://code.claude.com/docs/en/sub-agents). Documents the `model` field, valid values, defaults, and built-in agent configurations. +- [Claude Code Model Configuration](https://code.claude.com/docs/en/model-config). Alias resolution by provider. - [Choosing the Right Model](https://platform.claude.com/docs/en/about-claude/models/choosing-a-model). Model comparison covering capabilities, speed, and selection criteria. diff --git a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/graceful-degradation.md b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/graceful-degradation.md index 24e2645c..18c01b53 100644 --- a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/graceful-degradation.md +++ b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/graceful-degradation.md @@ -54,6 +54,28 @@ changed in the last 30 days. This helps the calling skill and user understand why certain analysis was omitted without treating it as a failure. +### Rule: Mark a claim you could not confirm, and say where you looked + +A research or analysis agent can have every tool it needs and still fail to establish a claim: the search turned up +nothing, or the evidence it found is partial. Without this rule, that gap disappears into the report, and the calling +skill treats an unconfirmed claim the same as a confirmed one. + +For any finding the agent could not confirm, mark it as unconfirmed on the finding itself and name where the agent +looked. This is the same "note the limitation" move as the rule above, triggered by missing evidence rather than a +missing tool. + +**Pattern:** + +> _"Mark anything you couldn't confirm, and say where you looked."_ + +**Noting it** means a line on the finding such as: + +> _"Unconfirmed: no caller of `retryPayment` found in `services/` or `jobs/`. Searched for the symbol name and the +> `payments.retry` event."_ + +"I couldn't find this" is worth reading, and asking for it makes it easy to find. The pattern comes from Anthropic's +[Opus 5.5 guidance](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/). + --- ## Summary Checklist @@ -61,6 +83,7 @@ This helps the calling skill and user understand why certain analysis was omitte 1. Check tool availability inline before tool-dependent steps. Do not assume tools are present. 2. Use the pattern _"If X is not available, skip this step and note this limitation."_ 3. Include an explicit note in agent output when a step is skipped due to tool absence. +4. For a research or analysis agent, mark each claim it could not confirm and name where it looked. --- diff --git a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/multi-agent-economics.md b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/multi-agent-economics.md index a972b6d8..c34b4792 100644 --- a/han-plugin-builder/skills/guidance/references/agent-building-guidelines/multi-agent-economics.md +++ b/han-plugin-builder/skills/guidance/references/agent-building-guidelines/multi-agent-economics.md @@ -37,7 +37,7 @@ earlier ones, and that delegation pays off on genuinely independent, sizeable tr time on small ones. Left unstated, that eagerness pushes a skill up the cascade below without any of the measurement the cascade asks for. -So a skill that dispatches agents should state its delegation policy rather than rely on the default. Three rules, +So a skill that dispatches agents should state its delegation policy rather than rely on the default. Four rules, which hold regardless of the model: - **Do not delegate work the skill can finish in a handful of tool calls.** The dispatch overhead exceeds the work. @@ -46,6 +46,11 @@ which hold regardless of the model: evaluates the output from a genuinely different perspective. That pattern still earns its cost; a self-check does not. - **Prefer one agent to several.** If a single agent can complete the track, dispatch one and keep spawn counts low. +- **When you fan out, check each subagent's evidence before accepting its finding.** A skill that gives each unit of + work its own subagent (one per service in an audit, one per module in a migration) reads the evidence each subagent + cites against its source before folding the finding into the result, then produces one consolidated result. This is + not the self-check the second rule forbids. Checking the evidence a subagent cites is allowed; dispatching another + agent to redo the skill's own reasoning is not. Where the policy needs to be deterministic rather than advisory, cap the count in the skill body instead of describing when delegation is appropriate. @@ -143,6 +148,9 @@ When designing a skill that dispatches agents: 5. **Match team composition to the task.** Not every invocation needs every agent. If a skill dispatches a security reviewer, accessibility reviewer, and performance reviewer, but the current change only affects API endpoints, skip the accessibility reviewer for that run. +6. **Do not finish while a background dispatch is still running.** A dispatch launched with `run_in_background: true` + returns before its work is done. Wait for its result and check it before the skill treats its run as complete. This + item rests on a single source, Anthropic's Opus 5.5 prompting page. ## Summary Checklist @@ -154,6 +162,8 @@ When designing a skill that dispatches agents: 6. Cap teams at 5 agents. Beyond this, coordination costs exceed benefits. 7. Apply the 45% threshold: optimize existing agents before adding new ones. 8. Dispatch independent agents in parallel. Avoid long sequential chains. +9. When fanning out, check each subagent's cited evidence before accepting its finding. +10. Wait for every background dispatch before treating the run as complete. ## Sources @@ -168,6 +178,10 @@ When designing a skill that dispatches agents: - [Prompting Claude Opus 5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5). Source for readier default delegation, the rule against delegating self-verification, and the advice to prefer one agent over several and keep spawn counts low. +- [Getting the most out of Opus 5.5 in Claude and Claude Code (Anthropic blog, Addy Osmani, 2026-09-22)](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/). + Source for fanning work out to one subagent per unit and checking each one's evidence before accepting it. +- [Prompting Claude Opus 5.5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5). + Source for not treating a task as done while a background subagent is still running. Cross-references: diff --git a/han-plugin-builder/skills/guidance/references/claude-marketplace-and-plugin-configuration/plugin-json-options.md b/han-plugin-builder/skills/guidance/references/claude-marketplace-and-plugin-configuration/plugin-json-options.md index 50c58be2..22ee96a5 100644 --- a/han-plugin-builder/skills/guidance/references/claude-marketplace-and-plugin-configuration/plugin-json-options.md +++ b/han-plugin-builder/skills/guidance/references/claude-marketplace-and-plugin-configuration/plugin-json-options.md @@ -140,6 +140,29 @@ Available for substitution in skill/agent content, hook commands, monitor comman Both are also exported as environment variables to hook processes and MCP/LSP server subprocesses. +### Keeping a skill's state across updates + +A skill shipped in a plugin sometimes needs to remember something between runs. Two uses come up most: + +- **First-run setup.** The skill needs a value from the user once, such as which channel to post to. Read a saved + answer from `${CLAUDE_PLUGIN_DATA}` first. When none exists, ask with `AskUserQuestion` (kept out of `allowed-tools`; + see [allowed-tools: AskUserQuestion](../skill-building-guidance/allowed-tools-AskUserQuestion.md)) and save the + answer there. +- **Run memory.** The skill reads its own history, such as the last report it posted, to say what changed since. Keep + an append-only log or a JSON file in `${CLAUDE_PLUGIN_DATA}` and read it at the start of the next run. + +Write this state to `${CLAUDE_PLUGIN_DATA}`, never to the skill's own directory. The skill directory sits under +`${CLAUDE_PLUGIN_ROOT}`, which is replaced when the plugin updates, so anything saved there is lost. + +Three cases call for something else: + +- **A value known at install time** belongs in [`userConfig`](#userconfig), which prompts for it when the plugin is + enabled. +- **A file the user is expected to edit by hand** belongs where the user edits it, such as the project or their own + configuration directory, not in a data directory they never open. +- **A skill checked into a repository's `.claude/skills/`**, including one vendored by `guidance init`, has no plugin + and so no `${CLAUDE_PLUGIN_DATA}`. It keeps its state in the repository or asks each run. + ## Complete Example ```json diff --git a/han-plugin-builder/skills/guidance/references/per-model-authoring.md b/han-plugin-builder/skills/guidance/references/per-model-authoring.md index 10a381ce..dbc63620 100644 --- a/han-plugin-builder/skills/guidance/references/per-model-authoring.md +++ b/han-plugin-builder/skills/guidance/references/per-model-authoring.md @@ -1,11 +1,29 @@ # Per-Model Authoring Guidance -Write your skill and agent instructions to be model-agnostic by default. When you know the model that will run them, three of Anthropic's models differ enough in how they follow instructions that you should adjust how you write. This document tells you when that adjustment is worth making and what it is. +## Contents -_Last checked against Anthropic's published guidance on 2026-07-31, for Sonnet 5, Opus 5, and Fable 5. The per-model behavior below comes from Anthropic's own prompting pages (see Sources). Treat it as current only as of that date: those pages are pinned snapshots that get revised and archived as new models ship._ +- Default to model-agnostic instructions +- What "model-agnostic" means when you do not know the target +- The difference that changes how you write: instruction style +- The difference that can cause a failure: reasoning echo on every current model +- Other settings the model differences affect: thinking mode, effort, and subagent eagerness +- Instructions to leave out on Opus 5 and Opus 5.5, and to keep on Sonnet 5.5 and Haiku 5.5 +- Calibrating length, narration, and scope on Opus 5 and Opus 5.5 +- Early stops and scope on Sonnet 5.5 and Haiku 5.5 +- What this guidance does not cover +- Cross-References +- Sources + +Write your skill and agent instructions to be model-agnostic by default. When you know the model that will run them, Anthropic's models differ enough in how they follow instructions that you should adjust how you write. This document tells you when that adjustment is worth making and what it is. + +_Last checked against Anthropic's published guidance on 2026-10-15, for Sonnet 5, Sonnet 5.5, Haiku 5.5, Opus 5, Opus 5.5, Fable 5, and Fable 5.1. The per-model behavior below comes from Anthropic's own prompting pages (see Sources). Treat it as current only as of that date: those pages are pinned snapshots that get revised and archived as new models ship._ This is author-time guidance. It shapes how you write instructions, not how a skill behaves while it runs. +Opus 5.5 inherits the Opus 5 guidance below except where a section says otherwise. Anthropic's Opus 5.5 prompting page says that existing Opus 5 prompts should perform well on Opus 5.5 without changes, so wherever this document says "Opus 5" without naming Opus 5.5, the advice holds for both. + +The other 5.5 models follow the same pattern. Anthropic says existing Sonnet 5 prompts should perform well on Sonnet 5.5, existing Haiku 4.5 prompts on Haiku 5.5 (there is no Haiku 5), and existing Fable 5 prompts on Fable 5.1. So wherever this document says "Sonnet 5" or "Fable 5", the advice holds for its successor too, except where a section names Sonnet 5.5 or Haiku 5.5 on its own. + A skill cannot reliably detect which model is running it, so do not try to branch skill content on the active model. Claude Code exposes no reliable model signal to a skill, and asking a model to name itself is unreliable. The source research covers the reasons in more detail. Keep your shipped skills model-agnostic, and act on the differences below as you write them. ## Default to model-agnostic instructions @@ -16,39 +34,54 @@ Reach for per-model tuning only when you have a specific reason: you know the ta ## What "model-agnostic" means when you do not know the target -The three models pull in opposite directions on how much to spell out (see the next section), so "write model-agnostic" needs a concrete meaning. Here it is: lead with the goal and the reasons behind it, state the load-bearing constraints and scope explicitly, and skip the exhaustive step-by-step micro-checklist. +The models pull in opposite directions on how much to spell out (see the next section), so "write model-agnostic" needs a concrete meaning. Here it is: lead with the goal and the reasons behind it, state the load-bearing constraints and scope explicitly, and skip the exhaustive step-by-step micro-checklist. -This middle path serves all three models. Stating the goal and the reasons gives Fable 5 the context it uses well, without the checklist that degrades its output. Stating the load-bearing constraints explicitly gives Opus 5 and Sonnet 5 the scope they need on the behaviors that matter. You are not writing to the lowest common denominator; you are giving each model what it needs and withholding what hurts one of them. +This middle path serves all of them. Stating the goal and the reasons gives Fable 5 the context it uses well, without the checklist that degrades its output. Stating the load-bearing constraints explicitly gives Opus 5, Opus 5.5, Sonnet 5, and Sonnet 5.5 the scope they need on the behaviors that matter. You are not writing to the lowest common denominator; you are giving each model what it needs and withholding what hurts one of them. ## The difference that changes how you write: instruction style -This is the one difference worth acting on. Opus 5 and Sonnet 5 follow instructions literally and do not generalize on their own, so they want each behavior spelled out and the scope stated. +This is the one difference worth acting on. Opus 5 and Sonnet 5 follow instructions literally and do not generalize on their own, so they want each behavior spelled out and the scope stated. Anthropic's Sonnet 5 page says the literal reading is strongest at lower effort. Neither the Sonnet 5.5 nor the Haiku 5.5 page restates this, so treat Sonnet 5.5 as inheriting it through the carried-over Sonnet 5 prompts, not as a claim Anthropic verified for 5.5. -Fable 5 runs the other way. A short, goal-based instruction works better for it, and spelling out every step with a checklist actively degrades its output. +Fable 5 runs the other way. Anthropic says you can steer most behaviors on it "with a brief instruction rather than enumerating each behavior by name", and that skills written for earlier models "are often too prescriptive" and "can degrade output quality". -Because the two directions are opposite, a skill written to one model's guidance points the other the wrong way. If you tune a skill for Fable 5 with terse goals and then it runs on Opus 5, the Opus run may under-specify. If you tune for Opus 5 with an exhaustive checklist and then it runs on Fable 5, the Fable run may degrade. +Because the two directions are opposite, a skill written to one model's guidance points the other the wrong way. If you tune a skill for Fable 5 with terse goals and then it runs on Opus 5, the Opus run may under-specify. If you tune for Opus 5 by enumerating every behavior and then it runs on Fable 5, the Fable run may degrade. Literal instruction-following cuts both ways, and on Opus 5 the cost of a careless limiting phrase is high. A review instruction that says "only report high-severity issues" or "be conservative" is followed literally, and the run reports less than it found. When you want a filtered result, have the skill report everything and filter in a separate step, rather than narrowing what the model is allowed to notice in the first place. +A limit is not the problem on its own; a vague one is. A limit is fine when it names a bar the reader could check, such as a consequence ("would block the merge") or a rubric the skill itself carries, and it requires evidence for each item it keeps. Anthropic's Opus 5.5 blog post recommends a review prompt of exactly that shape: + +> List only problems you'd block the merge for. For each one, give the file and line, why it's wrong, and how to show it fails. + +Compare "Only report high-severity issues", which names no bar anyone could check and asks for no evidence. Keep the report-then-filter approach for a limit like that one. + When you know the target, match its style. When you do not, use the model-agnostic middle above. -## The difference that can cause a failure: Fable 5 and reasoning echo +## The difference that can cause a failure: reasoning echo on every current model + +On every current Anthropic model, an instruction that tells the model to reproduce its own internal reasoning in its visible answer can be refused outright, with the `reasoning_extraction` refusal category. Anthropic's refusals page lists the classifiers on Fable 5.1, Fable 5, Opus 5.5, Opus 5, Sonnet 5.5, and Haiku 5.5, and the Fable 5, Opus 5, Opus 5.5, and Sonnet 5.5 prompting pages each name the category. This is the one difference here that causes a functional failure rather than a stylistic mismatch, so it is worth recognizing on sight. A skill whose agent hits it does not degrade; the request comes back declined, and server-side fallback does not retry it. -On Fable 5, an instruction that tells the model to reproduce or transcribe its own internal thinking into its visible answer can be refused outright. This is the one difference here that causes a functional failure rather than a stylistic mismatch, so it is worth recognizing on sight. +Haiku 5.5 is the one unclear case: its own prompting page lists four refusal categories and leaves this one out, while the refusals page puts Haiku 5.5 in scope. Write as if it applies. -Use this test to tell when the pattern is present. It applies when an instruction tells the model to copy its own internal reasoning or thinking into the deliverable it returns. It does not apply when you ask the model to write a normal explanation of a decision, or to produce a reasoned answer as ordinary content. Asking for an explanation written for the reader is fine; asking the model to echo its private thinking verbatim is the pattern to avoid. +The pattern is wider than "echo your thinking". Anthropic lists four shapes that trigger it, and says the wording can sit in a system prompt, a skill, or a tool description: -This warning rests on a single Anthropic source and is not independently corroborated (see Sources). It is a documented product behavior, not a subjective style claim, so it is worth heeding. Weigh it knowing the evidence is single-source and single-vendor. +- A ``, ``, or scratchpad section the model fills in before it answers. +- A `reasoning`, `thinking`, or `trace` field in JSON output or in a tool input. An agent output format with a `reasoning` field is this pattern. +- The model's private notes, or a running log of its reasoning. +- Reasoning asked for verbatim or in full. + +Asking for an explanation written for the reader is fine. A short explanation of the answer, the evidence behind a result, or a summary of the actions taken all stay safe. Name the field for what the reader gets (`rationale`, `evidence`, `why`) and ask for it in a sentence or two, rather than for the model's reasoning. + +These are all Anthropic publications, so the evidence is single-vendor rather than independently corroborated. It is a documented product behavior, not a subjective style claim, so it is worth heeding. ## Other settings the model differences affect: thinking mode, effort, and subagent eagerness -These three differences rarely decide how you write on their own, but each changes a specific choice: +These differences rarely decide how you write on their own, but each changes a specific choice: -- **Thinking mode.** The three models default differently. Opus 5 and Sonnet 5 have thinking on by default; on Opus 5 it can be disabled only at effort `high` or below. Fable 5 always has it on and cannot turn it off. So do not write "think step by step" prompt hacks or instructions that assume you can toggle thinking. Set the behavior you want through the model's own controls, not through prose that fights the default. Never write a rule telling the model not to think or not to reason: on Opus 5 with thinking disabled, that kind of rule increases the chance internal XML tags leak into the visible response. -- **Effort.** The reasoning-depth lever is the effort setting, not "think harder" phrasing in your instructions. The same effort label does not mean the same depth across the three models, so do not hardcode an assumption that a given level produces a fixed amount of reasoning. On Opus 5, `low` and `medium` hold quality on most work at a fraction of the tokens and latency, and `xhigh` is reserved for demanding coding and agentic work. Effort governs how much the model thinks, not how much it says, so lowering it will not reliably shorten a response. -- **Subagent eagerness.** The three models reach for subagents with different eagerness, and Opus 5 delegates more readily than earlier models. If your skill dispatches subagents, state the delegation you want rather than relying on the model's default tendency. [Multi-Agent Economics](./agent-building-guidelines/multi-agent-economics.md) covers the delegation rules that follow from this. +- **Thinking mode.** The models default differently. Opus 5 and Sonnet 5 have thinking on by default; on Opus 5 it can be disabled only at effort `high` or below. In Claude Code, Opus 5.5, Sonnet 5.5, Haiku 5.5, and the Fable models always have it on, and the thinking toggle has no effect on them. (On the API, Haiku 5.5 can still disable thinking at `high` or below, and Sonnet 5.5's lowest setting is `between_tools`, which skips thinking before the first tool call. Neither changes what a skill author writes.) So do not write "think step by step" prompt hacks or instructions that assume you can toggle thinking. Set the behavior you want through the model's own controls, not through prose that fights the default. Never write a rule telling the model not to think or not to reason: on Opus 5 with thinking disabled, that kind of rule increases the chance internal XML tags leak into the visible response. That leak is an artifact of disabled thinking, so it does not arise on Opus 5.5, but the rule still has no place in a skill. +- **Effort.** The reasoning-depth lever is the effort setting, not "think harder" phrasing in your instructions. The same effort label does not mean the same depth across models, so do not hardcode an assumption that a given level produces a fixed amount of reasoning. On Opus 5, `low` and `medium` hold quality on most work at a fraction of the tokens and latency, and `xhigh` is reserved for demanding coding and agentic work. On Opus 5.5 the same name means more thinking than it did on Opus 5: Anthropic's platform docs put Opus 5.5's API default at `medium`, where Opus 5's is `high`, and report `medium` on Opus 5.5 matching or beating Opus 5 at `high`. Reserve `xhigh` and `max` for work where you have measured a gain. Sonnet 5.5's levels were recalibrated against Sonnet 5 in the same way: the same name no longer produces the same amount of thinking. Haiku 5.5 is the first Haiku with effort levels at all. In Claude Code, Opus 5.5, Sonnet 5.5, and Haiku 5.5 all default to `medium`; on the API, Sonnet 5.5 defaults to `high`. If a skill or agent pins `effort` in its frontmatter and that value was tuned for an earlier model, re-test it on the 5.5 model rather than carrying it over, and say in the body why the value is pinned. Effort governs how much the model thinks, not how much it says, so lowering it will not reliably shorten a response. +- **Subagent eagerness.** The models reach for subagents with different eagerness, and Opus 5 delegates more readily than earlier models. If your skill dispatches subagents, state the delegation you want rather than relying on the model's default tendency. [Multi-Agent Economics](./agent-building-guidelines/multi-agent-economics.md) covers the delegation rules that follow from this. -## Instructions to leave out on Opus 5 +## Instructions to leave out on Opus 5 and Opus 5.5, and to keep on Sonnet 5.5 and Haiku 5.5 Opus 5 already does several things that older prompts told the model to do. Leaving those instructions in does not reinforce the behavior; it compounds with it and burns tokens for no quality gain. When you write for Opus 5, or when you inherit a skill written for an older model, cut the following. @@ -56,7 +89,9 @@ Opus 5 already does several things that older prompts told the model to do. Leav - **Re-check and double-check prompts.** "Double-check your answer" and "re-verify before responding" fall in the same category. Opus 5 catches and fixes its own mistakes well, and these phrasings add cost without improving the result. - **Rules that forbid thinking or reasoning.** Covered in the thinking-mode bullet above. -## Calibrating length, narration, and scope on Opus 5 +The verification items above are Opus advice. Do not carry them to Sonnet 5.5 or Haiku 5.5. Sonnet 5.5 generally checks its work, but at `low` effort it sometimes reports a code change done without running a check; Haiku 5.5 does the same at `low` and `medium`. Anthropic's pages for both supply a paragraph to add rather than remove: before reporting a code change done, run a real check that exercises it (the project's tests, type-checker, or build) and, if none can run, say which one was skipped and why. Fable 5 also asks for explicit self-verification on long runs. A model-agnostic skill that changes code and may run at low effort should keep a one-line verification instruction; on Opus it costs a little, and on the others it closes a real gap. + +## Calibrating length, narration, and scope on Opus 5 and Opus 5.5 Opus 5 runs longer and narrates more than earlier models, and it will extend a task's scope on its own judgment. None of these are fixed by lowering effort; each needs an explicit instruction. Write the behavior you want positively, describing the shape you are after rather than listing what to avoid, which works better on this model. @@ -64,8 +99,19 @@ Opus 5 runs longer and narrates more than earlier models, and it will extend a t - **Written deliverables.** Files a skill writes to disk run long on Opus 5 too, and that is a separate lever from conversational verbosity. A skill that produces a report or a document should tell the model to match length to what the task needs and to skip filler sections, redundant summaries, and boilerplate. - **Progress narration.** Describe the cadence you want rather than leaving it to the default: one sentence before the first tool call, brief updates only on a real finding or a change of direction, and a closing message that leads with the outcome before the supporting detail. - **Correction narration.** Opus 5 announces corrections to its own earlier statements more than earlier models. When that noise matters, instruct it to correct an earlier statement only when the error changes the reader's code, conclusions, or decisions, and to fix silent slips without narrating them. +- **Early stops on long runs.** On a long task with several parts, Opus 5.5 keeps the user posted, and some of those updates end the turn with a report instead of the next step. It responds to instructions that name the stops you do not want and the ones you do. [Workflow Patterns](./skill-building-guidance/workflow-patterns.md#naming-the-stops-in-an-autonomous-stretch) shows how, and when to leave the instruction out. - **Scope.** For a narrow task, constrain scope in the skill body: deliver what was asked at the scope intended, make routine judgment calls without checking in, and say so in a sentence and continue when the request looks mistaken rather than quietly reshaping it. +## Early stops and scope on Sonnet 5.5 and Haiku 5.5 + +The two newer tiers stop early and stretch scope in their own ways, mostly tied to effort. These matter for a skill that runs long or changes code. + +- **Sonnet 5.5 checks in early at `low` and `medium`.** On long agentic tasks at those levels it is more likely to stop and ask before it finishes. Raising effort helps; so does a sentence telling it to keep working until everything asked for is done and to stop only when it cannot go on without the user or before a risky step. +- **Sonnet 5.5 adds work it was not asked for.** It adds tests, docs, and small supporting files at every effort level, more at higher effort. When a skill wants changes limited to the request, say that once it is done and checked it should stop and report, and mention any addition it thinks would help instead of making it. +- **Sonnet 5.5 starts its own review rounds at `xhigh` and `max`.** At those levels it can launch reviewer subagents after it finishes. Anthropic reports that a sentence telling it not to start extra review rounds unless asked cut session cost by about a third with no change in quality. Prefer running routine work at `high` or below. +- **Haiku 5.5 stops early on long prompts at `low`.** With a short system prompt it rarely stops early; with a long coding-agent prompt at `low` it sometimes hands the task back. The same keep-working sentence helps, and so does `medium` effort, which in Anthropic's testing roughly halved early stops while more than doubling output tokens. Keep a Haiku agent's prompt short. +- **Asking any of them to think less in prose does not work.** Lower the effort instead. + ## What this guidance does not cover This document is about how to write instructions for a model. It is not about which model tier to run. For the tier question (opus, sonnet, or haiku, and at what effort), read [Specialization and Model Selection](./specialization-and-model-selection.md). @@ -80,10 +126,23 @@ It also does not cover run-time model detection or per-model skill variants. Tho ## Sources -The per-model behavior above comes from Anthropic's own prompting pages, plus this suite's own research report, `model-specific-guidance-for-skills.md`, which gathered and adversarially validated the instruction-style and Fable 5 claims. The Fable 5 reasoning-echo refusal is single-source on the Fable 5 page. +The per-model behavior above comes from Anthropic's own prompting pages, plus this suite's own research report, `model-specific-guidance-for-skills.md`, which gathered and adversarially validated the instruction-style and Fable 5 claims. The reasoning-echo refusal, single-source on the Fable 5 page when that report was written, is now stated on the Fable 5, Opus 5, Opus 5.5, and Sonnet 5.5 prompting pages and on the refusals page. The Opus 5 material (the instructions to leave out, the length and narration and scope calibration, the effort recommendations, the readier delegation, and the thinking defaults) comes directly from the Opus 5 prompting page and postdates that research report. It is single-vendor and not independently corroborated. +The Opus 5.5 material (the inherited Opus 5 guidance, the thinking and effort changes, the acceptable review limit, and the early-stop behavior) comes from the Opus 5.5 prompting page and Anthropic's Opus 5.5 blog post, and postdates that research report too. The two agree with each other, but both are Anthropic's, so this material is also single-vendor. The review-limit example appears only in the blog post. + +The Sonnet 5.5, Haiku 5.5, and Fable 5.1 material, the widened reasoning-echo section, and the Claude Code thinking and effort defaults come from the pages below as of 2026-10-15, gathered and adversarially validated in `docs/plans/opus-5-5-plugin-builder-guidance/research/claude-5-5-family-guidance.md`. It is single-vendor as well. + +- [Prompting Claude Opus 5.5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5) +- [Getting the most out of Opus 5.5 in Claude and Claude Code (Anthropic blog, Addy Osmani, 2026-09-22)](https://claude.dev/blog/getting-the-most-out-of-opus-5-5/) +- [Prompting Claude Sonnet 5.5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5) +- [What's new in Claude Sonnet 5.5 (Anthropic)](https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5) +- [Prompting Claude Haiku 5.5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5) +- [Prompting Claude Fable 5.1 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-1) +- [Refusals and fallback (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback) +- [Effort (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/effort) +- [Claude Code model configuration](https://code.claude.com/docs/en/model-config) - [Prompting Claude Opus 5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5) - [Prompting Claude Sonnet 5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5) - [Prompting Claude Fable 5 (Anthropic)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5) diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/allowed-tools-AskUserQuestion.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/allowed-tools-AskUserQuestion.md index 48a1b2ad..3edb8215 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/allowed-tools-AskUserQuestion.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/allowed-tools-AskUserQuestion.md @@ -80,5 +80,9 @@ require interactive user input — they are mode toggles called by the agent. No 3. `AskUserQuestion` will still work — it just won't be auto-approved 4. This is a workaround for Claude Code bug #29547; revisit when the fix ships -Cross-reference: [Context Injection Commands](./context-injection-commands.md) for related `allowed-tools` formatting -guidance (separate `Bash()` entries). +Cross-references: + +- [Context Injection Commands](./context-injection-commands.md) for related `allowed-tools` formatting guidance + (separate `Bash()` entries). +- [Keeping a skill's state across updates](../claude-marketplace-and-plugin-configuration/plugin-json-options.md#keeping-a-skills-state-across-updates) + for asking a first-run question once and saving the answer. diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/hardening-fuzzy-vs-deterministic.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/hardening-fuzzy-vs-deterministic.md index 2093a5bc..d7e886c7 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/hardening-fuzzy-vs-deterministic.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/hardening-fuzzy-vs-deterministic.md @@ -5,6 +5,14 @@ paths: # Hardening: Fuzzy vs. Deterministic Steps +## Contents + +- The Fuzzy-Deterministic Spectrum +- Recognition Signals +- The Hardening Process +- When NOT to Harden +- Summary Checklist + Other guidance docs tell you to extract deterministic operations to scripts ([Progressive Disclosure](./progressive-disclosure.md), [Writing Effective Instructions](./writing-effective-instructions.md)). This doc provides the decision framework for @@ -39,6 +47,14 @@ git commands with exact flags, escaping strings, computing checksums. **Where they belong:** Shell scripts in `scripts/`. See [Script Execution Instructions](./script-execution-instructions.md) for the invocation pattern. +**A library instead of a fixed script.** When a skill answers open-ended questions over the same data source ("what +happened on Tuesday?"), one fixed script per question does not scale. Put a small library of helper functions in +`scripts/` instead, each doing one deterministic thing (fetch the events, join the tables, compute the rate), with its +gotchas written in the function's own comments. The model then writes a one-off script that loads the library and +composes the helpers for the question at hand, spending its turns on the composition rather than on rebuilding the +plumbing. Each composed script is new, so it still prompts for permission when it runs; see +[Script Execution Instructions](./script-execution-instructions.md). + ### Hybrid Steps Require judgment to decide _what_ to do, then mechanical execution to _do it_. Split these into two sub-steps: the fuzzy diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/progressive-disclosure.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/progressive-disclosure.md index d4b38eaf..6c25eb75 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/progressive-disclosure.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/progressive-disclosure.md @@ -99,6 +99,8 @@ decision matrices — belongs in `references/`. - Rate tables, formulas, or scoring matrices (pricing tables, complexity scores, risk assessments) - Style guides or brand standards (voice guidelines, formatting rules) - Decision matrices with multiple criteria +- Gotchas: failure points found in use, which grow over the skill's life (see + [Skill Reference Files](./skill-reference-files.md)) **When to keep in SKILL.md:** @@ -113,6 +115,10 @@ decision matrices — belongs in `references/`. - Rules already enforced by the toolchain (linters, formatters, CI checks). If a linter catches it, documenting it in SKILL.md wastes attention budget and risks contradiction when the linter config changes. Reserve SKILL.md instructions for judgment calls — decisions that require context, tradeoffs, or domain knowledge that no automated tool checks. +- Instructions that restate what the model already does by default, such as "read the code before changing it" or + "check your work for errors." They add tokens without changing the output. Keep what pushes the model away from its + default, and cut what describes the default. See + [Context Hygiene](./context-hygiene.md#rule-every-token-must-earn-its-place). **Before (restating what the toolchain enforces):** diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-frontmatter-fields.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-frontmatter-fields.md index 8ef94288..fea3b083 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-frontmatter-fields.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-frontmatter-fields.md @@ -40,14 +40,14 @@ doc and that page disagree, the official page wins; re-verify before relying on ## Execution and model -| Field | Required | What it does | -| --------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `model` | No | Overrides the model for this skill's turn only; not saved to session settings. | -| `effort` | No | `low`, `medium`, `high`, `xhigh`, or `max`. Overrides the session's effort level for this skill. | -| `context` | No | Set to `fork` to run the skill in an isolated subagent context. See the caveat in [Skill Composition](./skill-composition.md) about forked data-fetch sub-skills. | -| `agent` | No | When `context: fork` is set, names which subagent type runs the skill (`Explore`, `Plan`, `general-purpose`, or a custom subagent). | -| `hooks` | No | Lifecycle hooks scoped to this skill's run. | -| `shell` | No | `bash` (default) or `powershell` for inline command blocks. | +| Field | Required | What it does | +| --------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `model` | No | Overrides the model for this skill's turn only; not saved to session settings. | +| `effort` | No | `low`, `medium`, `high`, `xhigh`, or `max`. Overrides the session's effort level for this skill. The same level means more thinking on some models than others; see [Per-Model Authoring](../per-model-authoring.md#other-settings-the-model-differences-affect-thinking-mode-effort-and-subagent-eagerness) before pinning one. | +| `context` | No | Set to `fork` to run the skill in an isolated subagent context. See the caveat in [Skill Composition](./skill-composition.md) about forked data-fetch sub-skills. | +| `agent` | No | When `context: fork` is set, names which subagent type runs the skill (`Explore`, `Plan`, `general-purpose`, or a custom subagent). | +| `hooks` | No | Lifecycle hooks scoped to this skill's run. A hook here lasts only while the skill runs, which suits a guardrail you want only then, such as blocking destructive shell commands. See the [hooks reference](https://code.claude.com/docs/en/hooks) for the schema. | +| `shell` | No | `bash` (default) or `powershell` for inline command blocks. | ## Portability (open-standard fields) diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-reference-files.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-reference-files.md index 367fb19f..006ce769 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-reference-files.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/skill-reference-files.md @@ -35,6 +35,11 @@ Extract content to `references/` when it represents domain knowledge rather than - **Style guides** that define standards (voice guidelines, formatting rules, naming conventions) - **Canonical examples** that demonstrate conventions the skill enforces (2-3 representative "do this / not this" code samples per convention) +- **Gotchas** that record failure points found in use: the places the model goes wrong while doing the skill's work, + each with the fact that corrects it ("the subscriptions table is append-only; the current row has the highest + `version`, not the latest `created_at`"). A gotchas list differs from a checklist, which says what to verify, and from + a canonical example, which shows what a convention looks like. It starts small and grows each time the skill hits a + new failure, and it is often the highest-signal content a skill carries. Reference files are not passive lookups — they are demonstration material the model pattern-matches against during execution. Every example in a reference file functions as a few-shot demonstration that calibrates the model's output. diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/success-criteria-and-testing.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/success-criteria-and-testing.md index 650889d5..4cd1e690 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/success-criteria-and-testing.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/success-criteria-and-testing.md @@ -10,6 +10,7 @@ paths: - Triggering Tests - Functional Tests - Performance Comparison +- Building a Verification Skill - Summary Checklist How do you know a skill is working? Without defined success criteria, "it seems fine" becomes the bar — and that bar @@ -20,9 +21,11 @@ Testing rigor should match the skill's audience. A skill used by one person inte deployed across an organization. But every skill benefits from at least running through the triggering tests and a few functional scenarios. -**Test against the model the skill targets.** Anthropic notes that smaller, faster models (Haiku) may need more -detailed, explicit instructions than larger ones to follow a skill reliably, while a larger model can fill gaps a terse -skill leaves. A skill that passes on Opus can still fail on Haiku. If a skill is meant to run on a faster tier (or to be +**Test against the model the skill targets.** A smaller, faster model can behave differently on the same skill. Haiku +5.5, for example, sometimes stops early on a long prompt at `low` effort and sometimes reports a code change done +without checking it, where Opus checks its own work unprompted (see +[Per-Model Authoring Guidance](../per-model-authoring.md#early-stops-and-scope-on-sonnet-55-and-haiku-55)). A skill that +passes on Opus can still fail on Haiku. If a skill is meant to run on a faster tier (or to be dispatched into a Haiku subagent), run the triggering and functional tests on that tier, not just on your session's default. See [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices). @@ -178,6 +181,24 @@ Beyond the numbers, watch for: - **Workflows complete without user correction** — steps execute in the right order with the right tools - **Consistent results across sessions** — a new user gets the same quality as an experienced one +## Building a Verification Skill + +Everything above tests a skill. A verification skill is a different thing: a skill whose job is to test the product, +by driving it the way a user would and checking that it behaves. Anthropic reports that verification skills had the +most measurable effect on Claude's output quality of any kind of skill it built, so they are worth the extra effort. + +- **Pair the skill with a driver.** The skill does not click or type itself. Give it an external tool that does, such + as browser automation for a web flow or a terminal driver for an interactive CLI, and keep the driving code in + `scripts/`. +- **Assert state at every step, not once at the end.** A final screenshot can look right while an earlier step failed + silently. Check the state the step should have produced (the database row, the page element, the exit code) before + moving on, so a failure names the step where it happened. +- **Record what was tested.** Where the driver supports it, save a recording or a step log of the run, so a person can + see what the skill exercised rather than trusting its summary. + +A verification skill still gets the triggering and functional tests above. They check that the skill runs; the skill +checks that the product works. + ## Summary Checklist 1. Build triggering tests: obvious triggers, paraphrased triggers, and unrelated prompts that should NOT trigger @@ -187,6 +208,7 @@ Beyond the numbers, watch for: 5. Run functional tests 3-5 times to check consistency 6. Compare performance with and without the skill: messages, tool calls, tokens, errors, corrections 7. Match testing rigor to the skill's audience and visibility +8. For a skill that verifies a product, pair it with a driver, assert state at every step, and record the run Cross-references: diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/workflow-patterns.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/workflow-patterns.md index 5bd6a7c9..2e0bed35 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/workflow-patterns.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/workflow-patterns.md @@ -14,6 +14,8 @@ paths: - Domain-Specific Intelligence - Combining Patterns - Human Gates in Workflow Steps +- Naming the Stops in an Autonomous Stretch +- Bounding a Step That Works Through an Unknown Number of Items - Ordering Within Steps: Recency Bias - Summary Checklist @@ -358,6 +360,79 @@ Present the review summary to the user and ask for confirmation before posting. manually. ``` +## Naming the Stops in an Autonomous Stretch + +Some skills contain a stretch meant to run without the user: a fan-out across many files, a long tool-calling loop, a +migration that works through every call site. Newer models keep the user posted as they work, and on a long stretch +some of those updates end the turn instead of taking the next step. The run stops, and nothing moves until someone +types "continue." [Per-Model Authoring](../per-model-authoring.md) covers which models do this. + +Name the stops in the step that starts the stretch. First, the ones you do not want: + +- A summary that announces the next step instead of taking it. +- An offer to carry on, which waits for an answer the user was not going to give. +- A list of choices for the user when none of them blocks the rest of the work. + +Then the ones you do want, so the instruction does not read as "never stop": + +- No work can move without the user's input. +- The next action is destructive or cannot be undone. A keep-going instruction never overrides a human gate. +- The stretch is finished. + +Status notes are welcome. Ask for them in the same message as the next action, so reporting never ends the turn. + +**Before (no stops named):** + +```markdown +## Step 3: Migrate Call Sites + +Update every call site of `oldClient` to use `newClient`. +``` + +**After (unwanted and wanted stops named):** + +```markdown +## Step 3: Migrate Call Sites + +Update every call site of `oldClient` to use `newClient`. When a call site does not need the user's input, keep going, +and put any status note in the same message as your next edit. Do not stop to summarize progress, to offer to +continue, or to list choices that block nothing. Stop only when a call site cannot be migrated without a decision from +the user, before deleting anything, or when every call site is done. +``` + +### When not to use it + +Leave the keep-going instruction out of any skill built to hand control back. An interview that asks one question per +turn, a collaborative mode that pauses for review after each piece of work, and every human gate are stops the design +wants. For work where someone is there to answer, Anthropic's guidance points the other way: a one-line plan before +starting and a short recap at the end. Scope the instruction to the stretch meant to run unattended, never to the whole +skill. + +## Bounding a Step That Works Through an Unknown Number of Items + +A fixed step ends when its instructions end. A step that works through a set whose size is not known up front (every +changed file, every endpoint, every finding a review turned up) has no natural last line. On a long run, the model can +also lose track of what is left once older turns are summarized. Two instructions keep the step bounded: + +- **State the finish line in checkable terms.** "Done means every call site uses `newClient` and the test suite passes" + tells the model when to stop. "Migrate the call sites" does not. +- **Keep the checklist in a file.** Have the step write its items to a file, check each one off as it finishes, and add + any new item it discovers. To decide what remains, it reads the file, not the conversation. A file survives context + summarization; earlier turns do not. + +```markdown +## Step 2: Fix Each Finding + +Write every finding from Step 1 to `{output_dir}/tasks.md` as an unchecked item. Work through them in order, checking +each one off as it is fixed and adding any new finding you discover. Done means every item in `tasks.md` is checked and +the test suite passes. Before stopping, re-read `tasks.md`. If an item is still open and nothing blocks it, continue +with it. +``` + +A fixed flowchart step needs neither. For the moment right after a sub-skill returns, see +[Skill Composition](./skill-composition.md#instruct-continuation-explicitly-after-the-skill-call). For items handed to +subagents, see [Multi-Agent Economics](../agent-building-guidelines/multi-agent-economics.md). + ## Ordering Within Steps: Recency Bias Within any step, the model weights the last instruction or example more heavily than earlier ones. When a step contains @@ -411,6 +486,8 @@ Check each changed file against: 6. Include validation gates, bounded loops, decision criteria, or audit trails as appropriate 7. Combine patterns by using one as the primary structure and embedding others within steps 8. Within a step, place the most critical instruction or most representative example last (recency bias) +9. In a stretch meant to run without the user, name the stops you do not want and the ones you do +10. Give a step that works through an unknown number of items a checkable finish line and a checklist file Cross-references: diff --git a/han-plugin-builder/skills/guidance/references/skill-building-guidance/writing-effective-instructions.md b/han-plugin-builder/skills/guidance/references/skill-building-guidance/writing-effective-instructions.md index 7145e060..203de01c 100644 --- a/han-plugin-builder/skills/guidance/references/skill-building-guidance/writing-effective-instructions.md +++ b/han-plugin-builder/skills/guidance/references/skill-building-guidance/writing-effective-instructions.md @@ -9,6 +9,7 @@ paths: - The Rules - Rule: Be specific and actionable +- Rule: Spell out what breaks when missed, and leave out what the model gets right unprompted - Rule: Write constraints with embedded reasoning - Rule: Include error handling in instructions - Rule: Prefer inline discovery over forked data-fetch sub-skills @@ -79,6 +80,39 @@ For each changed file in the diff: 4. Include the file path and line number for each finding ``` +### Rule: Spell out what breaks when missed, and leave out what the model gets right unprompted + +Specificity has a limit. A skill packed with blanket rules ("never write multi-line comments", "always add a +docstring") pins the model to one answer where the right answer depends on the code in front of it, and rules from +different layers start to conflict. Anthropic removed most rules of that kind from Claude Code's own system prompt for +the Claude 5 generation of models, with no measurable loss on its coding evaluations, and replaced them with guidance +the model applies to its surroundings. + +Spend specificity where a miss breaks something: a file format another tool parses, a command whose flags must be +exact, a convention the project enforces, an action that cannot be undone. For routine details the model already gets +right unprompted, leave the instruction out, or state the goal and the reason instead of a rule. + +This rule governs how much detail each instruction carries, not whether a step is fixed. A skill's steps still form a +flowchart ([Entity Taxonomy](../plugin-entity-taxonomy.md#skills-process-engine)); only the prose inside a step gets +lighter. For the same advice framed around an unknown target model, see +[Per-Model Authoring](../per-model-authoring.md#what-model-agnostic-means-when-you-do-not-know-the-target). + +**Before (blanket rules):** + +```markdown +## Step 4: Write the Fix + +Never write comments longer than one line. Always use early returns. Never create helper functions. +``` + +**After (the goal the rules were standing in for):** + +```markdown +## Step 4: Write the Fix + +Write code that reads like the surrounding code: match its comment density, naming, and idiom. +``` + ### Rule: Write constraints with embedded reasoning When a skill instruction prohibits or requires something, include the reason. A bare directive — "Never use X" — covers @@ -504,6 +538,7 @@ If validation fails, the script prints which checks failed. Fix each issue befor where one instruction ends and the next begins, restructure it 11. When a step drives several varying items, give each its resolved, self-contained inputs — don't make the model join a matrix against a separate list +12. Spend specificity where a miss breaks something; leave out rules for details the model gets right unprompted Cross-references: diff --git a/han-plugin-builder/skills/guidance/references/specialization-and-model-selection.md b/han-plugin-builder/skills/guidance/references/specialization-and-model-selection.md index 76ae5947..525e4a9b 100644 --- a/han-plugin-builder/skills/guidance/references/specialization-and-model-selection.md +++ b/han-plugin-builder/skills/guidance/references/specialization-and-model-selection.md @@ -6,7 +6,7 @@ stay on `opus` no matter how tightly the prompt is written. ## The mechanism -A well-specified skill or agent definition shifts work from inference-time compute (model tier, thinking budget) to +A well-specified skill or agent definition shifts work from inference-time compute (model tier, thinking depth) to prompt-time design. On the tasks the prompt was built for, this lets a smaller model and lower effort match a larger model and higher effort. **It does not raise the model's capability ceiling. It stops the model from wasting capability on disambiguation and planning.** @@ -27,13 +27,13 @@ systematic-decomposition paper (arXiv 2510.07772) reports 10–40 percentage-poi decomposition is applied. The _same_ model performs dramatically better when the task is pre-shaped for it. A specialized skill or agent definition is doing this work upfront. -**3. Anthropic's own design implies the same thing.** Claude's adaptive thinking _"dynamically decides when and how much -to think… at lower effort levels, it may skip thinking for simpler problems."_ A highly specialized prompt makes the -problem _appear simpler_ to the model. Fewer branches to consider, narrower output space, pre-resolved ambiguity. Which -is precisely what reduces the value of extended thinking. +**3. Anthropic's own design implies the same thing.** With adaptive thinking, _"Claude determines whether and how much to +think on each request, and at lower effort settings it may skip thinking entirely on easy inputs."_ A highly specialized +prompt makes the problem _appear simpler_ to the model. Fewer branches to consider, narrower output space, pre-resolved +ambiguity. Which is precisely what reduces the value of thinking. -**4. Few-shot/structured demonstrations substitute for raw reasoning.** Anthropic's chain-of-thought guidance notes that -demonstrating the reasoning pattern in the prompt causes Claude to _"mimic that approach… often to great effect."_ +**4. Few-shot/structured demonstrations substitute for raw reasoning.** Anthropic's prompting guidance notes that worked +examples in the prompt _"shape how Claude approaches similar problems in its own thinking blocks."_ Embedding the reasoning pattern in a skill or agent definition is a permanent few-shot. It shifts work from inference-time compute to prompt design. @@ -78,15 +78,17 @@ The pattern that falls out, by agent archetype: - [How task decomposition and smaller LLMs can make AI more affordable (Amazon Science)](https://www.amazon.science/blog/how-task-decomposition-and-smaller-llms-can-make-ai-more-affordable) - [An approach for systematic decomposition of complex LLM tasks (arXiv 2510.07772)](https://arxiv.org/abs/2510.07772) - [Decomposed Prompting (DecomP)](https://learnprompting.org/docs/advanced/decomposition/decomp) -- [Building with extended thinking (Claude Docs)](https://docs.claude.com/en/docs/build-with-claude/extended-thinking) -- [Let Claude think: chain of thought prompting (Anthropic)](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/chain-of-thought) -- [How Smaller Language Models Outperform LLMs (Deepgram)](https://deepgram.com/learn/the-underdog-revolution-how-smaller-language-models-outperform-llms) +- [Migrating to adaptive thinking (Claude Docs)](https://platform.claude.com/docs/en/build-with-claude/extended-thinking#migrating-to-adaptive-thinking) +- [Prompting best practices: leverage thinking (Claude Docs)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices#leverage-thinking-and-interleaved-thinking-capabilities) +- [How Smaller Language Models Outperform LLMs (Deepgram, archived 2025-09-28)](https://web.archive.org/web/20250928210050/https://deepgram.com/learn/the-underdog-revolution-how-smaller-language-models-outperform-llms). + The original page now returns 404. ## Cross-References - [Per-Model Authoring Guidance](./per-model-authoring.md). This document is about which model _tier_ to run (opus, sonnet, haiku) and at what effort. Its counterpart is about how to _write the instructions_ for a given model. Read - that one when the question is how to phrase instructions for Sonnet 5, Opus 5, or Fable 5, not which tier to run. + that one when the question is how to phrase instructions for a given Sonnet, Haiku, Opus, or Fable model, not which tier to + run. - [Agent Model Selection](./agent-building-guidelines/agent-model-selection.md). Decision criteria for the `model` frontmatter field on agents. - [Agent Domain Focus](./agent-building-guidelines/agent-domain-focus.md). How vocabulary routing, persona length, and diff --git a/han-plugin-builder/skills/skill-builder/SKILL.md b/han-plugin-builder/skills/skill-builder/SKILL.md index 74ddc190..328d0877 100644 --- a/han-plugin-builder/skills/skill-builder/SKILL.md +++ b/han-plugin-builder/skills/skill-builder/SKILL.md @@ -50,6 +50,7 @@ Map from decision to governing document (read just-in-time): | Frontmatter safety (angle brackets, YAML types) | `skill-building-guidance/security-restrictions.md` | | Hardening fuzzy steps into deterministic ones | `skill-building-guidance/hardening-fuzzy-vs-deterministic.md` | | Splitting or composing skills | `skill-building-guidance/skill-decomposition.md`, `skill-building-guidance/skill-composition.md` | +| Instructions for a named target model | `per-model-authoring.md` | | Defining success and tests | `skill-building-guidance/success-criteria-and-testing.md` | | New plugin needed (plugin.json, marketplace.json) | `claude-marketplace-and-plugin-configuration/` and `templates/` | @@ -174,9 +175,10 @@ Cover at minimum: `SKILL.md` cased exactly. 4. **Progressive disclosure** (`progressive-disclosure.md`, `skill-reference-files.md`) — body is process only and under 500 lines; domain knowledge is in `references/`; scripts hold deterministic work; nothing the toolchain already - enforces is restated; every reference file is linked directly from SKILL.md rather than only through another - reference file, every reference link stays inside the plugin directory, and any reference file over roughly 100 lines - opens with a `## Contents` list unless it is a template copied whole into output. + enforces, and no default the model already follows, is restated; every reference file is linked directly from + SKILL.md rather than only through another reference file, every reference link stays inside the plugin directory, and + any reference file over roughly 100 lines opens with a `## Contents` list unless it is a template copied whole into + output. 5. **Instruction quality** (`writing-effective-instructions.md`, `workflow-patterns.md`) — steps are specific and actionable; constraints embed reasoning; error handling is present; human gates sit only at irreversible actions; the most critical item in each list is placed last. @@ -190,6 +192,12 @@ Cover at minimum: `defining-plugin:agent-name`, and the agents actually exist in an installed plugin. 9. **Tests** (`success-criteria-and-testing.md`) — each use case maps to a triggering and functional test the user can run. +10. **Model-specific leftovers** (`per-model-authoring.md`) — no "think step by step" or "think carefully" line; no + instruction to reproduce the model's reasoning in its reply, and no output format with a `reasoning`, `thinking`, + or `trace` field, a scratchpad section, or a running reasoning log; no pinned `effort` without a reason stated in + the body; and no vague limiting phrase in a review step. A limit passes when it names a bar the reader could check and + asks for evidence on each item. "List only problems you'd block the merge for. For each one, give the file and + line, why it's wrong, and how to show it fails." passes; "Only report high-severity issues." does not. Apply the YAGNI discipline throughout: every step, reference file, tool permission, and frontmatter field must earn its place against a real use case. Cut anything added "for completeness" or "for future flexibility."