security-advisory skill never auto-loads under Claude: all six skills:eval positives score 0% (Copilot: 100%)

[cliffhall]

Problem

Under Claude, the security-advisory skill never auto-loads. npm run skills:eval (default AGENT=claude) scores all six of its positive cases at 0%, so a Claude session can reach the skill only when someone types /security-advisory.

Under Copilot the same cases pass. The v2.8.0 release ledger (#2397 row, AGENT=copilot RUNS=1) recorded 6/6 at 100%, and a re-run at RUNS=4 on unchanged v2/main gives the same result. The failure is specific to Claude.

FAIL   0%  security-advisory  Someone just reported a vulnerability against this repo privately. What do I do with it?
FAIL   0%  security-advisory  How do I fix a privately reported vulnerability without the patch being visible before the release goes out?
FAIL   0%  security-advisory  How do I reply to a reporter who filed a vulnerability privately?
FAIL   0%  security-advisory  A privately reported vulnerability turns out to be in an upstream package rather than our own code. How do I close it out?
FAIL   0%  security-advisory  When is it safe to make a privately reported vulnerability public, and who makes that call here?
FAIL   0%  security-advisory  Walk me through taking an accepted vulnerability report all the way to a shipped, disclosed fix.

Both negative cases pass, and every other model-invoked skill scores 100% in the same run (68/74 overall, AGENT=claude, RUNS=4).

Reproduce

On unchanged v2/main (dcc43ad6):

RUNS=2 npm run skills:eval -- security-advisory

→ 2/8 claude first-move cases at or above 80%. All six positives are at 0%.

Background

The skill was added by #2443 / PR #2444. That PR's body says skills:eval was not run. The v2.8.0 ledger then measured it through Copilot only; its #2443 row covered structure (verify:skills), not triggering. So the Claude side has never been measured, and this is not a regression in something that used to work.

Expected

Each positive case at or above the 80% threshold, with the negatives still clean and no drop in the other skills' rates. AGENTS.md asks for a full re-run after any description edit, because adding or changing one skill can pull attention away from the others.

Where to start

• Check what the model does first on these prompts instead of loading the skill. The likely candidates are answering straight from AGENTS.md, which already states the draft-card exception and the human gate on accepting and publishing, or reading SECURITY.md. docs/skill-authoring.md notes that a case whose answer is already in AGENTS.md measures as a miss.
• The description is also written differently from the ones that fire reliably. It leads with a long summary of the flow before "Use when …", while the guidance in docs/skill-authoring.md is to lead with the actions and then list the situations.
• Copilot firing on the same description suggests the difference is in how Claude weighs it against AGENTS.md, which Claude Code loads on every turn, rather than a description that matches nothing. Re-run AGENT=copilot after any fix so the Copilot side doesn't regress.

[cliffhall]

Triage: Priority Medium (total 6)

• Severity 3 — a real feature is missing in practice: the skill cannot load on its own; the workaround (typing /security-advisory) only helps someone who already knows it exists
• Urgency 2 — wanted eventually; advisories are rare and each one is handled by a maintainer who can invoke the skill by name
• Bonuses: +1 bug label

Board: #28, Status Todo.

[cliffhall]

Root cause: a local setting, not the skill. Closing as not planned; nothing in the repo needed to change.

The maintainer's gitignored .claude/settings.local.json contained

"skillOverrides": { "security-advisory": "user-invocable-only" }

It was set by accident in /skills: pressing Enter on a skill cycles it through several modes. user-invocable-only removes a skill from the model's listing and keeps /security-advisory working, which is exactly the symptom. Copilot does not read Claude settings, which is why it scored 100% on the same cases. Claude Code also applies the main checkout's local settings to its worktrees, so the skill was hidden there too.

How it was isolated

• The skill was registered (stream-json init.skills, slash commands) but absent from the model-facing listing. Asked to copy that listing out, headless sessions listed 34 entries without it.
• A usage-count or budget cause was ruled out. A drastically shortened description, a raised skillListingBudgetFraction, and hiding a sibling project skill all left it absent.
• Renaming the skill (advisory-flow) restored it, so the cause was keyed to the name.
• --setting-sources user,project restored it (35 entries), and user,project,local did not. With local settings excluded, Claude's first move was Skill security-advisory on all six eval prompts.

After removing the override: RUNS=2 npm run skills:eval -- security-advisory scores 8/8 at 100% under Claude.

Worth knowing for next time: skills:eval measures the evaluator's own local skillOverrides, not only the committed skills. If a skill scores 0% on every case, run jq .skillOverrides .claude/settings.local.json first.